Image-processing apparatus, and corresponding method and computer program product

Abstract

An image-processing apparatus (101) comprising: - at least one module (106) configured for acquiring a set of images of a scene (100), in particular a stereoscopic-vision system or a multiview stereoscopic system; - at least one depth sensor (104), in particular an active sensor, in particular, said at least one depth sensor (104) being of a LiDAR type, or radar type, or ToF type, or SLAM type, configured for detecting a plurality of sparse depth values (P1,..., PN) in said scene (100); and - at least one processing unit (106), wherein said at least one processing unit (106) is configured for: - receiving said plurality of sparse depth values (P1,..., PN) detected by said at least one depth sensor (104); and - performing at least one virtual -projection procedure (VPP) that comprises: - associating said plurality of sparse depth values (P1,..., PN) of said scene (100) to said set of acquired images of said scene (100), said operation of association comprising, for each depth value in said plurality of sparse depth values (P1,..., PN) detected via said at least one depth sensor (104), associating the depth value of a point of said scene (100) to a respective pixel of each image in said set of acquired images of said scene (100) that corresponds to the same point of the scene (100), said virtual -projection procedure (VPP) comprising generating a set of two- dimensional images (ImRx) of said scene (100) with projected patterns via an operation of augmenting each image in said set of acquired images via the steps of: - identifying a set of corresponding pixels that contains pixels belonging to different images in said set of images to which there has been associated one and the same depth value in said plurality of sparse depth values (P1,..., PN); and - overlaying on each pixel contained in said set of corresponding pixels one and the same image element chosen from a plurality of image elements.

Description

[0001] “Image-processing apparatus, and corresponding method and computer program product”

[0002] TEXT OF THE DESCRIPTION

[0003] Technical field

[0004] The embodiments of the present disclosure relate to image-processing apparatuses.

[0005] In particular, various embodiments of the present disclosure regard solutions for adding references to images, for example, in order to facilitate visual correspondence between a plurality of images.

[0006] Technological background

[0007] Perception and estimation of values regarding depth, three-dimensionality, geometry of surfaces, or any other similar quantity are of fundamental importance in applications regarding the so-called computer stereo vision, i.e., the extraction of information regarding three-dimensionality, depth, geometry of surfaces, or the like in scenes of which a digital representation is to be obtained.

[0008] Examples of such applications may include contexts of autonomous driving, three-dimensional reconstruction, robotics, augmented reality, or the like.

[0009] In the prior art, extraction of information regarding such parameters from scenes of which a digital representation is to be obtained is carried out using optical devices, for example photographic cameras and / or video cameras, given that these are able to reproduce the scene with high resolution.

[0010] A typical example of computer stereo vision is traditional stereoscopic vision, where an attempt is made to reconstruct a three-dimensional shape and / or environment via two different images of one and the same subject captured from different viewpoints and partially overlapping, for example captured by two optical devices, such as video cameras and / or photographic cameras, arranged at a short distance from one another. In traditional stereoscopic vision, information regarding three- dimensionality is usually obtained by examining the relative positions of the elements in the two different images and creating a respective disparity map that contains information regarding the difference of position of the elements in the two different images.

[0011] Hence, given that:

[0012] - the values contained in a disparity map are associated to given pixels of an image representing a given scene, and

[0013] - the aforesaid values are moreover inversely proportional to depth values associated to the respective pixels of the same image, it is possible to determine the depth of the elements in the scene represented and hence a consequent three-dimensionality of the scene itself.

[0014] The above process of examination of the relative positions of the elements in the two different images in order to create the respective disparity map is based upon resolution of the problem of correspondence, i.e., the problem of finding which part of a first image corresponds to a respective part of a second image, where the differences between the first and second images are due, for example, to different viewpoints of the same scene.

[0015] In the context of computer stereo vision, the above resolution of the problem of correspondence is obtained via stereo matchers, i.e., elements configured for receiving at input a plurality of images, for example two images in the case of traditional stereo vision or more than two images in the case of a multiview stereoscopic vision, which represent one and the same scene from different viewpoints, and for determining correspondences between the pixels belonging to such input images, i.e., which pixels of such images correspond to one and the same position in the aforesaid same scene.

[0016] The problem of visual correspondence can be solved via different known methods.

[0017] A first known method is based upon the correlation between the two images; i.e., a check is made to verify whether a location in a first image resembles another location in a second image. For instance, by dividing the images, made up of a plurality of pixels, into smaller sections containing a portion of such pixels, and by associating a section belonging to the first image to a section belonging to the second image if the value of the correlation between the two sections is higher than the value of the correlations calculated between the selected section of the first image and any other non-selected section belonging to the second image.

[0018] A first problem of the above first method is that the pixels of the two images must have a similar light intensity, i.e., a similar photometric consistency; hence, they must exclude any type of non-Lambertian surface and surfaces with any degree of reflectivity.

[0019] Other problems may regard:

[0020] - the need to have images with a sufficiently high resolution so as to avoid having false matches between the two images; and

[0021] - the difficulty in matching sections of images captured from different viewpoints, for example when there is the presence of reflective surfaces, when there are changes in lighting, or any other difference that affects light intensity between the pixels of the two images.

[0022] Moreover, images that present repeated patterns could be misleading for the aforesaid first method, which might match a section of a first image with an erroneous section of a second image, given that, for example, the value of the correlation calculated between the aforesaid section of the first image and the aforesaid erroneous section of the second image, that, for example, represents a repetition of a correct section of the second image, could be higher than the value of correlation calculated between the aforesaid section of the first image and the correct section of the second image.

[0023] A second known method that is able to overcome partially the aforesaid problems is based upon recognition of similar features between the first and second images, for example angles, segments, boundaries of the elements, etc.

[0024] The above second method also presents problems, amongst which there is an inadequacy for analyzing images that do not present well-defined features, for example images with elements having smooth and uniform surfaces, or images with repeated patterns.

[0025] Moreover, a disparity map obtained via the aforesaid second known method does not have the same density as that obtained via the first known method described previously, given that only the points that can be defined via the aforesaid features are considered in associating the portions of the first image with the portions of the second image.

[0026] Also in this case, very reflective elements might cause problems in the association between the two images.

[0027] The aforesaid first and second known methods are applications of so-called passive stereo vision. Moreover, notwithstanding the progress achieved through analysis of images using neural networks, for example via deep learning, it is still difficult to identify visual correspondences in the circumstances set forth above and / or in contexts different from those on which the aforesaid neural networks have been trained; for example, there would be a drastic drop in accuracy if a neural network trained on data in outdoor environments were used for analysing data coming from indoor environments.

[0028] The solution that is hence used in the prior art for mitigating the effect of such problems is a third known method, application of which is referred to as active stereo vision, which is based upon adding light references, whether structured-light references or non-structured light references, the so-called patterns, for example, via laser sources, projectors, or other types of sources, to a scene that is to be captured via an optical device, for example a photographic camera or a video camera, so that the image produced starting from the aforesaid scene will contain such references irrespective of the viewpoint from which the capture is made, thus facilitating association between the two images, i.e., between corresponding regions in the two images.

[0029] The aforesaid light references are usually light patterns, for example vertical or horizontal stripes of light, grids, or other patterns, whether regular or irregular / random, actively projected onto a scene to be captured so that the distortion of the aforesaid known patterns due to the elements of the scene will in any case enable association of points belonging to the first image, for example the points of the aforesaid first image belonging to a given projected horizontal or vertical stripe, to points belonging to the second image, for example the points of the aforesaid second image belonging to the same projected horizontal or vertical stripe.

[0030] However, also this solution can be applied only in indoor environments or in environments with poor solar lighting, given that high levels of environmental lighting would interfere with the light pattern used for creating the references in the images.

[0031] In fact, this third method can be used in conditions of controlled environmental lighting given that the reading of the data and the accuracy of the references depend upon the performance of the light source, for example a projector. Scans made outdoors or in daylight usually produce poor results even when a laser source is used as light source given that the pattern projected on the scene has to be clearly visible.

[0032] Moreover, the instruments for generation of patterns may be costly and, even when the lighting conditions are optimal, the light beams projected are not in any case suited for long-distance measurements given that such light beams are not able to propagate over long distances, tending to attenuate already after just a few metres and becoming increasingly noisy and imprecise as the distance from the source increases, thus rendering the known patterns projected on the scene not clearly distinguishable.

[0033] In addition, in some acquisition structures, the above light beams may interfere with one another, for example in the case of reflective surfaces or in the case where a number of projectors or sources are used, or else may be affected by thermal drifts or by other factors that alter the shape and precision thereof, for example in the case where a pattern has to remain unvaried.

[0034] It would hence be advantageous to have available alternative solutions for generation of images that solve the problem of correspondence, for example, between images to be supplied at input to a stereo matcher, that are able to obtain the aforesaid images also in unfavourable environmental conditions like the ones mentioned in regard to the known methods described previously, in any case maintaining a resolution comparable with that of the optical devices used in the known solutions, and increasing the precision and robustness of the resolution of the problem of correspondence.

[0035] Object and summary

[0036] An object of one or more embodiments is to overcome the problems described previously, favouring the solution of the problem of correspondence in different environmental conditions and in a precise, robust, way and maintaining a high resolution.

[0037] According to one or more embodiments, the above object may be achieved via an image-processing apparatus that is able to detect a plurality of depth values, including sparse values, for example via a depth sensor of an active or non-active type, and generate images that contain corresponding image elements, for example one and the same image pattern at one and the same depth value of the aforesaid plurality of depth values.

[0038] One or more embodiments refer to a corresponding method and to a corresponding computer-program product that can be loaded into at least one processing unit, for example a computer, and comprises portions of code of a program for executing the steps of the aforesaid method when the computer- program product is run on the at least one processing unit.

[0039] Reference to a computer-program product as used in the present document is to be understood as being equivalent to reference to a computer-readable medium containing instructions for controlling a processing unit or a plurality of processing units in order to implement the aforesaid method according to one or more embodiments of the solution described herein.

[0040] Consequently, the present detailed description regards an image-processing apparatus comprising:

[0041] - at least one module configured for acquiring a set of images of a scene, in particular a stereoscopic-vision system or a multiview stereoscopic system; - at least one depth sensor, in particular an active sensor, said depth sensor being in particular of a LiDAR type, or a radar type, or a ToF type, or a SLAM type, that is configured for detecting sparse depth values in the aforesaid scene; and

[0042] - at least one processing unit, wherein said at least one processing unit is configured for:

[0043] - receiving said sparse depth values detected by said depth sensor; and

[0044] - performing at least one virtual -project! on procedure comprising:

[0045] - associating said plurality of sparse depth values of said scene to said set of acquired images of said scene, said operation of association comprising, for each depth value in said plurality of sparse depth values detected via said at least one depth sensor, associating a considered depth value corresponding to a point of said scene to a respective pixel of each of said images in said set of acquired images of said scene that corresponds to the same point of the scene, said virtual -projection procedure comprising generating a set of two- dimensional images of said scene with projected patterns via an operation of augmenting said acquired images through the steps of:

[0046] - identifying a set of corresponding pixels that contains pixels belonging to different images of said set of images to which one and the same depth value in said plurality of sparse depth values has been associated; and

[0047] - overlaying on each pixel contained in said set of corresponding pixels one and the same image element chosen from a plurality of image elements.

[0048] In various embodiments, said apparatus is coupled to, or comprises, a stereo matcher, configured for receiving at input said set of two-dimensional images with projected patterns from said image-processing apparatus, where in particular each two-dimensional image of said set of two-dimensional images with projected patterns corresponds to a respective image of said set of images acquired by the at least one module.

[0049] In various embodiments, said at least one module configured for acquiring said set of images of said scene comprises at least one image-capture sensor, which in particular corresponds to optical devices that implement a stereoscopic-vision system or a multiview stereoscopic system and is configured for acquiring a first subset of said set of images, and said at least one processing unit is further configured for receiving said first subset of said set of images acquired via said at least one image-capture sensor.

[0050] In various embodiments, said at least one image-capture sensor and said at least one depth sensor are obtained via a single sensor configured both for detecting sparse depth values in said scene and for acquiring said first subset of said set of images of said scene.

[0051] In various embodiments, said at least one module configured for acquiring said set of images of said scene comprises the processing unit, said processing unit being further configured for simulating at least one virtual image-capture sensor that is configured for acquiring a second subset of said set of images, in particular where each image in said second subset of said set of images comprises virtual pixels that are configured for containing any colour, in particular black.

[0052] In various embodiments, applied to said at least one image-capture sensor and to said at least one depth sensor is a process of calibration with respect to depth, and said calibration with respect to depth comprises, for each depth value in said plurality of sparse depth values detected via said at least one depth sensor and for each image that is to be calibrated with respect to depth in said first subset of said set of images acquired by said at least one image-capture sensor:

[0053] - determining, for a considered depth value in said plurality of sparse depth values detected via said at least one depth sensor, a respective pixel belonging to an image considered to be calibrated with respect to depth in said first subset of said set of images acquired by said at least one image-capture sensor, where said considered depth value and said respective pixel are associated to one and the same point of the scene;

[0054] - determining a set of co-ordinates associated to said respective pixel by considering a reference system associated to said considered image to be calibrated with respect to depth that contains said respective pixel; and

[0055] - associating said set of co-ordinates to said considered depth value in said plurality of sparse depth values detected via said at least one depth sensor. In various embodiments:

[0056] - said first subset of said set of images comprises at least two images;

[0057] - at least one image in said first subset of said set of images is an image calibrated with respect to depth;

[0058] - at least one image in said first subset of said set of images is an image not calibrated with respect to depth, not subjected to said process of calibration with respect to depth; and

[0059] - association, for each depth value in said plurality of sparse depth values detected via said at least one depth sensor, between a considered depth value corresponding to a point of said scene and a respective pixel of said at least one image not calibrated with respect to depth corresponding to said scene and to the same point of the scene, via said virtual -projection procedure comprises:

[0060] - computing at least one disparity value indicating a distance between a first position of the at least one image-capture sensor during a process of acquisition of said at least one image calibrated with respect to depth and a second position of the at least one image-capture sensor during a process of acquisition of said at least one image not calibrated with respect to depth, as a function of said considered depth value;

[0061] - computing at least one value of a co-ordinate that is associated to said respective pixel by considering a reference system associated to said image not calibrated with respect to depth that contains said respective pixel, as a function of said at least one disparity value; and

[0062] - associating said considered depth value to said respective pixel of said at least one image not calibrated with respect to depth via said at least one calculated value of co-ordinate.

[0063] In various embodiments:

[0064] - the set of images comprises at least two images;

[0065] - at least one image in said set of images is acquired from a first position that corresponds to the position of said at least one depth sensor to obtain at least one image in which each of the sparse depth values that correspond to a point of the scene is associated to a pixel of said at least one image that corresponds to the same point of said scene;

[0066] - at least one image in said set of images is acquired from a second position different from said first position; and

[0067] - association, for each depth value in said plurality of sparse depth values detected via said at least one depth sensor, between a considered depth value that corresponds to a point of said scene and a respective pixel of said at least one image acquired from said second position that corresponds to the same point of the scene, via said virtual -projection procedure, comprises:

[0068] - computing at least one disparity value indicating a distance between said first position and said second position as a function of said considered depth value;

[0069] - computing at least one value of a co-ordinate associated to said respective pixel by considering a reference system associated to said at least one image acquired from said second position that contains said respective pixel, as a function of said at least one disparity value; and

[0070] - associating said considered depth value to said respective pixel of said at least one image acquired from said second position via said at least one calculated value of co-ordinate.

[0071] In various embodiments:

[0072] - each image element in said plurality of image elements is associated to at least one respective value; and

[0073] - said image element to be overlaid on said pixels contained in said set of corresponding pixels is chosen randomly; in particular said image element is chosen if the at least one respective value associated to said image element is drawn from a probability distribution, in particular, a uniform probability distribution.

[0074] In various embodiments, said image element to be overlaid on said pixels contained in said set of corresponding pixels is chosen via the following operations:

[0075] - generating a plurality of windows, wherein said plurality of windows comprises at least one window for each pixel in said set of corresponding pixels that is centred on said pixel in said set of corresponding pixels; - defining a plurality of histograms, wherein said plurality of histograms comprises one histogram for each window in said plurality of windows, said histogram containing a value of frequency of occurrence within said window for each image element in said plurality of image elements;

[0076] - defining a sum histogram as the sum of the histograms contained in said plurality of histograms, containing a value of frequency of occurrence within said plurality of windows for each image element in said plurality of image elements;

[0077] - defining a distance histogram that contains, for each image element in said plurality of image elements, an indication on the minimum distance between said considered image element in said plurality of image elements and any one image element that corresponds to a value of frequency of occurrence greater than zero in said sum histogram; and

[0078] - choosing, as image element to be overlaid on said pixels contained in said set of corresponding pixels, the image element in said plurality of image elements the value of minimum distance of which contained in said distance histogram is greater than the values of minimum distance that are associated to the other image elements in said plurality of image elements contained in said distance histogram.

[0079] In various embodiments, said sum histogram contains values of frequency of occurrence higher than zero for each image element in said plurality of image elements, the chosen image element to be overlaid on said pixels contained in said set of corresponding pixels is the image element the value of frequency of occurrence of which contained in the sum histogram is lower than the values of frequency of occurrence contained in said sum histogram associated to the other image elements in said plurality of image elements.

[0080] In various embodiments, said overlaying, on each pixel contained in said set of corresponding pixels, of one and the same image element chosen from a plurality of image elements further comprises:

[0081] - overlaying said same image element on pixels comprised in a given window that is centred on said pixel contained in said set of corresponding pixels; or else - overlaying further image elements chosen from said plurality of image elements on pixels comprised in a given window that is centred on said pixel contained in said set of corresponding pixels.

[0082] In various embodiments, said overlaying on each pixel contained in said set of corresponding pixels one and the same image element chosen from a plurality of image elements is obtained by combining said pixel contained in said set of corresponding pixels with said image element via the alpha-blending technique.

[0083] In various embodiments:

[0084] - at least one image in said set of images is a reference image; and

[0085] - said at least one processing unit is further configured to execute, for each image, except for said reference image, in said set of images, operations for:

[0086] - computing a plurality of disparity values indicating a distance between a first position of acquisition of a considered image in said set of images and a second position of acquisition of said reference image, as a function of the plurality of depth values;

[0087] - mapping said plurality of disparity values on a plurality of cells of a grid, selecting for each cell the disparity value having the highest value from among a set of disparity values that contains disparity values that have to be mapped on said considered cell;

[0088] - classifying each cell in said plurality of cells of said grid as occluded if there exists at least one other cell in a neighbourhood centred on said cell that corresponds to a disparity value that differs from a disparity value corresponding to said cell by a value higher than a given threshold; and

[0089] - obtaining an occlusion mask that indicates the positions, within the grid, of the occluded cells as a function of the cells classified as occluded. In various embodiments, said occlusion mask is superimposed upon at least two images in said set of images to obtain a correspondence between the cells classified as occluded and the pixels of said at least two images considered by applying the following criteria: - preventing overlaying an image element on said pixels corresponding to said cells classified as occluded;

[0090] - dividing said pixels corresponding to said cells classified as occluded into occluded pixels, in which a respective considered point of said scene is visible, and occluding pixels, in which a respective considered point of said scene is not visible; and

[0091] - replacing the content of said occluded pixels that belong to one of said at least two considered images with the content of said occluding pixels that belong to another of said at least two considered images, wherein an occluded pixel and a respective occluding pixel the content of which is substituted for that of the occluded pixel are associated to one and the same respective point of said scene.

[0092] The solutions as described in this detailed description hence facilitate provision of an apparatus capable of processing images that enables the problem of correspondence to be solved, for example an apparatus capable of obtaining the aforesaid images even in unfavourable environmental conditions like the ones mentioned in relation the known methods described previously, in any case maintaining a resolution comparable to that of the optical devices used in the known solutions and increasing the precision and robustness of the solution to the problem of correspondence.

[0093] Brief description of the drawings

[0094] One or more embodiments will now be described, purely by way of example, with reference to the annexed drawings, wherein:

[0095] - Figure 1 is an image-processing apparatus that is able to generate images that contain image elements according to embodiments of the solutions described;

[0096] - Figure 2 is an image-processing apparatus that is able to add image elements to images obtained via a single optical device and, possibly, also to generated images, according to embodiments of the solutions described;

[0097] - Figure 3 is an image-processing apparatus that is able to add image elements to images obtained via one or more optical devices and, possibly, also to generated images, according to embodiments of the solutions described; and - Figure 4 is a possible application of the apparatus of Figure 1, Figure 2, or Figure 3, where the aforesaid apparatus is used to obtain a disparity map via a stereo matcher according to embodiments of the solutions described.

[0098] The above figures are reproduced to illustrate clearly the important aspects of the embodiments and are not necessarily represented in scale.

[0099] Detailed description of examples of embodiments

[0100] In the ensuing description, one or more specific details are illustrated in order to enable an in-depth understanding of examples of embodiments of the present disclosure. The embodiments may be obtained without one or more of the specific details or with other methods, components, materials, etc. In other cases, known operations, materials, or structures are not illustrated or described in detail so that certain aspects of the embodiments will not be obscured.

[0101] Reference to “an embodiment” or “one embodiment” in the framework of the present description is intended to indicate that a particular configuration, structure, or characteristic described with reference to the embodiment is comprised in at least one embodiment. Hence, phrases such as “in an embodiment” or “in one embodiment” or the like that may be present in one or more points of the present disclosure description do not necessarily refer precisely to one and the same embodiment.

[0102] Moreover, particular configurations, structures, or characteristics may be combined in any adequate way in one or more embodiments.

[0103] The references used herein are provided merely for convenience and hence do not define the sphere of protection or scope of the embodiments.

[0104] In all the figures annexed hereto and in the entire ensuing detailed description, unless the context indicates otherwise, the parts or elements that are similar are designated by similar references / numbers, and a corresponding description will not be repeated for reasons of brevity.

[0105] As described previously, one or more embodiments of the present solution regard an image-processing apparatus that is able to detect a plurality of even sparse depth values, for example via a depth sensor of an active or non-active type, and to generate images that contain corresponding image elements, for example one and the same image pattern, at one and the same depth value in the aforesaid plurality of depth values.

[0106] Embodiments of the proposed solution regard an image-processing apparatus that is able to generate images that contain image elements, for example image elements obtained as a function of corresponding depth values, via the use of a sensor, for example, but not necessarily, an active sensor, for measuring sparse depth values, for example of a LiDAR (Light Detection And Ranging) type, a radar (Radio Detection and Ranging) type, a ToF (Time of Flight) type, a SLAM (Simultaneous Localization And Mapping) type, or any other known sensor, in particular of an active type, or any other device that is able to measure depth values, including sparse depth values.

[0107] By “active sensor” is here meant, for example, a sensor that uses a controlled source with structured emission of energy, such as a source of a scanning laser or of a light pattern or of some other emission of electromagnetic energy. Active sensors, albeit able to measure depth values, are rarely used in the prior art given that they are typically characterized by a low resolution as compared to the resolution of images obtained by optical devices, for example photographic cameras or video cameras; i.e., they are characterized by a low density of depth values detected as compared to a density of pixels detected by the aforesaid optical devices. Such a low resolution is not sufficient to be able to obtain information on three- dimensionality, or depth, or on the geometry of surfaces in a way sufficiently accurate to enable a digital representation of a scene considered to be obtained.

[0108] For instance, a sensor of a LiDAR type may be considered reliable, but it is characterized by a density of detection much lower than that of current optical devices, and the increase of the aforesaid density in a LiDAR sensor would be difficult to implement on account of the high costs and the technological difficulties involved.

[0109] For instance, a sensor of a radar type makes it possible to measure depth values at a longer distance, but turns out to have a density even lower than that of a sensor of a LiDAR type, to be noisier, and to have a field of view that extends vertically in a very limited way.

[0110] It should be noted that the aforesaid depth sensors, in particular active sensors or non-active sensors, are rarely used for solving the problem of visual correspondence, for example via stereo matching, given that these depth sensors are not configured to provide a plurality of images to be used as input elements for a stereo matcher, for example a pair of images, as a function of the depth values detected; consequently, usually such input elements for a stereo matcher are obtained via passive sensors, i.e., optical devices such as stereoscopic photographic cameras and / or video cameras, which are able to obtain information at a higher resolution, also with the possible aid of lights, whether structured, i.e., with known patterns, or non-structured (i.e., via possible physical projectors).

[0111] Figure 1 shows an image-processing apparatus 10i that is able to generate images that contain image elements ImRx(two-dimensional images with projected patterns, for example, virtual patterns) according to embodiments of the solutions described.

[0112] Shown in Figure 1 is a scene 100, a digital representation of which is to be obtained. This scene 100 is acquired via a set of depth sensors 104, i.e., one or more sensors, in particular active ones, that are able to detect sparse depth values, for example a LiDAR sensor.

[0113] The aforesaid set of depth sensors 104, in particular a set containing at least one active sensor, for example a LiDAR sensor, a radar sensor, a ToF sensor, or the like, is configured for acquiring a plurality of sparse depth values Pi, ... , PN regarding the aforesaid scene 100, where N denotes the total number of the depth values detected by the set of depth sensors 104, and for sending the aforesaid plurality of sparse depth values Pi, . . ., PN to one or more processing units 106.

[0114] The aforesaid one or more processing units 106 may be either local or remote, for example a control logic, a microcontroller, a microprocessor, a personal computer, a computer system, including a distributed system, for example a cloud distributed system, or any other similar unit. In various embodiments, the aforesaid set of depth sensors 104 configured for acquiring a plurality of sparse depth values Pi, . . . , PN may comprise one or more non-active sensors. In particular, the aforesaid sparse depth values Pi, ..., PN may even be obtained from other types of systems, for example from images, e.g., via SLAM, SfM (Structure from Motion), or similar systems.

[0115] Consequently, to use the aforesaid sparse depth values Pi, ..., PN in a context of stereoscopic vision with the aim of overcoming the problem of correspondence, for example with the aid of a stereo matcher, the one or more processing units 106 are configured for simulating a plurality of virtual optical devices VOD (i.e., of virtual image-capture sensors), for example a plurality of virtual photographic cameras (e.g., two photographic cameras in the case of traditional stereoscopic vision), for example modelled via a pinhole model, that simulate the behaviour of real optical devices; i.e., they are configured for acquiring virtual images Imvx, where the subscript x denotes a number associated to a corresponding virtual image.

[0116] For instance, if the virtual optical devices VOD acquire two virtual images from two different viewpoints, a first virtual image acquired is designated by Imvi, while a second virtual image acquired is designated by Imv2.

[0117] It should be noted that, when we refer to acquisition of virtual images Imvxby the aforesaid virtual optical devices VOD, we are not referring to a process of image acquisition via image-capture sensors, for example as occurs in real optical devices comprising the aforesaid image-capture sensors.

[0118] In fact, the above virtual optical devices VOD do not comprise any image- capture sensor and so that they are not able to acquire an image of the aforesaid scene 100 according to the acquisition process that characterizes real optical devices that comprise such image-capture sensors.

[0119] Consequently, when we refer to acquisition of virtual images Imvxby such virtual optical devices VOD, we are referring a method that makes it possible to obtain a virtual image Imvxthat does not contain information on the aforesaid scene 100, but the virtual pixels Imvx(m, y) of which are associated to any colour of fill, for example a fill with a colour obtained according to an RGB (Red, Green, Blue) model, in particular a black fill or a fill of any other colour or set of colours.

[0120] Each of the aforesaid virtual optical devices VOD is positioned at a different set of known co-ordinates, for example chosen in an arbitrary way or according to any criterion, in a given reference system.

[0121] In fact, once any reference system has been chosen, it is possible to:

[0122] - select, for example in an arbitrary way, intrinsic parameters for each virtual optical device in the aforesaid plurality of virtual optical devices VOD, for example, a focal distance f and a respective centre c; and

[0123] - select, for example once again in an arbitrary way or according to any criterion, a different set of co-ordinates for each virtual optical device in the aforesaid plurality of virtual optical devices VOD with respect to the reference system chosen.

[0124] It should be noted that the intrinsic parameters and the sets of co-ordinates may be chosen so as to be advantageous, for example so as to implement a traditional stereoscopic-vision system or a multiview stereoscopic system, or else by positioning one of the aforesaid virtual optical devices VOD so as to get it to correspond to one of the one or more depth sensors 104 in order to reduce the number of occlusions and facilitate matching of the depth values detected to the pixels of the images acquired by the aforesaid virtual optical device.

[0125] Moreover, the one or more processing units 106 are further configured to execute a virtual -projection procedure VPP, which consists in associating each of the sparse depth values Pi, ..., PN of the scene 100 to one or more virtual pixels Imvx(m, y) belonging to different virtual images Imvx.

[0126] For instance, a depth value Pi corresponding to a given point PTh of the aforesaid scene 100 is associated to:

[0127] - a first virtual pixel Imvi(m, y) belonging to a first virtual image Imvi, the co-ordinates (m, y) of which are expressed according to a reference system fixed with respect to the aforesaid first virtual image Imvi obtained via a first virtual optical device; and - a second virtual pixel Imv2(m’, y’) belonging to a second virtual image Imv2, the co-ordinates (m’, y’) of which are expressed according to a reference system fixed with respect to the aforesaid second virtual image Imv2 obtained via a second virtual optical device, wherein the aforesaid first and second virtual pixels Imvi(m, y), Imv2(m’, y’) correspond to the same point PTh of the scene 100.

[0128] The above operation of association can be carried out given that the position of a virtual optical device considered, and, consequently, of a corresponding virtual image Imvxacquired via the aforesaid virtual optical device, is defined in an arbitrary way (as described above) and is hence known.

[0129] Consequently, given that the position of the virtual image Imvxacquired via the aforesaid virtual optical device is known, it is possible to identify the virtual pixel Imvx(m, y), the co-ordinates (m, y) of which, expressed in a reference system fixed with respect to the virtual image Imvx, correspond to a given point PTh of the scene 100 (e.g., the same point PTh of the scene 100 as the one corresponding to which is a depth value Pi).

[0130] Furthermore, since also the relative position of the aforesaid virtual optical device and of the corresponding virtual image Imvxwith respect to each of the depth sensors contained in the set of depth sensors 104 is known, it is possible to associate each of the sparse depth values Pi, . . ., PN corresponding to given points PTh of the scene 100 to respective virtual images Imvxand consequently to respective virtual pixels Imvx(m, y) that correspond to the same given points PTh of the scene 100.

[0131] A method provided by way of example for making the aforesaid association between depth values and corresponding pixels will be described hereinafter with reference to Figure 3.

[0132] In particular, a sparse depth value Pi associated to a corresponding virtual pixel Imvx(m, y) is denoted in the ensuing description by the reference Pi(m, y), where (m, y) are the co-ordinates of the virtual pixel Imvx(m, y) with respect to the respective virtual image Imvx.

[0133] Consequently, it is possible to associate, via said virtual -projection procedure VPP, said plurality of sparse depth values Pi, . . ., PN of said scene 100 to said set of images of said scene 100, for example, virtual images Imvxacquired via virtual optical devices.

[0134] Hence, the aforesaid operation of association comprises, for each depth value in said plurality of sparse depth values Pi, . . ., PN detected via said at least one depth sensor 104, associating a considered depth value regarding a point PTh of said scene 100 to a respective pixel, for example a virtual pixel Imvx(m, y), of each of said images in said set of images of the scene 100, for example virtual images Imvx, acquired for the same point PTh of the scene 100.

[0135] In this way, it is possible to identify a set of corresponding pixels that contains pixels, for example virtual pixels Imvx(m, y), that belong to different images in said set of images, for example virtual images Imvx, to which there has been associated one and the same depth value in said plurality of sparse depth values Pi, . . . , PN.

[0136] For instance, a set of corresponding pixels may contain a plurality of pixels, for example virtual pixels Imvx(m, y), that belong to different images in said set of images, e.g., virtual images Imvx, which, during the above operation of association, have been associated to one and the same depth value in said plurality of sparse depth values Pi, . . . , PN.

[0137] The virtual -projection procedure VPP further comprises an operation of proj ection of the aforesaid depth values Pi, . . . , PN on the above virtual images Imvx, in particular, on the corresponding virtual pixels Imvx(m, y) associated to the aforesaid depth values Pi, . . . , PN, SO as to generate a set of two-dimensional images ImRxwith projected patterns, for example virtual patterns, i.e., images containing a plurality of image elements.

[0138] The aforesaid operation of projection consists in the selection of an image element EV from a set of image elements for each set that contains a depth value Pi and the corresponding virtual pixels Imvx(m, y) associated to the above depth value Pi, i.e., for each set that contains a depth value Pi corresponding to a given point PTh of a scene 100 and each virtual pixel Imvx(m, y) corresponding to the same point PTh of the same scene 100, and in replacement of the content of the aforesaid virtual pixels Imvx(m, y) associated to the aforesaid depth value Pi (and possibly, also of the pixels comprised in a neighbourhood of one of such associated pixels, for example ones comprised within a given window centred on this pixel) with the selected image element EV.

[0139] The aforesaid image elements EV may correspond, for instance to:

[0140] - individual coloured pixels, for example each having a different colour;

[0141] - groups of pixels, for example pixels comprised within a window of fixed or variable known dimensions, where the pixels that belong to one and the same group are coloured with one and the same colour, whereas, for example, pixels that belonging to different groups may be coloured with different colours;

[0142] - groups of pixels where each pixel belonging to one and the same group is coloured with a different colour to form a given pattern of colours and where different groups are characterized by different patterns of colours; and / or

[0143] - any other type of pattern based upon colouring of one or more pixels with one or more colours using different strategies, for example taking into consideration different shapes of the window considered.

[0144] Consequently, said virtual -projection procedure VPP further comprises generating a set of two-dimensional images ImRxwith projected patterns (i.e., a plurality of image elements A) of said scene 100 via an operation of augmenting the said acquired images, for example, virtual images Imvx, by overlaying one and the same image element A chosen from a plurality of image elements on each pixel, e.g., virtual pixels Imvx(m, y) belonging to said virtual images Imvx, that is contained in said set of corresponding pixels, i.e., by overlaying one and the same image element A on each pixel that during the aforesaid operation of association has been associated to one and the same depth value in said plurality of sparse depth values Pi, ..., PN.

[0145] The aforesaid operation of overlaying hence corresponds to an operation of projection of patterns, for example virtual patterns, on the aforesaid images, e.g., virtual images Imvx, to obtain the aforesaid two-dimensional images ImRxwith projected patterns.

[0146] The aforesaid pattern is hence formed by a plurality of image elements A that are projected, i.e., substituted for corresponding pixels, on the images, for example virtual images Imvx, so as to obtain the aforesaid two-dimensional images ImRx with projected patterns.

[0147] To sum up, the aforesaid virtual -projection procedure VPP, i.e., an operation that makes it possible to project a pattern, for example a virtual pattern, on acquired images, in order to carry out augmenting thereof, corresponds to an operation of projection performed by overlaying on each pixel contained in a set of pixels identified as being associated to one and the same depth value one and the same image element A chosen from a plurality of image elements, for example different colours for each set of pixels identified.

[0148] Hence, if the content of each of the virtual pixels Imvx(m, y) associated to one and the same depth value Pi is replaced with one and the same image element EV, it is possible to facilitate, if the virtual images Imvxare at least two (i.e., if at least two two-dimensional images ImRxare obtained that correspond to the aforesaid virtual images Imvxbut contain a projected pattern, for example a virtual pattern, i.e., a plurality of image elements A), an association between them, namely, to facilitate resolution of the problem of correspondence, i.e., identification of corresponding points, i.e., pixels, between different images, with the aid of a stereo matcher.

[0149] In fact, the aforesaid virtual -projection procedure VPP is able to augment the information content of the virtual images Imvxat input, in particular by substituting image elements obtained as a function of depth values for the content of given virtual pixels Imvx(m, y), thus obtaining the aforesaid two-dimensional images ImRxwith projected patterns, for example a virtual pattern. The aforesaid two-dimensional images ImRxwith projected patterns are hence characterized by an increase of the distinctive features between different pixels of one and the same image and an increase of the similarities between corresponding pixels, i.e., ones to be associated during solution of the problem of correspondence, of different images, thus facilitating association between a plurality of two-dimensional images ImRxwith projected patterns even in the absence of context, i.e., without acquisition of any real image. The processing unit 106 is hence configured for receiving the aforesaid plurality of sparse depth values Pi, PN, simulating a plurality of virtual optical devices VOD so as to obtain a plurality of virtual images Imvx, and supplying at output one or more images ImRx, where the subscript x has the same meaning as described previously, which contain a projected patterns, for example virtual patterns, where each image ImRxat output corresponds to a respective virtual image Imvxat input, for example to an image containing black virtual pixels Imvx(m, y), but unlike the virtual image Imvxat input contains, in addition, information obtained as a function of the aforesaid depth values Pi, ..., PN, i.e., a plurality of image elements chosen as a function of the depth values.

[0150] In variant embodiments of the present solution, to facilitate even further solution of the problem of correspondence, for example with the aid of a stereo matcher, a set of image-capture sensors 102, in particular passive sensors, i.e., one or more optical devices, for example video cameras, photographic cameras, and / or any other similar device, are added to the aforesaid image-processing apparatus 10i .

[0151] These variant embodiments are illustrated in Figure 2, which represents an image-processing apparatus IO2, and in Figure 3, which represents an image- processing apparatus IO3.

[0152] It should be noted that when reference is made to a plurality of optical devices in the descriptions of the ensuing embodiments, it is understood that, if the scene to be captured is substantially static, it is always possible to use, as variant embodiment, an apparatus containing a single optical device that is positioned at different instants in different locations to acquire images of one and the same subject from different viewpoints using the same optical device.

[0153] It should be noted that embodiments of the present solution may be used to insert image elements into images in various contexts of application.

[0154] In fact, in addition to the possibility of being used for estimating a depth, obtaining information on geometries of surfaces, reconstructing a three- dimensionality, or for other similar applications in a context of application of a traditional stereoscopic vision, i.e., an application where it is attempted to reconstruct, for example, three dimensionality via two different and partially overlapping views of one and the same subject coming from two optical devices arranged at a short distance from one another, the aforesaid embodiments may also be used in other contexts, such as:

[0155] - a context of multiview stereoscopic vision, i.e., an application where it is attempted to reconstruct, for example, three-dimensionality, via more than two partially overlapping images of one and the same subject obtained from different viewpoints;

[0156] - a context of depth vision obtained with a single optical device, i.e., an application where it is attempted to reconstruct, for example, three-dimensionality and a corresponding depth, via a single image obtained using a single optical device; or else

[0157] - a context of optical flow, i.e., an application in which the motion of an object in a digital visual representation is considered, for example by representing the aforesaid motion with a vector that originates from and terminates on given pixels in a sequence of images, and assigning to each pixel belonging to a current image a vector that points towards the position of a corresponding pixel, i.e., that represents the same portion of the moving object, in a subsequent image.

[0158] More in general, the aforesaid embodiments maybe used in any context in which it is necessary to determine a depth, a three-dimensionality, a geometry of the surfaces, or an optical flow starting from one or more images, or else in any context in which it is necessary to solve the problem of correspondence, for example with the aid of a stereo matcher.

[0159] The above contexts of application may be defined in general as contexts of computer stereo vision, i.e., applications where it is attempted to extract information on three-dimensionality, depth, or geometry of surfaces in scenes represented in digital images.

[0160] Embodiments of the proposed solution are based upon the integration of information coming from the aforesaid image-capture sensors, for example passive sensors, i.e., one or more optical devices, and information coming from the aforesaid depth sensors, in particular active ones, i.e., at least one sensor that is able to detect depth values, without the aid of a corresponding projector or light source, so as to facilitate solution of the problems of the prior art described previously.

[0161] In various embodiments, the set of image-capture sensors 102 may comprise a single element, for example a single device configured for image acquisition that may be fixed with respect to the at least one depth sensor 104, thus obtaining a vision of a monocular type.

[0162] Figure 2 is an image-processing apparatus IO2 that is able to add image elements to images obtained via a single (monocular) optical device 102 according to embodiments of the solutions described.

[0163] Represented in Figure 2 is a scene 100 of which a digital representation is to be obtained. The aforesaid scene 100 is acquired via the image-processing apparatus IO2, and in particular by a single image-capture sensor 102, in particular a passive sensor, i.e., an optical device, for example a video camera, a photographic camera, and / or any other similar device, and by a set of depth sensors 104, for example a single sensor, in particular, an active one, that is able to detect sparse depth values, for example a LiDAR sensor.

[0164] The aforesaid single image-capture sensor 102 is configured for acquiring a plurality of two-dimensional images Imxof the aforesaid scene 100, where the subscript x denotes a number associated to a corresponding image; for example, if the single image-capture sensor 102 acquires two images from two different viewpoints, a first acquired image is denoted by Imi, while a second acquired image is denoted by Irro.

[0165] Moreover, the aforesaid single image-capture sensor 102 is configured for acquiring a plurality of two-dimensional images Imxfrom different viewpoints, for example viewpoints separated horizontally or vertically by a given distance; namely, the single image-capture sensor 102 is configured to execute a movement, for example a horizontal movement, a vertical movement, or any other movement, between acquisition of a first image and acquisition of a second image that is acquired from a viewpoint different from the one corresponding to the aforesaid first image and at a second instant. The aforesaid single image-capture sensor 102 is hence configured for sending the aforesaid plurality of two-dimensional acquired images Imxto one or more processing units 106, which, as in the case of Figure 1, may be either local or remote, for example a control logic, a microcontroller, a microprocessor, a personal computer, a computer system, including a distributed system, for example a cloud distributed system, or any other similar unit.

[0166] The set of depth sensors 104, in particular an active sensor, for example a LiDAR sensor, a radar sensor, a ToF sensor, or the like, is configured for acquiring a plurality of sparse depth values Pi, . . ., PN that correspond to the aforesaid scene 100, where N denotes the total number of the depth values detected by the set of depth sensors 104, and sending the aforesaid plurality of sparse depth values Pi, . . . , PN to the one or more processing units 106.

[0167] It should be noted that also in the embodiments illustrated in Figure 2, the aforesaid depth sensor 104 may be any depth sensor, whether active or not, as described in relation to Figure 1.

[0168] In the embodiments illustrated in Figure 2, the above single image-capture sensor 102 may be integrated with the aforesaid depth sensor 104 to obtain, that is, a single sensor 102-104 configured for acquiring a plurality of images Imxvia an image-capture sensor 102, detecting a plurality of sparse depth values Pi, . . ., PN via a depth sensor 104, and sending the plurality of images Imxand the plurality of sparse depth values Pi, . . ., PN to the one or more processing units 106.

[0169] In various embodiments, the single sensor 102-104 has to be configured so that it can be shifted into a plurality of different positions to be able to acquire the plurality of images Imxof the same scene 100 from different viewpoints. It should moreover be noted that these acquisitions are made at different instants in so far as the single sensor 102-104 must have the time to shift from one position to another before acquiring a subsequent image (consequently, the aforesaid scene 100 is static; i.e., it remains the same at the different instants of acquisition).

[0170] In various embodiments, also the sparse depth values Pi, ..., PN associated to the scene 100 can be acquired a plurality of times and at different instants; for example, an acquisition may be made at each position of the plurality of positions. It should be noted that, by acquiring the aforesaid sparse depth values Pi, . . . , PN from at least two different positions at two different instants, it is possible to integrate the different sets of sparse depth values, thus increasing the density of values obtained.

[0171] It should be noted that, by integrating the aforesaid single image-capture sensor 102 with the aforesaid depth sensor 104 in a single sensor 102-104, the association between the sparse depth values Pi, ..., PN and the respective pixels Imx(m, y) of the images Imxacquired by the aforesaid single sensor 102-104 is known, in particular, is known for each of the images Imx, if the sparse depth values Pi, ..., PN are acquired for each position, or else is known for at least one image Imx, if the aforesaid sparse depth values Pi, . . ., PN are acquired just once.

[0172] The one or more processing units 106 are hence configured to execute a virtual -projection procedure VPP, which consists in associating each of the sparse depth values Pi, . . ., PN of the scene 100 to one or more pixels Imx(m, y) that belong to different images Imx.

[0173] For instance, a depth value Pi corresponding to a given point PTh of the aforesaid scene 100 is associated to:

[0174] - a first pixel Imi(m, y) belonging to a first image Imi, the co-ordinates (m, y) of which are expressed according to a reference system fixed with respect to the aforesaid first image Imi obtained via the aforesaid single image-capture sensor 102 integrated in the aforesaid single sensor 102-104 positioned in a first position; and

[0175] - a second pixel Inkin’, y’) belonging to a second image Inu, the co- ordinates (m’, y’) of which are expressed according to a reference system fixed with respect to the aforesaid second image hm obtained via the same single image- capture sensor 102 integrated in the single sensor 102-104 positioned in a second position, for example, a position reached following upon displacement of the aforesaid single sensor 102-104, for example a horizontal displacement, a vertical displacement, or a displacement in any other direction, with respect to the position of acquisition of the first image Imi, where the aforesaid first and second pixels Imi(m, y), Inkin’, y’) correspond to the same point PTh of the scene 100.

[0176] If the sparse depth values Pi, . . PN are acquired for each position, i.e., for each of the images Imx, the aforesaid operation of association can be performed given that the single image-capture sensor 102 is integrated with the depth sensor 104 so as to constitute a single sensor 102-104, and hence a correspondence between the sparse depth values Pi, . . . , PN detected for each position and the respective pixels Imx(m, y) of the images Imxacquired by the aforesaid single sensor 102-104 positioned in the respective positions is known.

[0177] If the aforesaid sparse depth values Pi, . . . , PN are acquired, for example, just once, for at least one image Imi, the aforesaid operation of association can in any case be carried out if there are known the correspondence between the sparse depth values Pi, . . . , PN and the at least one image Imi and the positions, for example even an estimate of these positions, from which the other images Imxhave been acquired with respect to the position from which the at least one image Imi has been acquired.

[0178] A method provided by way of example for making the aforesaid association between depth values and corresponding pixels will be described in the sequel with reference to Figure 3.

[0179] It should be noted that also in this case a sparse depth value Pi associated to a corresponding pixel Imx(m, y) will be denoted by the reference Pi(m, y), where (m, y) are the co-ordinates of the aforesaid pixel Imx(m, y) with respect to the respective image Imxacquired via an image-capture sensor 102.

[0180] It should be noted that the conclusions regarding the operation of association discussed with reference to Figure 1 apply also in this case and will not be repeated again so as not to burden the present description.

[0181] Also in this case, the virtual -projection procedure VPP further comprises an operation of projection of the aforesaid depth values Pi, ..., PN on the aforesaid images Imx, in particular on the corresponding pixels Imx(m, y) associated to the aforesaid depth values Pi, . . . , PN, SO as to generate a set of two-dimensional images ImRxwith projected patterns, for example, virtual patterns, i.e., images containing a plurality of image elements. It should be noted that the aforesaid operation of projection consists in executing the operations described previously with reference to Figure 1, considering real pixels Imx(m, y) instead of virtual pixels Imvx(m, y) so as to achieve the same advantages as those described previously.

[0182] It should be noted that the conclusions regarding the operation of overlaying discussed with reference to Figure 1 apply also in this case and will not be repeated again so as not to burden the present description.

[0183] The processing unit 106 is hence configured for receiving the aforesaid plurality of sparse depth values Pi, . . ., PN and the aforesaid plurality of images Imxand supplying at output one or more images ImRx, where the subscript x has the same meaning as it did previously, which contain projected patterns, for example virtual patterns, where each image ImRxat output corresponds to a respective image Imxat input, but unlike the latter contains in addition information obtained as a function of the aforesaid depth values Pi, . . . , PN, i.e., a plurality of image elements chosen as a function of the depth values.

[0184] In various embodiments, the aforesaid single sensor 102-104 can be kept fixed in a given position from which it acquires the plurality of sparse depth values Pi, . . . , PN and at least one image Imi from the pre-set viewpoint.

[0185] In this case, to use the aforesaid acquisitions for solving the problem of correspondence, for example via stereo matching, the one or more processing units 106 are configured for simulating at least one virtual optical device VODi configured for acquiring at least one virtual image Imvi from a viewpoint different from that from which the aforesaid at least one image Imi is acquired.

[0186] It should be noted that, in embodiments like the one just described, the scene 100 does not necessarily have to be static given that the aforesaid single sensor 102- 104 acquires images from a single viewpoint and the images from other viewpoints are virtual images (i.e., the images from different viewpoints are not acquired at different instants and are independent of the scene 100).

[0187] In the above embodiments, the virtual -projection procedure VPP is able to associate the sparse depth values Pi, . . ., PN to the pixels of the at least one image Imi, given that acquisition of the aforesaid elements is made by the single sensor 102-104, and to associate the aforesaid sparse depth values Pi, . . . , PN to the at least one virtual image Imvi, given that the position of the at least one virtual optical device VODi is known.

[0188] Also in this case the association of the aforesaid sparse depth values Pi, ... , PN to the at least one virtual image Imvi can be made via the method provided by way of example described in the sequel with reference to Figure 3.

[0189] Hence, the virtual -projection procedure VPP further comprises an operation of projection of the aforesaid depth values Pi, . . . , PN both on the at least one image Imi and on the at least one virtual image Imvi, in particular on the corresponding pixels, whether real or virtual, associated to the aforesaid depth values Pi, . . ., PN.

[0190] The aforesaid operation of projection consists in carrying out the operations described previously with reference to Figure 1, considering both real pixels Imx(m, y) and virtual pixels Imvx(m, y), for example black pixels, so as to achieve the same advantages described previously.

[0191] Consequently, the processing unit 106 is configured for receiving the aforesaid plurality of sparse depth values Pi, . . ., PN, at least one image Imi, and at least one virtual image Imvi and to supply at output one or more images ImRxwith projected patterns, for example virtual patterns, in which each image ImRxat output corresponds either to the at least one image Imi at input or to the at least one virtual image Imvi at input, but, unlike these input images, contains, in addition, information obtained as a function of the aforesaid depth values Pi, ..., PN, i.e., a plurality of image elements chosen as a function of the depth values.

[0192] It should be noted that, in the above embodiments, solution of the problem of correspondence and subsequent stereo matching are performed in the presence of context, i.e., with acquisition of at least one real image.

[0193] It should moreover be noted that, in the presence of context, it is possible to reconstruct an image that represents the scene 100 instead of a virtual image, for example a black image, thanks to the association between the sparse depth values Pi, . . . , PN and the pixels of the at least one image Imi and of the at least one virtual image Imvi. In fact, the aforesaid reconstruction can be obtained by replacing the black content of a plurality of virtual pixels Imvx(m, y) contained in a neighbourhood of a given virtual pixel Imvi(m, y) associated to a given depth value Pi with the content, that represents the scene 100, of a plurality of pixels Imx(m, y) contained in a neighbourhood, for example of the same dimensions as the neighbourhood used for the virtual image, of a given pixel Imi(m, y) associated to the same depth value Pi.

[0194] Figure 3 is an image-processing apparatus IO3 that is able to add image elements to images Imxobtained via one or more optical devices 102 according to embodiments of the solutions described.

[0195] Unlike the embodiments illustrated in Figure 2, the embodiments that are based upon structures like the one exemplified in Figure 3 do not integrate the image-capture sensors 102 with the depth sensors 104 so that, to obtain a correspondence between sparse depth values Pi, ... , PN detected and the pixels Imx(m, y) of the images Imx, it is necessary to carry out a preliminary operation of calibration with respect to depth (described hereinafter) on the aforesaid image- capture sensors 102 and depth sensors 104.

[0196] Represented in Figure 3 is a scene 100 of which a digital representation is to be obtained. This scene 100 is acquired via the image-processing apparatus 103, and in particular by a set of image-capture sensors 102, in particular passive sensors, i.e., one or more optical devices, for example video cameras, photographic cameras, and / or any other similar device, and by a set of depth sensors 104, i.e., one or more sensors, in particular active ones, that are able to detect sparse depth values, for example a LiDAR sensor.

[0197] The aforesaid set of image-capture sensors 102 is configured for acquiring a plurality of two-dimensional images Imxof the aforesaid scene 100, where the subscript x indicates a number associated to a corresponding image; for example, if the set of image-capture sensors 102 acquires two images from two different viewpoints, a first acquired image is denoted by Imi, while a second acquired image is denoted by Irro. Moreover, the two-dimensional images Imxin the aforesaid plurality of images are acquired from different viewpoints, for example viewpoints separated horizontally or vertically by a given distance.

[0198] The aforesaid set of image-capture sensors 102 may correspond, for example, to optical devices that implement a stereoscopic-vision system or a multiview stereoscopic system, as defined previously.

[0199] It should be noted that, if the aforesaid set of image-capture sensors 102 contains a single element, this element is configured for acquiring the plurality of images from different viewpoints at different instants and following upon displacements thereof into given positions.

[0200] For instance, if the aforesaid single element 102 is configured for acquiring two images Imx, a first image Imi is acquired in a first position at a first instant t, then the aforesaid single element 102 is displaced from the first position to a second position, where a second image Irm is acquired at a second instant t + 1 (consequently, the scene 100 is a static scene, i.e., one that remains the same at the first instant t and at the second instant t + 1).

[0201] Likewise, it is possible to use a single element 102 for acquiring more than two images Imx.

[0202] The set of the image-capture sensors 102 is hence configured for sending the plurality of two-dimensional images Imxacquired from one or more processing units 106, which may be either local or remote, for example a control logic, a microcontroller, a microprocessor, a personal computer, a computer system, including a distributed system, for example cloud distributed system, or any other similar unit.

[0203] The set of depth sensors 104, in particular a set containing at least one active sensor, for example, a LiDAR sensor, a radar sensor, a ToF sensor, or the like, is configured for acquiring a plurality of sparse depth values Pi, ..., PN that correspond to the aforesaid scene 100, where N denotes the total number of the depth values detected by the set of depth sensors 104, and sending the aforesaid plurality of sparse depth values Pi, . . ., PN to the one or more processing units 106. In various embodiments, the aforesaid set of depth sensors 104 configured for acquiring a plurality of sparse depth values Pi, ..., PN may be a non-active sensor. In particular, the aforesaid sparse depth values Pi, ..., PN may even be obtained via other types of systems, for example from images, e.g., via SLAM, SfM, or the like.

[0204] The one or more processing units 106 are hence configured to execute a virtual -projection procedure VPP that consists in associating each of the sparse depth values Pi, . . ., PN of the scene 100 to one or more pixels Imx(m, y) belonging to different images Imx.

[0205] To carry out the aforesaid operation of association, the image-capture sensors 102 and the depth sensors 104 have to be initially calibrated with respect to depth.

[0206] The above calibration with respect to depth makes it possible to associate each depth value detected by the aforesaid one or more depth sensors 104, in particular active sensors such as a LiDAR in the example, to a corresponding pixel of one of the images of the plurality of images Imxacquired by the aforesaid one or more image-capture sensors 102, a stereoscopic video camera in the example described.

[0207] It may here be noted that the aforesaid calibration with respect to depth is specific, as has been said, to association of the depth values to the pixels of the acquired images. Hence it does not correspond, for example, to calibration per se that is applied in passive sensors such as a stereoscopic video camera to establish parameters thereof, for example the so-called metrical calibration or affine calibration. The calibration described herein applies in general to passive sensors, for example video cameras, which are in themselves already calibrated in some other way, but not with respect to depth, and supply acquired images that are indeed calibrated, but not with respect to depth as defined above.

[0208] For instance, in a system in which the set of the image-capture sensors 102 is configured for detecting only two images Imi and Irm and in which a given image Imxis considered as a matrix of pixels, for example of dimensions M x Y, where M is the total number of the columns of pixels and Y is the total number of the rows of pixels, a given point PTh of a scene 100 may correspond to a first pixel in the first image Imi, for example a pixel that is located in a column m and in a row y and is denoted by Imx(m, y), and to a second pixel in the second image I rm, for example a pixel that is located in a column m’ and in a row y’ and is denoted by Imx(m’, y’), given that the aforesaid images Imi and Irm are obtained from two different viewpoints.

[0209] In this case, if a given depth value, for example the i-th depth value Pi, is associated to the same given point PTh of the scene 100, an example of calibration with respect to depth may consist in associating the aforesaid depth value to the first pixel of the first image Imi and to the second pixel of the second image Inu; hence this depth value may be expressed as Pi(m, y), i.e., with the same co-ordinates as those of the first pixel with respect to the first image Imi, and as Pi(m’, y’), i.e., with the same co-ordinates as those of the second pixel with respect to the second image Inu.

[0210] For instance, in a system in which the set of the image-capture sensors 102 is configured for detecting only two images Imi and Im 2 and if the second image Inu is acquired via a second optical device positioned at a certain distance along a horizontal axis and at the same height as a first optical device used to acquire the first image Imi or if one and the same optical device is configured for acquiring at a first instant the first image Imi and at a second instant, after a movement along a horizontal axis, the second image Inu, a given point PTh of a scene 100 will share one and the same row value in the two images, but will be distinguished by a different column value; i.e., the aforesaid point PTh can correspond to a first pixel in the first image Imx(m, y) and to a second pixel in the second image Imx(m’, y) that is characterized by the same row value y as the first pixel.

[0211] In this case, calibration with respect to depth of the aforesaid set of image- capture sensors 102 and the aforesaid set of depth sensors 104 can be carried out with respect to a single image, for example the first image Imi, and a given depth value Pi, associated to the same given point PTh of the scene 100, can be associated exclusively to the first pixel Imi(m, y) of the first image Imi and expressed as Pi(m, y). It should be noted that the above method of calibration with respect to depth may be readily extended, following the principles described previously, to a plurality of optical devices arranged in one and the same horizontal plane.

[0212] Likewise, for example, it is possible to calibrate with respect to depth a system where the set of the image-capture sensors 102 is configured for detecting only two images Imi and Irm and where the second image Irm is acquired via a second optical device positioned at a certain distance along a vertical axis from a first optical device used to acquire the first image Imi or else if one and the same optical device is configured for acquiring at a first instant the first image Imi and at a second instant, after a movement along a vertical axis, the second image Imi.

[0213] It should be noted that, also in this case, the above method of calibration with respect to depth may be readily extended to a plurality of optical devices arranged in one and the same vertical plane.

[0214] Likewise, following methodologies similar to those employed for calibration with respect to depth of optical devices in one and the same horizontal or vertical plane, a calibration with respect to depth could be made for one or more optical devices arranged in any plane.

[0215] It should be noted that the above examples of calibration with respect to depth may be readily extended, following the same principles described above, also to systems with a single image Imi or to systems with a plurality of images Imx.

[0216] As described previously, the aforesaid plurality of sparse depth values Pi, ..., PN is much less dense than the resolutions of the two-dimensional images Imxobtained so that the processing unit 106 implements a virtual -projection procedure VPP that consists in associating each of the sparse depth values Pi, . . . , PN of the scene 100 to a corresponding pixel of one of the two-dimensional images Imxof the scene 100.

[0217] In various embodiments, if the aforesaid set of image-capture sensors 102 and the aforesaid set of depth sensors 104 have initially been calibrated with respect to depth, the aforesaid association between each of the sparse depth values Pi, . . . , PN of the scene 100 and a corresponding pixel of one of the two-dimensional images Imxof the scene 100 may be made, for example, by associating the depth values Pi, . . PN to the corresponding pixels, i.e., to the pixels that share the same coordinates, for example the co-ordinates (m, y) or (m’, y’).

[0218] In various embodiments, if the aforesaid set of image-capture sensors 102 and the aforesaid set of depth sensors 104 have initially been calibrated with respect to depth and if a given point PTh of a scene 100 shares one and the same row value in the plurality of images Imx:

[0219] - the association between each of the sparse depth values Pi, ..., PN of the scene 100 and a first image Imi, i.e., the image with respect to which the aforesaid depth values Pi, ..., PN are calibrated with respect to depth, may be made, for example, by associating the aforesaid depth values Pi, . . ., PN to the corresponding pixels, i.e., to the pixels that share the same coordinates, for example the co- ordinates (m, y); and

[0220] - the association between each of the sparse depth values Pi, ..., PN of the scene 100 and any image Imxother than the first image Imi may be made by following the steps below for each image other than the aforesaid first image Imi (it should be noted that the aforesaid method may be used also for association of depth values to corresponding pixels in the embodiments of Figure 1 and Figure 2, where association between the depth values and the pixels of a first image is not made via calibration with respect to depth but is known given that at least one image-capture 102 - or at least one virtual optical device VOD - is fixed with respect to, or in the same position as, the depth sensor 104): transforming the depth values Pi(m, y) into disparity values di(m, y) by applying the following formula: where / is a focal distance of the corresponding optical device used to obtain the respective considered image Imx, and b is a base, i.e., a distance measured following a straight line between the optical device used for acquiring the first image Imi and the corresponding optical device used for acquiring the respective image Imxconsidered; obtaining, for each depth value Pi(m, y), a respective column value m’ using the aforesaid disparity value di(m, y) by applying the following formula: given that the aforesaid disparity value di(m, y) represents the offset along the horizontal axis necessary for obtaining the position of the respective considered image Imxstarting from the position of the first image Imi, and hence also the offset along the horizontal axis necessary for obtaining the position of a pixel Imx(m’, y) associated to the respective considered image Imxstarting from the position of a pixel Imi(m, y) associated to the first image Imi; obtaining the depth values Pi(m’, y) expressed according to the reference system of the respective considered image Imxby replacing the column values m with the new values m’ computed in the previous step. It should be noted that, if one of the coordinates, for example m, m’ or y, obtained, for example, via the aforesaid formula that uses the disparity values di(m, y), is not an integer value, then the corresponding depth value, for example, Pi(m’, y), is associated to all the surrounding pixels obtained by rounding off, both by defect and by excess, the aforesaid non-integer value.

[0221] For instance, if the co-ordinate m’ obtained is not an integer value, the corresponding depth value Pi(m’, y) will be applied both to the previous pixel Imx(|m’J, y), i.e., by obtaining a first depth value Pi([m’J, y) associated to the previous pixel Imx(|m’J, y), and to the subsequent pixel Imx([m’], y), i.e., by obtaining a second depth value Pi([m’], y) equal to the first depth value Pi(|m’J, y) and to the original depth value Pi(m’, y) associated to the subsequent pixel Imx(|m’], y).

[0222] Hence, if the aforesaid set of image-capture sensors 102 and the aforesaid set of depth sensors 104 have initially been calibrated with respect to depth and if a given point of a scene 100 shares one and the same row value in the plurality of images Imx, i.e., if the optical devices are arranged along one and the same horizontal plane, it is possible to determine the association between each of the sparse depth values Pi, . . ., PN of the scene 100 and all the images Imxacquired by one or more optical devices by merely knowing the association between the aforesaid depth values Pi, . . ., PN and the pixels of one of the images Imx, i.e., thus calibrating with respect to depth just one of the aforesaid one or more optical devices or calibrating with respect to depth the single optical device present in one of the possible positions of acquisition.

[0223] Likewise, in various embodiments, it is possible to determine the association between each of the sparse depth values Pi, ... , PN of the scene 100 and all the images Imxacquired by the aforesaid set of image-capture sensors 102, i.e., via one or more optical devices, if the aforesaid set of image-capture sensors 102 and the aforesaid set of depth sensors 104 have initially been calibrated with respect to depth and if a given point of a scene 100 shares one and the same column value in the plurality of images Imx, i.e., if the optical devices are arranged apart from another in one and the same vertical plane.

[0224] In this case, a disparity value di(m, y) represents the offset along the vertical axis necessary for obtaining the position of a respective image Imxstarting from the position of a first image, for example, the image Imi, and the parameter m’ corresponds to a respective row value.

[0225] Likewise, in various embodiments, it is possible to determine the association between each of the sparse depth values Pi, ... , PN of the scene 100 and all the images Imxobtained by one or more optical devices if the aforesaid set of image- capture sensors 102 and the aforesaid set of depth sensors 104 have initially been calibrated with respect to depth and if the aforesaid optical devices are arranged apart from one another in any plane.

[0226] In this case, a disparity value di(m, y) represents the offset along an axis corresponding to the above any plane necessary for obtaining the position of a respective image Imxstarting from the position of a first image, for example the image Imi. It should be noted that also in this case the conclusions regarding the operation of association discussed with reference to Figure 1 apply also in this case and will not be repeated again so as not to burden the present description.

[0227] Moreover, also in this case the virtual -projection procedure VPP further comprises an operation of projection of the aforesaid depth values Pi, . . ., PN on the aforesaid images Imx, in particular on the corresponding pixels Imx(m, y) associated to the aforesaid depth values Pi, . . ., PN, SO as to generate a set of two-dimensional images ImRxwith projected patterns, for example virtual patterns, i.e., images containing a plurality of image elements.

[0228] It should be noted that the aforesaid operation of projection consists in carrying out the operations described previously with reference to Figure 1, considering real pixels Imx(m, y) instead of virtual pixels Imvx(m, y), so as to achieve the same advantages described previously.

[0229] It should be noted that also in this case the conclusions regarding the operation of association discussed with reference to Figure 1 apply also in this case and will not be repeated again so as not to burden the present description.

[0230] The processing unit 106 is hence configured for receiving the aforesaid plurality of images Imxand the aforesaid plurality of sparse depth values Pi, . . ., PN and supplying at output one or more images ImRxthat contain projected patterns, for example virtual patterns, the subscript x having the same meaning as it did previously, where each image ImRxat output corresponds to a respective image Imxat input, but unlike the latter contains, in addition, information obtained as a function of the aforesaid depth values Pi, . . . , PN, i.e., a plurality of image elements chosen as a function of the depth values Pi, . . . , PN.

[0231] In variant embodiments, the aforesaid virtual -projection procedure VPP may be repeated for a number of times (space-time stereo VPP) equal to N so as to obtain a plurality of images Imx, in particular a number N of images Imx, which represent the same scene and contain different image elements corresponding to the aforesaid sparse depth values Pi, . . ., PN. For instance, each image in the aforesaid plurality of N images Imxmay contain a plurality of image elements that are obtained as a function of a subset of the sparse depth values Pi, . . . , PN that differs from one image to another.

[0232] In particular, it is possible to obtain a different subset of the sparse depth values Pi, . . . , PN for each image Imxacquired by selecting the aforesaid subset from the aforesaid sparse depth values Pi, ..., PN, for example via sampling, such as random sampling, of part of the aforesaid sparse depth values Pi, . . ., PN, where the percentage of sparse depth values Pi, ..., PN that is sampled for each image may also be selected in a variable way; for instance, all of the values may be considered for a first image, 95% of the total number of values may be randomly sampled for a second image, and so on.

[0233] Hence, one and the same image representing a scene 100 or a plurality of images representing one and the same scene 100 is / are used so as to generate multiple images of the same scene 100 that contain different image elements and are obtained starting from the aforesaid image / images of the scene 100 and from the sparse depth values Pi, . . ., PN, which are acquired just once and are sampled in a different way for each image.

[0234] The multiple images with different image elements thus generated may, for example, be used to integrate intermediate results (e.g., using any aggregation technique, such as an average, or else by integrating the aforesaid results over the volume of the costs or in the maps) or generate multiple disparity images according to various methodologies to obtain more robust results.

[0235] It should be noted that, if just one starting image representing a scene 100 is used for obtaining the multiple images, it is possible to apply the aforesaid repeated virtual -projection procedure VPP also to dynamic scenes.

[0236] In variant embodiments, at least one image in the aforesaid plurality of images is not acquired by the set of image-capture sensors 102, but by at least one virtual optical device VODi configured for acquiring at least one virtual image Imvi from a viewpoint different from those of the aforesaid image-capture sensors 102.

[0237] Operation of the above embodiment is similar to that of the variant embodiment of Figure 2, where the single sensor 102-104 is fixed, but the virtual- projection procedure VPP is able to associate the sparse depth values Pi, . . PN to the pixels of the images obtained via image-capture sensors 102 following the method described previously that makes use of calibration with respect to depth, and association of the sparse depth values Pi, . . . , PN to the at least one virtual image Imvi is accordingly made using the same method.

[0238] Also these variant embodiments solve the problem of correspondence, for example via stereo matching, in the presence of context.

[0239] In the embodiments described previously, it may be noted that, using an active and / or passive sensor 104 that is able to measure depth values, for example instead of a physical projector or a physical light source, it is possible to use the apparatus 10i, IO2, or 103 for acquiring images that contain image elements at any distance, i.e., both at a long range and at a short range, and in any environmental condition, for example even in the presence of a strong light intensity or in daylight, and in both indoor and outdoor environments, without the need for specific training for such situations, thus favouring resolution of the problem described previously regarding the change of context, which arises when using any known type of system comprising a neural network or the like for processing the images.

[0240] Furthermore, given that the aforesaid apparatuses 10i, IO2, or 103, to which reference will be made in what follows as a whole by the reference 10, do not use physical projectors or physical light sources, they are independent of the movement of the elements in the scene 100 considered and of the movement in three- dimensional space of the image-capture sensors 102 (the so-called ego-motion). It should be noted that the aforesaid statement may not apply in the cases where the different images are acquired at different instants.

[0241] The proposed solution can provide far more accurate results as compared to the solutions used in the prior art that, by way of non-limiting example, use sensors such as to acquire depth values that are so sparse that they correspond to approximately one percent of an entire image.

[0242] Hence, in general, the solution described herein regards an image- processing apparatus 10 comprising: - at least one module, for example the image-capture sensors 102 and / or the virtual image-capture sensors (virtual optical devices VOD) simulated by the at least one processing unit 106, configured for acquiring a set of images, for example a set containing virtual images Imvxand / or images acquired via a sensor for capturing real images Imxof a scene 100, and in particular a stereoscopic-vision system or a multiview stereoscopic system;

[0243] - at least one depth sensor 104, in particular an active sensor (possibly, also a non-active sensor), in particular, said depth sensor being of a LiDAR type, or a radar type, or a ToF type, or a SLAM type, configured for detecting sparse depth values Pi, . . ., PN in said scene 100; and

[0244] - at least one processing unit 106, wherein said at least one processing unit 106 is configured for:

[0245] - receiving said sparse depth values Pi, . . . , PN detected by said depth sensor 104;

[0246] - performing at least one virtual -projection procedure VPP that comprises:

[0247] - associating said plurality of sparse depth values Pi, ..., PN of said scene 100 to said set of acquired images Imvx, Imxof said scene 100, said operation of association comprising, for each depth value in said plurality of sparse depth values Pi, . . . , PN detected via said at least one depth sensor 104, associating a considered depth value corresponding to a point PTh of said scene 100 to a respective pixel Imvx(m, y) or Imx(m, y), corresponding to the same point PTh of the scene 100, of each of said images in said set of acquired images Imvxand / or Imxof said scene 100, said virtual -projection procedure VPP comprising generating a set of two- dimensional images ImRxof said scene 100 with projected patterns, i.e., with a projected plurality of image elements, via an operation of augmenting said acquired images Imvxand / or Imxvia the steps of:

[0248] - identifying a set of corresponding pixels that contains pixels Imvx(m, y) and / or Imx(m, y) belonging to different images in said set of images Imvxand / or Imxto which there has been associated one and the same depth value in said plurality of sparse depth values Pi, . . PN; and

[0249] - overlaying on each pixel Imvx(m, y) and / or Imx(m, y) that is contained in said set of corresponding pixels one and the same image element A chosen from a plurality of image elements.

[0250] Moreover, said plurality of two-dimensional images ImRxis generated to establish a visual correspondence between each two-dimensional image and the other two-dimensional images in said plurality of two-dimensional images ImRx.

[0251] Consequently, in various embodiments, said apparatus 10 is coupled to, or comprises, for example integrated within the apparatus 10, a stereo matcher 208 configured for receiving at input said set of two-dimensional images ImRxwith projected patterns, i.e., with a projected plurality of image elements, from said image-processing apparatus 10, where in particular each two-dimensional image ImRxin said set of two-dimensional images ImRxwith projected patterns corresponds to a respective image in said set of images Imvxand / or Imxacquired by the at least one module 102; 106.

[0252] The solution described hence regards in general also a method for processing images via embodiments of the apparatus 10 just described, said method comprising at least the operations of:

[0253] - acquiring, via at least one module 102 and / or 106, in particular a stereoscopic-vision system or a multiview stereoscopic system, a set of images Imvxand / or Imxof a scene 100;

[0254] - detecting, via the at least one depth sensor 104, in particular an active sensor, in particular, said depth sensor being of a LiDAR type, or a radar type, or a ToF type, or a SLAM type, sparse depth values Pi, . . ., PN of said scene 100; and

[0255] - carrying out, via at least one processing unit 106 configured for receiving said sparse depth values Pi, ..., PN detected by said depth sensor 104, at least one virtual -projection procedure VPP, said virtual -projection procedure VPP comprising:

[0256] - associating, via said at least one processing unit 106, said plurality of sparse depth values Pi, . . ., PN of said scene 100 to said set of acquired images Imvx and / or Imxof said scene 100, said operation of association comprising, for each depth value in said plurality of sparse depth values Pi, ..., PN detected via said at least one depth sensor 104, associating a considered depth value of a point PTh of said scene 100 to a respective pixel Imvx(m, y) or Imx(m, y), which corresponds to the same point PTh of the scene 100, of each of said images in said set of acquired images Imvxand / or Imxof said scene 100; and

[0257] - generating, via said at least one processing unit 106, a set of two- dimensional images ImRxof said scene 100 with projected patterns via an operation of augmenting said acquired images Imvxand / or Imxthrough the steps of:

[0258] - identifying, via said at least one processing unit 106, a set of corresponding pixels that contains pixels Imvx(m, y) and / or Imx(m, y) belonging to different images in said set of images Imvxand / or Imxto which there has been associated one and the same depth value in said plurality of sparse depth values Pi, . . ., PN; and

[0259] - overlaying, via said at least one processing unit 106, on each pixel Imvx(m, y) and / or Imx(m, y) contained in said set of corresponding pixels, one and the same image element A chosen from a plurality of image elements.

[0260] It should be noted that the solution described also regards a computer- program product that can be loaded into the memory of at least one computer and comprises portions of software code for executing the steps of the method described above.

[0261] As described previously, for each embodiment of the present solution the one or more processing units 106 are configured for carrying out a virtual- projection procedure VPP that comprises an operation of projection of the depth values Pi, ..., PN on the images (whether virtual or not), and, in particular, on the corresponding pixels (whether virtual or not) associated to the aforesaid depth values Pi, ..., PN, SO as to generate a set of two-dimensional images ImRxwith projected patterns, for example virtual patterns, i.e., images that contain a plurality of image elements.

[0262] As described previously, the aforesaid operation of projection consists in the selection of an image element EV from a set of image elements for each set containing a depth value Pi and the corresponding pixels (whether virtual or not) associated to the aforesaid depth value Pi, i.e., each pixel (whether virtual or not) associated to one and the same point PTh of one and the same scene 100 that corresponds also to the depth value Pi, and in the replacement of the content of the aforesaid associated pixels (whether virtual or not) with the aforesaid selected image element EV.

[0263] In various embodiments, replacement of the content of pixels (whether virtual or not) with selected image elements corresponds to an operation of augmenting the acquired images via depth values Pi, . . ., PN detected by the set of depth sensors 104 and makes it possible to obtain images ImRxwith projected patterns.

[0264] On the basis of how the aforesaid operation of augmenting is carried out, it is possible to favour further the association between different images, whether real Imxor virtual Imvx, i.e., by obtaining corresponding images ImRxwith projected patterns that can be associated more easily to one another thanks to the presence of a plurality of visual elements that is obtained as a function of the aforesaid depth values Pi, ..., PN.

[0265] The aforesaid operation of augmenting the acquired images via depth values Pi, . . . , PN detected by the set of depth sensors 104 that is able to favour further the association between different images (whether virtual or not) Imxand / or Imvxcomprises associating virtually projected patterns to one or more pixels of corresponding images to obtain images ImRxwith projected virtual patterns that facilitate the aforesaid association, i.e., solution of the problem of correspondence, for example via stereo matching.

[0266] In particular, one and the same image pattern, i.e., one and the same image element, is virtually projected on one or more pixels that have been identified as corresponding within the different images Imxand / or Imvxthanks to a prior association of the aforesaid one or more pixels via the depth values Pi, . . ., PN; i.e., the same image pattern is projected on pixels, and possibly also on the surrounding ones, that belong to different images to which the same depth value Pi, . . ., PN has been previously associated. In fact, knowing which pixels of a first image correspond to given pixels of a second image thanks to the association described previously that exploits the depth values Pi, ..., PN obtained via the set of depth sensors 104, it is possible to increase the similarities between each of the aforesaid pixels of the first image and a corresponding pixel of the second image via projection of an image pattern thereon.

[0267] In this way, the obtained images ImRxwith projected patterns enable even visual association of the aforesaid pixels, and the differences increase between one of the aforesaid pixels, for example a pixel PX belonging to the first image, and the pixels not corresponding to the aforesaid pixel PX, for example all the pixels of the second image except for the pixel corresponding to the aforesaid pixel PX, and also the pixels close to the aforesaid pixel PX, for example, ones that are located in the vicinity of the aforesaid pixel PX in the first image.

[0268] Hence, the images ImRxwith projected virtual patterns, i.e., with a projected plurality of image elements, may be supplied at input to stereo matchers of any known type, such as conventional stereo matchers, for example ones also used in traditional passive stereo systems or in systems that use physical projectors of structured and / or non-structured light.

[0269] It should be noted that, advantageously, embodiments of the present solution like the ones just described can re-use known traditional stereo matchers, i.e., stereo matchers used for associating images obtained via optical devices with or without the use of a physical projector that is able to project light beams on a scene to be captured, without requiring any modification thereof.

[0270] Furthermore, any stereo matcher that may be developed in future, for example with better performance, for applications that use physical projectors or else for applications that do not use projectors of any type, i.e., for passive stereo systems, may be used also for matching images in the embodiments of the present solution like the ones just described.

[0271] It should moreover be noted that the possibility of projecting virtual patterns without having to use a physical projector makes it possible to select, for each group of images corresponding to a given scene, a pattern optimized for the scene considered. In fact, such an optimization is not allowed by applications that contain physical projectors on account of the limitations of the technology used.

[0272] It should be noted that any operation of augmenting considered, i.e., any type of image pattern projected on the images, would achieve, even though to a different extent, the same advantage, i.e., it would favour association between different images Imxand / or Imvx, so that the examples of projected image patterns that are described hereinafter are not to be understood as in any way limiting the sphere of protection of the present disclosure.

[0273] Described in greater detail hereinafter are, by way of example, three operations of augmenting images, i.e., three methodologies used for selection of the image elements that make up a virtual pattern associated to an image ImRxwith projected patterns:

[0274] - a projection of fixed virtual patterns;

[0275] - a projection of random virtual patterns; and

[0276] - a projection of virtual patterns based upon histograms.

[0277] The operations here described, or more in general any other operation of augmenting considered, are performed, as described previously, on corresponding pixels in the images Imxand / or Imvx.

[0278] For instance, in the case of two real images Imi and Inu, the aforesaid operations of augmenting can be performed on a first pixel Imi(m, y) associated to the first image Imi and on a second pixel Im2(m’, y), corresponding to the aforesaid first pixel Imi(m, y), associated to the second image Inu, via replacement of the content of the aforesaid pixels, and possibly, of the surrounding pixels, with one and the same image pattern A, for example, A(m,m’, y).

[0279] It should be noted that, in order not to burden the description, in what follows the images and the corresponding pixels will be considered as real, i.e., acquired via image-capture sensors 102, even though the equivalent conclusions and methods also apply in the case of virtual images and corresponding virtual pixels or in the case of images, whether real or virtual, and the corresponding pixels.

[0280] In particular, in various embodiments the following operations may be envisaged: - an image pattern A, for example A(m,m’, y), is associated to a first pixel in a first image, for example to the first pixel Imi(m, y) that is associated to the first image Imi and corresponds to a given depth value Pi; and

[0281] - the same image pattern A, for example A(m,m’, y), is associated to a second pixel in a second image, for example to the second pixel Im2(m’, y) that is associated to the second image Irm and corresponds to the same given depth value Pi.

[0282] It should be noted that, if one of the coordinates, for example m, m’, or y, obtained for example via the formula previously discussed that uses disparity values di(m, y), is not an integer value, then the image pattern A, for example A(m,m’, y), is applied to all the surrounding pixels in a weighted way.

[0283] For instance, if the co-ordinate m’ obtained is not an integer value, the image pattern A(m,m’, y) will be applied in a weighted way:

[0284] - both to the previous pixel Imx(|m’J, y), i.e., by projecting on the aforesaid previous pixel Imx([m’ J, y) a first image pattern A(m, |m’ J, y), obtained by applying the following formula:

[0285] A(m, |m’J,y) = b / m%(|m’J,y) + (1 — b)A(m, m’, y) where b is the corresponding weight coefficient obtained via the formula b = m’- Lm’J;

[0286] - and to the subsequent pixel Imx([m’], y), i.e., by projecting on the aforesaid subsequent pixel Imx([m’], y) a second image pattern A(m,[m’], y), obtained by applying the following formula:

[0287] A(m, [m’],y) = (1 — 6) / mx([m’],y) + bA(m, m’,y) where b is the same weight coefficient b = m’-|m’J.

[0288] A difference between the various operations of augmenting may consist in how said image pattern A, for example A(m,m’, y), that is overlaid on the images Imxis generated.

[0289] A first way to generate the aforesaid image pattern A is based upon a projection of fixed virtual patterns. Various implementations of this first way can be used in any image-processing apparatus 10 according to any of the embodiments described previously.

[0290] In particular, it is possible to determine a specific fixed virtual pattern, for example according to the application considered, to be applied to the pixels of the corresponding images Imxand / or Imvx, according to any criterion.

[0291] It should be noted that the aforesaid fixed virtual pattern may be sub-optimal in the embodiments that operate in the presence of context, i.e., according to the content of the acquired images, so that alternative methodologies for generating patterns would prove advantageous.

[0292] A second way to generate the aforesaid image pattern A is based upon a projection of random virtual patterns. Various implementations of this second way can be used in any image-processing apparatus 10 according to any of the embodiments described previously.

[0293] The projection of random virtual patterns consists in associating a random image element to corresponding pixels in different images, for example a first projection of random virtual patterns may be a projection of a non-distinctive pattern, i.e., a sub-optimal pattern that consists in associating to pixels corresponding image elements of one and the same tone of one and the same colour, for example of one and the same shade in a grey scale.

[0294] For instance, the aforesaid tone of one and the same colour, for example a shade in the grey scale, may be generated using the following distribution, which in particular is a probability distribution:

[0295] A (m, m’, y)~U(Q, ν) i.e., by sampling, with a uniform probability, values from a uniform probability distribution in the range comprised between 0 and a value v, for example a value less than or equal to 255, i.e., the total number of shades of grey in a grey scale represented on 8 bits, where each integer value in the aforesaid range (between 0 and 255) corresponds to a respective shade of grey, starting from the zero value corresponding to black up to the value 255 corresponding to white. Hence, for each set, for example for each pair, of corresponding pixels in different images, e.g., two images, a random, i.e., equiprobable, value is sampled in the aforesaid uniform probability distribution, and associated to each pixel in the set the aforesaid pixels is one and the same image element corresponding to the aforesaid sampled random value, for example, a shade of grey in the 8-bit grey scale that is associated to the aforesaid sampled random value.

[0296] It should be noted that any other value may be considered for the upper value v of the range, for example considering a different grey scale, e.g., one represented on a different number of bits, or any other form of visual identification that can correspond to a single value, for example even one created ad hoc for a specific situation.

[0297] In various embodiments, the upper value v of the range is set at the maximum value, for example 255. In this way, it is advantageously possible to maximize the similarity between corresponding points and the differences between non-corresponding points given that it is more unlikely for two pixels close to one another and not corresponding to be associated to one and the same value, i.e., to one and the same shade of grey. Hence, in the embodiments just described, the image pattern is generated using the probability distribution A (m, m’, y) ~ [7(0,255) .

[0298] An image pattern of a non-distinctive type, albeit increasing the information contained in the images and thus facilitating achievement of the advantages set forth previously, might not increase sufficiently the similarities between corresponding points and the differences between non-corresponding points given that the aforesaid points are differentiated only on at the most (in the case where the 8-bit grey scale is used) 255 shades of grey that are difficult to distinguish from one another.

[0299] Hence, for example, a second projection of random virtual patterns may be a projection of an image pattern generated by sampling three values from the following probability distribution:

[0300] A (m, m’, y) ~ [7(0, v) i.e., by sampling, with a uniform probability, a triplet of values from a uniform probability distribution in the range comprised between 0 and a value v, for example a value less than or equal to 255, i.e., the number of values that generally define the range of values of one of the channels of the RGB (Red Green Blue) representation of colours, i.e., the channel of red, green, or blue, and by associating to each triplet obtained via the aforesaid probability distribution a distinct image feature, for example a different colour in the RGB colour scale.

[0301] A triplet obtained by sampling three values from the aforesaid probability distribution may be referred to as hexadecimal triplet, and the aforesaid association of the hexadecimal triplet to a distinct image feature, for example to an RGB colour, may be carried out, for example, by getting a first value of the aforesaid triplet to correspond to a red component, a second value of the aforesaid triplet to correspond to a green component, and a third value of the aforesaid triplet to correspond to a blue component and obtaining the colour resulting from the combination of the three components.

[0302] Also in this case, for each set of corresponding pixels in different images a triplet of random values is sampled, for example by sampling each of the values of the triplet independently of the other values of the triplet in the aforesaid uniform probability distribution, and associated to each pixel in the set of the aforesaid pixels is one and the same image element corresponding to the aforesaid triplet of sampled random values, for example a colour obtained from an RGB colour scale associated to the aforesaid sampled random values.

[0303] Likewise, also in this case it may be noted that any other value may be considered for the upper value v in the range, for example considering a different scale of colours or any other form of visual identification that can correspond to a triplet of values, for example even ones created ad hoc for a specific situation.

[0304] In various embodiments, the upper value v of the range is set at the maximum value, for example 255, so as to achieve the same advantages set forth for the previous case. Hence, also in this case the image pattern may be generated via the following probability distribution: Αl(m, m’, y)~[7(0,255). It should be noted that, both in the case of the image pattern of a non- distinctive type and in the case of the image pattern based upon triplets of values, we have an advantageously fast operation, i.e., one based upon the extraction of a random number or triplet of random numbers from a uniform probability distribution and that is able to introduce distinctive features to a certain degree, i.e., by associating according to a random criterion a shade or a colour to corresponding pixels, even if not entirely corresponding; for example, there is a probability, albeit low, of two pixels close to one another in a first image and associated to respective pixels in a second image having the same sampled value or the same sampled triplet of values, thus maintaining as main distinctive feature the respective depth values Pi, . . ., PN associated to the aforesaid pixels.

[0305] Consequently, it would be preferable for the aforesaid values or triplets of values not to be ambiguous, for example, to be unique, at least for pixels that are close to one another and / or are one and the same horizontal and / or vertical line of a given image.

[0306] To guarantee the aforesaid characteristic, it is possible to use, for example, a projection of virtual patterns based upon histograms or else a projection of random virtual patterns that is configured to repeat a sampling operation in the case where the aforesaid sampled value or sampled triplet of values is already present in pixels that are close to one another and / or on the same horizontal and / or vertical line.

[0307] It should be noted that the projection of virtual patterns A based upon histograms may preferably be used in any image-processing apparatus 10 according to any one of the embodiments described previously in the presence of context.

[0308] It should be noted that in embodiments that comprise both image-capture sensors 102 and virtual optical devices VOD, the virtual patterns A based upon histograms may be projected exclusively onto the real images Imxacquired via the aforesaid image-capture sensors 102.

[0309] The projection of virtual patterns A, for example A(m,m’, y), based upon histograms hxis obtained via the analysis of the image elements already present in the images, i.e., the image elements the information of which has already been integrated in an image. Hence, to each set of corresponding pixels belonging to different images, i.e., for a given pixel Imi(m, y) of a first image Imi and for each pixel Imx(m’, y) of other images Imxthat corresponds to the aforesaid pixel Imi(m, y) of the first image Imi, there must be associated one and the same image element EV, for example, a colour or a shade, which will make it possible to facilitate identification of points belonging to one and the same set so as to increase the similarities between corresponding points and the differences between non-corresponding points.

[0310] Consequently, for example, for the aforesaid given pixel Imi(m, y) of the aforesaid first image Imi and for each pixel Imx(m’, y) of the other images Imxthat corresponds to the aforesaid pixel Imi(m, y) of the first image Imi, one or more windows are considered, for example one window for the pixel Imi(m, y) and one window for each other pixel Imx(m’, y).

[0311] The aforesaid windows may have one and the same predefined size or may have a variable size, chosen, for example, on the basis of the scene considered and of the type of application.

[0312] In various embodiments, the aforesaid windows have a height of 3 pixels and a width of L pixels, where L depends upon the width of the images Imxconsidered.

[0313] The aforesaid windows are moreover centred on the respective pixel, i.e., the window corresponding to the pixel Imi(m, y) is centred on the pixel Imi(m, y) and the window corresponding to each other pixel Imx(m’, y) is centred on the respective pixel Imx(m’, y).

[0314] Hence, on each of the aforesaid windows there may be defined a respective histogram hxthat indicates the frequency of occurrence of the possible image elements (e.g., of the elements, or triplets of elements, from 0 to 255) within the respective window.

[0315] The aforesaid histograms hxare then added together to obtain a respective sum histogram Hsthat indicates the frequency of occurrence of the possible image elements within all the windows considered for a given set of corresponding pixels belonging to different images. Hence, the operation that is able to generate the virtual pattern A, for example, A(m,m’, y), chooses, for the aforesaid set of corresponding pixels that belong to different images, a respective image element EV, for example, a shade or a colour, that corresponds to the image element that is able to maximize the distance between the aforesaid chosen image element EV and any other possible image element.

[0316] The information on the aforesaid distances are contained in a histogram hdist, which contains, for each possible image element, an indication on the minimum distance that exists between the aforesaid possible image element and any other image element that is contained in the aforesaid sum histogram Hsand has a value other than zero, i.e., that is present at least once in the set of image elements contained in the windows considered.

[0317] Consequently, the operation that is able to generate the virtual patterns A, for example, A(m,m’, y), chooses, for the aforesaid set of corresponding pixels belonging to different images, a respective image element EV that corresponds to the image element contained in the aforesaid histogram hdist that has the greatest value of minimum distance.

[0318] In particular, the aforesaid histogram hdist may be obtained by applying the following formula for calculating the values of each of its bars: hdistfel) = minp EP| el — p | where:

[0319] - el is one of the possible image elements, for example, any one of the elements, or of the triplets of elements, from 0 to 255;

[0320] - hdist(el) corresponds to a bar of the histogram hdist corresponding to the image element el; i.e., it corresponds to the value indicating the minimum distance between the aforesaid image element el and any other image element contained in the aforesaid sum histogram Hsthat has a value other than zero;

[0321] - P is the set containing all the image elements contained in the aforesaid sum histogram Hsexcept for the image element el and the image elements that have a value in the aforesaid sum histogram Hsequal to zero, i.e., that are not present even once in the set of image elements contained in the windows considered; and

[0322] -p is an index that indicates a p-th element belonging to the aforesaid set P.

[0323] It should be noted that, in the case where all the possible image elements were present in a number other than zero in the sum histogram Hs, i.e., if all the image elements appear at least once in the set of image elements contained in the windows considered, the image element EV selected is the one that has the minimum occurrence, i.e., the one that appears less in the set of image elements contained in the windows considered.

[0324] For instance, if only two images are considered, i.e., a first pixel Imi(m, y) associated to a first image Imi and a second pixel Im2(m’, y) associated to a second image Irm and corresponding to the aforesaid first pixel Imi(m, y), i.e., where the aforesaid first pixel Imi(m, y) and the aforesaid second pixel Im2(m’, y) are characterized by one and the same depth value Pi, the following steps can be followed:

[0325] - considering two respective windows, i.e., a first window Fi and a second window F2, both of height 3 and width L and centred on the aforesaid first pixel Imi(m, y) and on the aforesaid second pixel Im2(m’, y), respectively;

[0326] - defining two respective histograms hi and I12, which indicate the frequency of occurrence of the possible image elements within the first window Fi and the second window F2, respectively;

[0327] - defining a sum histogram Hsas sum of hi and I12, i.e., Hs= hi + I12, indicating the frequency of occurrence of the possible image elements within both of the windows Fi and F2;

[0328] - if there is at least one value in the sum histogram Hsequal to zero, determining, via the formula described previously, a histogram hdist that contains, for each possible image element, the minimum distance between the aforesaid possible image element and any other image element that is contained in the sum histogram Hsand has a value other than zero, and hence choosing for the aforesaid first pixel Imi(m, y) and for the aforesaid second pixel Im2(m’, y) the image element EV that corresponds to the image element contained in the aforesaid histogram hdist that has the largest value of minimum distance; and

[0329] - if there is not at least one value in the sum histogram Hsequal to zero, choosing as image element EV for the aforesaid first pixel Imi(m, y) and for the aforesaid second pixel Im2(m’, y) the image element that has the minimum occurrence in the set of image elements contained in the first window Fi and in the second window F2, i.e., the image element that has the minimum frequency of occurrence in the sum histogram Hs.

[0330] In various embodiments, both the projection of random virtual patterns and the projection of virtual patterns based upon histograms may be used so as to exploit a locality of the images, that is, by applying a given image element not only to a corresponding pixel Imx(m, y), i.e., in point mode, but also to a larger area comprising surrounding pixels so as to increase further the visual correspondence between the two images.

[0331] For instance, in embodiments like the ones just described, both the projection of random virtual patterns and the projection of virtual patterns based upon histograms may be extended so as to operate no longer on a single pixel, but on areas of pixels, for example on a 3 * 3 window of pixels or on a 5 * 5 window of pixels, implicitly assuming the same disparity within one and the same area.

[0332] It should be noted that the aforesaid window of fixed size, for example of a single pixel, of 3 * 3 pixels, of 5 * 5 pixels, or some other size, may be used in any image-processing apparatus 10 according to any one of the embodiments described previously.

[0333] In various embodiments, both the projection of random virtual patterns and the projection of virtual patterns based upon histograms may operate, as described previously, on areas of pixels, for example, on a 3 * 3 window of pixels or on a 5 x 5 window of pixels, but associating to each of the pixels comprised in the area selected, for example to each of the 9 pixels of a 3 x 3 window, a different image element sampled according to the methodologies described previously.

[0334] In various embodiments, if a considered stereo matcher presents limitations in processing images that contain projected virtual patterns, it is possible to combine the content of an original image with a corresponding virtual pattern projected via the alpha-blending technique, i.e., by applying the following formulas:

[0335] / m((m,y) = (1 — <z) / m((m,y) + aA(m,m',y')

[0336] Imr(m',y) = (1 — <z) / mr(m',y) + aA(m,m’,y) where a is a hyperparameter.

[0337] In this way, it is possible to use the embodiments described also with stereo matchers that are not able to process images that contain projected virtual patterns.

[0338] In various embodiments, alpha-blending may be applied as a function of a distance, for example as a function of a depth detected for the aforesaid pixel Imx(m, y).

[0339] In particular, alpha-blending as a function of the distance may be used in any image-processing apparatus 10 according to any one of the embodiments described previously.

[0340] For instance, given that the set of depth sensors 104 is already configured for acquiring a plurality of sparse depth values Pi, ..., PN and given that the aforesaid sparse depth values Pi, . . ., PN are associated, via calibration with respect to depth, to respective pixels Imx(m, y), for example obtaining depth values associated to the co-ordinates of the pixels of a given image Pi(m, y), it is possible to attenuate alpha-blending as a function of the distance, i.e., of the depth, so as to compensate for the perspective effect.

[0341] In particular, the further away an object is, the more alpha-blending is attenuated, i.e., the patterns obtained via association of the aforesaid sparse depth values Pi, . . . , PN to the aforesaid images are less and less detectable when compared to the images themselves as the distance increases.

[0342] The aforesaid attenuation can be carried out using different strategies, for example various strategies of mapping between sparse depth values Pi, . . ., PN and values of width of a corresponding window, for example either in a linear way or in a nonlinear way, using the sparse depth values Pi, . . ., PN for estimating a metric range. In various embodiments, the at least one processing unit 106 is further configured for implementing an algorithm of a heuristic type that is able to improve the results obtained in the presence of diversities between the occluded regions, i.e., regions not directly visible in an image captured from a given viewpoint, in the images Imxconsidered.

[0343] In fact, in the case of diversity between the occluded regions in the images Imx, the image pattern projected on them might not be consistent for all the pixels considered.

[0344] For instance, the set of depth sensors 104 may measure, for a given point PTh in a given scene 100, a corresponding depth value Pi, which may, for example, be consistent with a first image Imi; i.e., the point PTh is visible in the first image Imi, thus rendering the association between Imi(m, y) and Pi(m, y) correct, but not consistent with a second image Inu, for example because the aforesaid point PTh is occluded, i.e., not visible, in the aforesaid second image Irm and hence the association between Im2(m’, y) and Pi(m’, y) is wrong.

[0345] Hence, in the case of diversity between the occluded regions in the images Imx, it becomes important to determine which of the points PTh in a given scene 100 correspond to regions differently occluded in the images Imx. In this way, it is possible to prevent one and the same pattern from being projected both on the occluded pixel, for example belonging to a first image Imi, and on the occluding pixel, for example belonging to a second image IITO.

[0346] The algorithm of a heuristic type for improving the results obtained in the presence of diversity between the occluded regions is able to classify the disparities di(m, y) computed for respective sparse depth points Pi(m, y) projected on a given image Imxon the basis of differences in the disparity values di(m, y) and of the distance between a given disparity value di(m, y), i.e., of the distance of a corresponding depth point Pi(m, y), and the other disparity values di(m, y), i.e., other sparse depth points Pi(m, y) acquired by the set of depth sensors 104.

[0347] The aforesaid algorithm comprises the following steps:

[0348] - computing disparity values di(m, y) by applying the formula described previously:

[0349] - mapping the aforesaid disparity values di(m, y) on a grid W, for example in a position (j, k), where j is a co-ordinate indicating one of the columns of the grid W, and k is a co-ordinate indicating one of the rows of the grid W, and, in the case of collisions, i.e., of multiple disparity values di(m, y) that should correspond to one and the same location in the grid W, for example, to the same location (j, k), the disparity value di(m, y) with highest value is stored;

[0350] - classifying a cell W(j, k) of the aforesaid grid W as occluded if the following inequality is satisfied for at least one cell W(j’, k’), with j’ and k’ ranging in a window of size q x rk around the cell W(j, k): where A, y, q, rk, and t are hyperparameters that can be set, respectively, at, for example, 2, 0.4375, 9, 7, and 1; and

[0351] - using the aforesaid information to obtain an occlusion mask to be used for correcting the results.

[0352] In various embodiments, it is possible to define the size of a window around a given pixel Imx(m, y) as a function of a distance, for example, as a function of a depth detected for the aforesaid pixel Imx(m, y).

[0353] In particular, the size of the windows as a function of the distance may be used in any image-processing apparatus 10 according to any one of the embodiments described previously.

[0354] For instance, given that the set of depth sensors 104 is already configured for acquiring a plurality of sparse depth values Pi, ..., PN and since the aforesaid sparse depth values Pi, . . ., PN are associated, via calibration with respect to depth, to respective pixels Imx(m, y), for example obtaining depth values associated to the co-ordinates of the pixels of a given image Pi(m, y), it is possible to reduce the size of a window around a given pixel Imx(m, y) as the distance, i.e., the depth, increases so as to compensate for the perspective effect. In fact, in the acquired images one and the same object positioned further away is smaller than the same object when this is positioned at a shorter distance, and consequently the aforesaid object positioned at a greater distance occupies a smaller area, i.e., a smaller number of pixels.

[0355] Hence, at greater distances the pixels of a given image cover a real area, for example an area of the corresponding real object, that is larger.

[0356] The aforesaid reduction of the size of a window around a given pixel Imx(m, y), as the distance increases can be obtained using various strategies, for example various strategies of mapping between sparse depth values Pi, . . ., PN and values of size of a corresponding window, for example either in a linear way or in a nonlinear way, using the sparse depth values Pi, ..., PN for estimating a metric range.

[0357] In various embodiments, it is possible to define the size and shape of a window around a given pixel Imx(m, y) of an image as a function of a context that is detected within the aforesaid image.

[0358] In particular, the size of the windows as a function of the context may be used in any image-processing apparatus 10 according to any one of the embodiments described previously except for the embodiments characterized by a structure like the one illustrated in Figure 1 on account of the absence of context.

[0359] It should be noted that, in embodiments that comprise both image-capture sensors 102 and virtual optical devices VOD, the size of the windows as a function of the context may be used exclusively on the real images Imxacquired via the aforesaid image-capture sensors 102.

[0360] In these embodiments, not only the size, but also the shape of a window can be adapted as a function of a context detected within an image, i.e., as a function of the content of an image.

[0361] Hence, a window can assume different shapes, for example a polygonal shape or a shape of some other sort and not only square so as to adapt to the contents of an image, for example, by including the points that are close to one another and linked by similar photometric properties. In a first embodiment, for example, it is possible to exploit the spatial and chromatic information, for example information regarding a photometric similarity, of a pair of stereo images or of a single image. A possible example of application could be a window applied to an image that depicts a road sign, which is enlarged according to the shape, for example, triangular, square, circular, etc., of the aforesaid road sign.

[0362] In a second embodiment, for example, a given window may not be square, but, for example, rectangular (i.e., cropped) or of any other shape, for example circular, polygonal, etc., according to the shape and colour of an object associated to which is a sparse depth value corresponding to the aforesaid given window.

[0363] The inequality described previously is defined so as to identify the occluded points, i.e., the points that are not visible in all the images.

[0364] In particular, the aforesaid inequality is used for identifying cells that are in very close positions in corresponding images, for example, within one and the same neighbourhood, i.e., within a circumference of a given radius that is centred on one of the cells under analysis, that are, however, characterized by very different disparity values, for example ones higher than a given threshold.

[0365] Hence, for each cell under analysis, if there exists at least one other cell in a neighbourhood centred on the under analysis that has a disparity value very different from the aforesaid cell under analysis, this cell under analysis is classified as being occluded.

[0366] In particular, each considered cell of the aforesaid grid is classified as occluded if there exists at least one other cell in a neighbourhood centred on the aforesaid considered cell, i.e., a range of a given length centred on the aforesaid considered cell, that has a disparity value that differs from a disparity value corresponding to the aforesaid considered cell by a value higher than a given threshold, for example a threshold defined on the basis of the application.

[0367] In fact, the aforesaid inequality is defined for identifying the occluded cells by considering both the distance between different cells and the respective disparity values associated to them; for example, if a first cell W(j, k) is characterized by a disparity value smaller than that of a second cell W(j’, k’), i.e., if W(j k') - W(j, k) > t, but the distance between the first cell W(j, k) and the second cell W(j’, k’) is great, i.e., k([j’ - j| + (1 - y)|k’ - k|) » 1, then the aforesaid first cell W(j, k) is not considered as occluded given that it is less likely for there to be an occlusion between the two cells.

[0368] It should be noted that any other known method of identification of the occluded points in the images analysed may be used to obtain one and the same technical effect of the inequality described above, i.e., identification of the aforesaid occluded points.

[0369] In various embodiments, the aforesaid correction of the results made via the occlusion mask is performed so as to ignore, for example by not projecting, corresponding image elements on all the images Imxin correspondence of disparity values di(m, y), and hence of corresponding depth points Pi(m, y), considered as occluded.

[0370] In various embodiments, the aforesaid correction of the results obtained via the occlusion mask, in addition to making it possible to avoid projecting the corresponding image elements for the disparity values di(m, y) and for the corresponding depth points Pi(m, y) considered as occluded, is configured for replacing the original content of the occluded pixels contained in some of the images Imxwith the content of corresponding occluding pixels contained in others from among the images Imx.

[0371] It should be noted that the aforesaid replacement does not alter the association between the images Imx, but makes it possible, instead, to stimulate the stereo matcher so that this can establish a further correspondence, namely, a correspondence additional to the one determined by the depth points Pi(m, y), between the occluded pixels and the corresponding occluding pixels.

[0372] In various embodiments, it is possible to use a so-called padding technique, for example, a padding-left technique, so as to prevent occlusions and render possible the use of depth values that otherwise would fall outside one or more of the aforesaid images. The aforesaid padding operation consists in extending the borders of an image, for example by adding a given number of pixels of a predefined colour, e.g., black, around the aforesaid image.

[0373] For instance, the padding-left operation consists in extenting an image along the left border, for example, by adding a given number of pixels of a predefined colour, e.g., black, along the aforesaid left border of the image (hence, for example, an image of 50 x 50 pixels with padding-left adds 10 columns of pixels along the left border to obtain a 50 x 60 image).

[0374] The aforesaid padding operation may be performed also along other borders of an image, on the right, along the top border, or along the bottom border, extending the image along the respective border. Advantageously, padding, in the case of an image without context, i.e., at least one virtual image Imvx, can be performed without any loss of information of context.

[0375] In various embodiments, the aforesaid one or more padding operations may comprise one or more of the following: constant padding, replicate padding, reflect padding, and wrap padding.

[0376] Figure 4 is a schematic illustration of an image-processing apparatus 20, containing any image-processing apparatus 10, that is able to add image elements obtained as a function of depth values Pi, . . ., PN to (real and / or virtual) images Imxand / or Imvxin an application used to obtain a disparity map DM according to embodiments of the solutions described.

[0377] Illustrated in Figure 4 is a possible application of embodiments of the present solution; in particular, Figure 4 shows any image-processing apparatus 10 associated to a scene 100 and comprising at least one set of depth sensors 104 and one or more processing units 106. For a general description of possible image- processing apparatuses 10, reference may be made to the description of Figures 1, 2, and 3.

[0378] Moreover present in Figure 4 is a stereo matcher 208 configured for receiving at input one or more images ImRxwith projected patterns, for example virtual patterns, from the aforesaid image-processing apparatus 10 and in particular from the aforesaid one or more processing units 106, where each image ImRx corresponds to a respective image Imxand / or Imvxand contains image elements obtained as a function of sparse depth values Pi, . . PN.

[0379] In various embodiments, the aforesaid stereo matcher 208 is configured for being coupled to an image-processing apparatus 10 according to any one of the embodiments described previously.

[0380] In various embodiments, the aforesaid stereo matcher 208 is included in an image-processing apparatus 10 according to any one of the embodiments described previously; i.e., the above stereo matcher 10 may form part of the image-processing apparatus.

[0381] The aforesaid stereo matcher 208 is further configured for implementing a stereo-matching function SM of any known type, for example one of the stereo- matching operations described in what follows or any other known stereo-matching operation, and supplying at output one or more disparity maps DM that contain information regarding the difference of position of the elements in the different images. The stereo-matching function SM in general operates to solve the so-called problem of correspondence, i.e., to find a single mapping between the points belonging to two stereo images of the same scene.

[0382] In fact, given two or more image-capture sensors 102, for example two or more calibrated stereoscopic cameras, located at a given distance, for example a horizontal distance, apart, referred to as baseline, and configured for acquiring images of a given scene 100, it is possible to calculate respective depth values associated to the aforesaid scene 100 via an operation of triangulation.

[0383] The aforesaid operation of triangulation may be performed exclusively if a correspondence is known between pixels belonging to the different images acquired by the aforesaid two or more image-capture sensors 102.

[0384] The operation that is able to obtain the aforesaid correspondence between pixels belonging to different images is referred to as matching operation, and the aforesaid stereo matcher 208 is configured to execute the aforesaid matching operation in stereoscopic-vision systems, for example traditional or multiview systems, the aforesaid matching operation being defined as stereo-matching operation in the aforesaid contexts of stereoscopic vision. For instance, a first method for performing the aforesaid stereo-matching operation is referred to as dense stereo matching and is used to obtain an image, i.e., a disparity map, in which each pixel is associated to a disparity value, the aforesaid image being obtained as result of the application of the aforesaid first method to a plurality of input images, for example to the various images acquired by the aforesaid two or more image-capture sensors 102.

[0385] The aforesaid first method may comprise the following operations:

[0386] - a pre-processing operation, in which one or more pre-processing operations are carried out on the images in the aforesaid plurality of images at input, for example one or more operations of filtering, normalization, transformation, or, for example, the virtual -projection procedure described previously;

[0387] - an operation of calculation of the matching cost, in which a function to be used is selected from a plurality of possible functions, for example, an absolute- difference operation, which is configured to compare the pixels belonging to the images in the aforesaid plurality of images at input taken singly (pixel-based matching function) or in groups (area-based matching function);

[0388] - if the operation of calculation of the matching cost selects a comparison of pixels in groups, i.e., an area-based matching function, an operation of aggregation of the costs can be carried out, in which the criteria for selection of the various groups are determined, for example, using a window of fixed dimensions (fixed window), a window of variable dimensions, or any other criterion;

[0389] - an operation of calculation of the disparity, in which a strategy is selected for making the aforesaid calculation, and the aforesaid calculation of the disparity values is carried out; and

[0390] - an operation of disparity refinement, in which one or more processing operations are carried out on the disparity map obtained so as to improve the contents thereof.

[0391] Examples of strategies for performing the aforesaid operation of disparity calculation may be:

[0392] - local strategies (WTA), where, as corresponding pixel, there is selected the pixel that belongs to a given group of pixels, for example the group containing the pixels belonging to a given neighbourhood, and that is associated to a cost lower than that of the other pixels contained in the aforesaid group;

[0393] - global strategies, where the disparity map obtained is optimized via an energy function applied to the images in the aforesaid plurality of images at input; and

[0394] - hybrid (semi-global) strategies, where it is attempted to combine local strategies with global strategies.

[0395] For instance, a second method for performing the aforesaid stereo-matching operation is referred to sparse stereo matching and is used to obtain an image, i.e., a disparity map, in which only a given subset of pixels, for example a subset comprising highly distinguishable pixels referred to as features, is associated to a disparity value, the aforesaid image being obtained as a result of application of the aforesaid first method to a plurality of input images, for example to the various images acquired by the aforesaid two or more image-capture sensors 102.

[0396] The aforesaid second method may comprise the following operations:

[0397] - a pre-processing operation, for example the same pre-processing operation as described for the first method;

[0398] - an operation of feature extraction, in which a procedure of extraction of the aforesaid features is selected, for example, a SIFT (Scale-Invariant Feature Transform) procedure, which is able to extract distinguishable pixels, for example ones corresponding to corners of objects in the aforesaid scene 100;

[0399] - an operation of construction of the descriptors, in which, via the feature- extraction procedure, a label is constructed for each feature; and

[0400] - an operation of calculation of the disparity, in which the labels associated to the pixels of the images in the aforesaid plurality of images at input are compared in order to obtain a sparse disparity map by seeking to match the most similar labels.

[0401] Moreover, the aforesaid stereo-matching operation can be carried out via deep-learning networks:

[0402] - by replacing a part of the traditional pipeline, for example by replacing the cost function with a Siamese 2D convolutional network in an MC-CNN (Matching Cost -Convolutional Neural Network); or - by replacing the entire traditional end-to-end stereo pipeline, for example using 3D convolutions and softargmax for replicating the entire stereo pipeline, for example via GC-Nets (Global Context Networks), or else using recurrent neural networks for iteratively constructing the disparity map, for example via RAFT- Stereo (Recurrent All-Pairs Field Transforms - Stereo).

[0403] It should be noted that advantageously the embodiments of the present solution, since they are independent of the stereo matcher 208 used, do not require any modification of the aforesaid stereo matcher 208, which can be implemented according to any known structure and / or algorithm, thus rendering the proposed solution compatible with any known pre-existing stereo matcher or any stereo matcher that will be developed in future.

[0404] Consequently, the solution described in detail in the present document makes it possible to obtain an image-processing apparatus that is able to add references, for example image elements, to real and / or virtual images via the use of a depth sensor, for example an active depth sensor that is able to detect sparse depth values, e.g., a sensor of a LiDAR (Light Detection and Ranging) type, a radar (Radio Detection and Ranging) type, or any other sensor of a known type that is able to measure depth values, including sparse depth values.

[0405] Hence, from the foregoing it emerges clearly how the solution disclosed in the present detailed description can facilitate solution of the problem of correspondence, rendering matching of sections of a first image with respective sections of a second image more accurate and robust, maintaining a high resolution.

[0406] Furthermore, various embodiments of the solution described advantageously make it possible to obtain images with references in any condition of lighting, for example, even in the presence of intense sunlight, at any distance, i.e., both at a long range and at a short range, and without generating any interference or thermal drift; for instance, there is no interference or thermal drift due to a physical projector.

[0407] In addition, given that the solution described exploits one or more active sensors to obtain the aforesaid references, the presence of a physical projector is not necessary, thus enabling a saving on costs as compared to known solutions, a personalization and optimization of the image pattern virtually projected on the images according to the scene represented, and making it possible to consider also scenes that contain moving elements or in the presence of a respective movement of the optical devices during acquisition, from any viewpoint, without well-defined features.

[0408] Independence of the proposed solution from the stereo matcher moreover makes it possible to choose a stereo matcher according to any known structure and / or algorithm, thus increasing the compatibility of the proposed solution both with pre-existing stereo matchers and with stereo matchers that will be developed in future.

[0409] Without prejudice to the underlying principles of the invention, the details and embodiments may vary, even appreciably, with respect to what has been described herein purely by way of example, without thereby departing from the sphere of protection and the scope of the invention as this is defined in the annexed claims.

Claims

CLAIMS1. An image-processing apparatus (10) comprising:- at least one module (102; 106) configured for acquiring a set of images (Imvx, Imx) of a scene (100), in particular a stereoscopic-vision system or a multiview stereoscopic system;- at least one depth sensor (104), in particular an active sensor, in particular said at least one depth sensor (104) being of a LiDAR type, or radar type, or ToF type, or SLAM type, configured for detecting a plurality of sparse depth values (Pi, . . ., PN) in said scene (100); and- at least one processing unit (106), wherein said at least one processing unit (106) is configured for:- receiving said plurality of sparse depth values (Pi, . . . , PN) detected by said at least one depth sensor (104); and- performing at least one virtual -projection procedure (VPP) comprising:- associating said plurality of sparse depth values (Pi, . . ., PN) of said scene (100) to said set of acquired images (Imvx, Imx) of said scene (100), said operation of association comprising, for each depth value in said plurality of sparse depth values (Pi, ..., PN) detected via said at least one depth sensor (104), associating the depth value of a point (PTh) of said scene (100) to a respective pixel (Imvx(m, y), Imx(m, y)) of each image in said set of acquired images (Imvx, Imx) of said scene (100) that corresponds to the same point (PTh) of the scene (100), said virtual -projection procedure (VPP) comprising generating a set of two- dimensional images (ImRx) of said scene (100) with projected patterns via an operation of augmenting each image in said set of acquired images (Imvx, Imx) via the steps of:- identifying a set of corresponding pixels that contains pixels (Imvx(m, y), Imx(m, y)) belonging to different images in said set of acquired images (Imvx, Imx) to which there has been associated one and the same depth value in said plurality of sparse depth values (Pi, . . ., PN); and- overlaying on each pixel (Imvx(m, y), Imx(m, y)) contained in said set of corresponding pixels one and the same image element (A) chosen from a plurality of image elements.

2. The apparatus (10) according to claim 1, wherein said apparatus (10) is coupled to, or comprises, a stereo matcher (208) configured for receiving at input said set of two-dimensional images (ImRx) with projected patterns from said image- processing apparatus (10), wherein, in particular, each two-dimensional image (ImRx) in said set of two-dimensional images (ImRx) with projected patterns corresponds to a respective image in said set of images (Imvx, Imx) acquired by the at least one module (102; 106).

3. The apparatus (10) according to any one of the preceding claims, wherein:- said at least one module (102; 106) configured for acquiring said set of images (Imvx, Imx) of said scene (100) comprises at least one image-capture sensor (102), which in particular corresponds to optical devices that implement a stereoscopic-vision system or a multiview stereoscopic system and is configured for acquiring a first subset (Imx) of said set of images (Imvx, Imx); and- said at least one processing unit (106) is further configured for receiving said first subset (Imx) of said set of images (Imvx, Imx) acquired via said at least one image-capture sensors (102).

4. The apparatus (10) according to claim 3, wherein said at least one image- capture sensor (102) and said at least one depth sensor (104) are obtained via a single device configured both for detecting said plurality of sparse depth values (Pi, ..., PN) in said scene (100) and for acquiring said first subset (Imx) of said set of images (Imvx, Imx) of said scene (100).

5. The apparatus (10) according to any one of the preceding claims, wherein:- said at least one module (102; 106) configured for acquiring said set of images (Imvx, Imx) of said scene (100) comprises the at least one processing unit (106), said at least one processing unit (106) being further configured for simulating at least one virtual image-capture sensor (VOD), which is configured for acquiring a second subset (Imvx) of said set of images (Imvx, Imx), in particular, wherein eachimage in said second subset (Imvx) of said set of images (Imvx, Imx) comprises virtual pixels (Imvx(m, y)) configured for containing any colour, in particular black.

6. The apparatus (10) according to claim 3, wherein applied to said at least one image-capture sensor (102) and said at least one depth sensor (104) is a process of calibration with respect to depth, and said calibration with respect to depth comprises, for each depth value in said plurality of sparse depth values (Pi, . . . , PN) detected via said at least one depth sensor (104) and for each image that is to be calibrated with respect to depth in said first subset (Imx) of said set of images (Imvx, Imx) acquired by said at least one image-capture sensor (102):- determining, for the depth value in said plurality of sparse depth values (Pi, ..., PN) detected via said at least one depth sensor (104), a respective pixel (Imx(m, y)) belonging to the image to be calibrated with respect to depth in said first subset (Imx) of said set of images (Imvx, Imx) acquired by said at least one image-capture sensor (102), wherein said depth value and said respective pixel (Imx(m, y)) are associated to one and the same point (PTh) of the scene (100);- determining a set of co-ordinates (m, y; m’, y) associated to said respective pixel (Imx(m, y)), considering a reference system associated to said image to be calibrated with respect to depth that contains said respective pixel (Imx(m, y)); and- associating said set of co-ordinates (m, y; m’, y) to said depth value in said plurality of sparse depth values (Pi, ..., PN) detected via said at least one depth sensor (104).

7. The apparatus (10) according to claim 6, wherein:- said first subset (Imx) of said set of images (Imvx, Imx) comprises at least two images;- at least one image in said first subset (Imx) of said set of images (Imvx, Imx) is an image calibrated with respect to depth (Imi);- at least one image in said first subset (Imx) of said set of images (Imvx, Imx) is an image not calibrated with respect to depth (Imx), which is not subjected to said process of calibration with respect to depth; and- association, for each depth value in said plurality of sparse depth values (Pi, . . ., PN) detected via said at least one depth sensor (104), of the depth value ofa point (PTh) of said scene (100) to a respective pixel (Imx(m, y)) of said at least one image of said scene (100) not calibrated with respect to depth (Imx) and to the same point (PTh) of the scene (100), via said virtual -projection procedure (VPP), comprises:- computing at least one disparity value (di(m, y)) indicating a distance between a first position of the at least one image-capture sensor (102) during a process of acquisition of said at least one image calibrated with respect to depth (Imi) and a second position of the at least one image- capture sensor (102) during a process of acquisition of said at least one image not calibrated with respect to depth (Imx), as a function of said depth value (Pi);- computing at least one value of a co-ordinate (m’) associated to said respective pixel (Imx(m, y)), considering a reference system associated to said at least one image not calibrated with respect to depth (Imx) that contains said respective pixel (Imx(m, y)), as a function of said at least one disparity value (di(m, y)); and- associating said depth value (Pi) to said respective pixel (Imx(m, y)) of said at least one image not calibrated with respect to depth (Imx) via said at least one calculated value of co-ordinate (m’).

8. The apparatus (10) according to any one of claims 3 to 5, wherein:- the set of images (Imvx, Imx) comprises at least two images;- at least one image in said set of images (Imvx, Imx) is acquired from a first position that corresponds to the position of said at least one depth sensor (104) to obtain at least one image where each depth value in said plurality of sparse depth values (Pi, ... , PN) of a point (PTh) of the scene (100) is associated to a pixel (Imvx(m, y), Imx(m, y)) of said at least one image that corresponds to the same point (PTh) of said scene (100);- at least one image in said set of images (Imvx, Imx) is acquired from a second position different from said first position; and- association, for each depth value in said plurality of sparse depth values (Pi, . . ., PN) detected via said at least one depth sensor (104), of the depth value ofa point (PTh) of said scene (100) to a respective pixel (Imvx(m, y), Imx(m, y)) of said at least one image acquired from said second position and corresponding to the same point (PTh) of the scene (100), via said virtual -projection procedure (VPP), comprises:- computing at least one disparity value (di(m, y)) indicating a distance between said first position and said second position, as a function of said depth value (Pi);- computing at least one value of a co-ordinate (m’) associated to said respective pixel (Imvx(m, y), Imx(m, y)), considering a reference system associated to said at least one image acquired from said second position that contains said respective pixel (Imvx(m, y), Imx(m, y)), as a function of said at least one disparity value (di(m, y)); and- associating said depth value (Pi) to said respective pixel (Imvx(m, y), Imx(m, y)) of said at least one image acquired from said second position via said at least one calculated value of co-ordinate (m’).

9. The apparatus (10) according to any one of the preceding claims, wherein:- each image element (A) in said plurality of image elements is associated to at least one respective value; and- said image element (A) to be overlaid on said pixels (Imvx(m, y), Imx(m, y)) contained in said set of corresponding pixels is chosen randomly, in particular, said image element (A) is chosen if the at least one respective value associated to said image element (A) is drawn from a probability distribution, in particular a uniform probability distribution.

10. The apparatus (10) according to claim 3 or claim 4, wherein said image element (A) to be overlaid on said pixels (Imvx(m, y), Imx(m, y)) contained in said set of corresponding pixels is chosen via the following operations:- generating a plurality of windows (Fi, F2), where said plurality of windows (Fi, F2) comprises at least one window for each pixel in said set of corresponding pixels that is centred on said pixel in said set of corresponding pixels;- defining a plurality of histograms (hx), where said plurality of histograms (hx) comprises one histogram for each window in said plurality of windows (Fi, F2), said histogram containing a value of frequency of occurrence within said window for each image element (A) in said plurality of image elements;- defining a sum histogram (Hs) as the sum of the histograms contained in said plurality of histograms (hx), which contains a value of frequency of occurrence within said plurality of windows (Fi, F2) for each image element (A) in said plurality of image elements;- defining a distance histogram (hdist), which contains, for each image element (A) in said plurality of image elements, an indication on the minimum distance between said image element (A) in said plurality of image elements and any image element (A) that corresponds to a value of frequency of occurrence greater than zero in said sum histogram (Hs); and- choosing, as image element (A) to be overlaid on said pixels (Imvx(m, y), Imx(m, y)) contained in said set of corresponding pixels, the image element (A) in said plurality of image elements the value of minimum distance of which contained in said distance histogram (hdist) is greater than the values of minimum distance associated to the other image elements in said plurality of image elements contained in said distance histogram (hdist).

11. The apparatus (10) according to claim 10, wherein, if said sum histogram (Hs) contains values of frequency of occurrence greater than zero for each image element (A) in said plurality of image elements, the chosen image element (A) to be overlaid on said pixels (Imvx(m, y), Imx(m, y)) contained in said set of corresponding pixels is the image element (A) the value of frequency of occurrence of which contained in the sum histogram (Hs) is less than the values of frequency of occurrence associated to the other image elements in said plurality of image elements contained in said sum histogram (Hs).

12. The apparatus (10) according to any one of the preceding claims, wherein said overlaying on each pixel (Imvx(m, y), Imx(m, y)) contained in said set of corresponding pixels one and the same image element (A) chosen from a plurality of image elements further comprises:- overlaying said same image element (A) on pixels (Imvx(m, y), Imx(m, y)) comprised in a given window centred on said pixel (Imvx(m, y), Imx(m, y)) contained in said set of corresponding pixels; or- overlaying further image elements (A) chosen from said plurality of image elements on pixels (Imvx(m, y), Imx(m, y)) comprised in a given window centred on said pixel (Imvx(m, y), Imx(m, y)) contained in said set of corresponding pixels.

13. The apparatus (10) according to any one of the preceding claims, wherein said overlaying on each pixel (Imvx(m, y), Imx(m, y)) contained in said set of corresponding pixels one and the same image element (A) chosen from a plurality of image elements is obtained by combining said pixel (Imvx(m, y), Imx(m, y)) contained in said set of corresponding pixels with said image element (A) via the alpha-blending technique.

14. The apparatus (10) according to any one of the preceding claims, wherein:- at least one image in said set of images (Imvx, Imx) is a reference image (Imvi, Imi); and- said at least one processing unit (106) is further configured to execute, for each image, except for said reference image (Imvi, Imi), in said set of images (Imvx, Imx), operations for:- calculating a plurality of disparity values (di(m, y)), indicating a distance between a first position of acquisition of the image in said set of images (Imvx, Imx) and a second position of acquisition of said reference image (Imvi, Imi), as a function of the plurality of sparse depth values (Pi);- mapping said plurality of disparity values (di(m, y)) on a plurality of cells (W(j, k)) of a grid (W), selecting for each cell (W(j, k)) the disparity value (di(m, y)) having the highest value from among a set of disparity values (di(m, y)) that contains disparity values (di(m, y)) that have to be mapped on said cell (W(j, k));- classifying each cell (W(j, k)) in said plurality of cells (W(j, k)) of said grid (W) as occluded if there exists at least one other cell (W(j’, k’)) in a neighbourhood centred on said cell (W(j, k)) that corresponds to adisparity value (di(m’, y’)) that differs from a disparity value (di(m, y)) corresponding to said cell (W(j, k)) by a value higher than a given threshold; and- obtaining an occlusion mask indicating the positions, within the grid (W), of the occluded cells as a function of the cells classified as occluded.

15. The apparatus (10) according to claim 14, wherein said occlusion mask is superimposed upon at least two images in said set of images (Imvx, Imx), to obtain a correspondence between the cells classified as occluded and the pixels (Imvx(m, y), Imx(m, y)) of said at least two images, by applying the following criteria:- avoiding overlaying an image element (A) on said pixels (Imvx(m, y), Imx(m, y)) that correspond to said cells classified as occluded;- dividing said pixels (Imvx(m, y), Imx(m, y)) corresponding to said cells classified as occluded into occluded pixels, in which a respective point (PTh) of said scene (100) is visible, and occluding pixels, in which a respective point (PTh) of said scene (100) is not visible; and- replacing the content of said occluded pixels belonging to one of said at least two images with the content of said occluding pixels belonging to another of said at least two images and wherein an occluded pixel and a respective occluding pixel the content of which is substituted for that of the occluded pixel are associated to one and the same respective point (PTh) of said scene (100).

16. A method for processing images via an apparatus (10) according to any one of the preceding claims, said method comprising at least the operations of:- acquiring, via at least one module (102; 106), in particular a stereoscopic- vision system or a multiview stereoscopic system, a set of images (Imvx, Imx) of a scene (100);- detecting, via at least one depth sensor (104), in particular an active sensor, in particular, said at least one depth sensor (104) being of a LiDAR type, or radar type, or ToF type, or SLAM type, a plurality of sparse depth values (Pi, . . ., PN) in said scene (100); and- executing, via at least one processing unit (106) configured for receiving said plurality of sparse depth values (Pi, . . ., PN) detected by said at least one depth sensor (104), at least one virtual -projection procedure (VPP), said virtual -projection procedure (VPP) comprising:- associating, via said at least one processing unit (106), said plurality of sparse depth values (Pi, ..., PN) of said scene (100) to said set of acquired images (Imvx, Imx) of said scene (100), said operation of association comprising, for each depth value in said plurality of sparse depth values (Pi, . . ., PN) detected via said at least one depth sensor (104), associating the depth value of a point (PTh) of said scene (100) to a respective pixel (Imvx(m, y), Imx(m, y)) of each image in said set of acquired images (Imvx, Imx) of said scene (100) that corresponds to the same point (PTh) of the scene (100); and- generating, via said at least one processing unit (106), a set of two- dimensional images (ImRx) of said scene (100) with projected patterns via an operation of augmenting each image in said set of acquired images (Imvx, Imx) via the steps of- identifying, via said at least one processing unit (106), a set of corresponding pixels that contains pixels (Imvx(m, y), Imx(m, y)) belonging to different images in said set of acquired images (Imvx, Imx) to which there has been associated one and the same depth value in said plurality of sparse depth values (Pi, . . ., PN); and- overlaying, via said at least one processing unit (106), on each pixel (Imvx(m, y), Imx(m, y)) contained in said set of corresponding pixels one and the same image element (A) chosen from a plurality of image elements.

17. A computer-program product that can be loaded into the memory of at least one computer and comprises portions of software code for executing the steps of the method according to claim 16.

Images