Self-supervised method for obtaining depth, albedo and surface orientation estimates of a space illuminated by a light source

Abstract

The present invention discloses a method for obtaining data of a space (1) illuminated by a light source (10) comprised by a joint camera-light source system (2), in which a neural network (3) is trained using the illumination decline profile of every pixel in a training set of 2D image data. In this way, the neural network (3) can estimate the depth (5), albedo (6) and surface orientation (7) of each pixel of 2D input image data (8) allowing for a posterior 3D reconstruction (9) of the space (1) depicted in said 2D input image data, including its structures, shapes and colors. The use of the method of the invention may be particularly relevant for processing endoscopic images, among other applications. A system comprising means to carry out the method of the invention and a computer program comprising instructions to carry out the method of the invention are also disclosed.

Description

[0001] DESCRIPTION

[0002] SELF-SUPERVISED METHOD FOR OBTAINING DEPTH, ALBEDO AND SURFACE ORIENTATION ESTIMATES OF A SPACE ILLUMINATED BY A LIGHT SOURCE

[0003] FIELD OF THE INVENTION

[0004] The present invention belongs to the field of image processing and 2D-3D reconstruction by employing artificial intelligence methods such as neural networks. More specifically, the invention describes a method for obtaining data of a space depicted in an image, such as depth, albedo, and surface orientation, by means of a neural network trained under a selfsupervision technique based on illumination decline. This allows for using image data both for training and inference, leading to an accurate, robust, and domain-shift-free reconstruction of the mentioned space.

[0005] BACKGROUND OF THE INVENTION

[0006] Comprehension of structures, colors, and characteristics of a space, cavity or volume can be relevant for many industrial applications. Nevertheless, in many situations, the only available information of these spaces comes from image data, i.e., from a two-dimensional (2D) representation of said spaces, but most of the time, three-dimensional (3D) information may be needed. These situations can be found in a plethora of procedures, typically when the space of interest is shallow, small, or somehow else inaccessible. In these cases, probes incorporating a camera and a light source are employed to survey the space, and, by acquiring 2D images, the space characteristics are then reconstructed. In many of these cases, the size of the probe is especially relevant, since it may need to be as small as possible in order to fit in the space of interest and / or in order not to alter or interact with the structures present in said space.

[0007] An example of such a situation relates to medical imaging procedures involving endoscopic techniques such as gastroscopies, colonoscopies, or bronchoscopies. These procedures require the employed imaging instruments to be small sized in order to remain as much non-invasive as possible for the patient. For this reason, endoscopes typically comprise a single camera and several illumination points within their housing, but rarely depth or stereo cameras which increment the size of the endoscope considerably. However, 3D comprehension of the explored structures is very relevant in such procedures since, for instance, obtaining accurate estimations of the size and shape of tumors or other lesions may lead to faster, and more precise diagnostics and treatments. Therefore, there is a great need for developing accurate and robust 3D reconstruction techniques suitable for application to endoscopic images, among other applications. In this regard, Artificial Intelligence (Al) algorithms and techniques have been proposed to obtain 3D reconstruction methods that provide good results for this kind of techniques, both with single-view and with multi-view depth estimation.

[0008] Different works have demonstrated the effectiveness of deep neural networks for supervised pixel-wise depth regression in generic, single-view, natural images. Complementary research efforts have also made contributions in many different directions, such as, to name a few, network architectures that evolved to fully convolutional, and, more recently, to transformers. In some works, the continuous depth space is discretized into bins and the problem is formulated as an ordinal regression. Other advances include interpretability, uncertainty quantification, and modelling camera intrinsic characteristics. All these approaches rely on supervised training of neural networks, and, therefore, require depth ground-truth data, which can be difficult and expensive to acquire. Some proposals use computerized tomography (CT) renderings for depth supervision in bronchoscopies, but CT scans in particular, and ground-truth depth data in general, are very rare in endoscopy and other applications.

[0009] Self-supervised methods seek to overcome this limitation and reduce the need for groundtruth data, often by exploiting multi-view photometric consistency. This also enables depth refinement at test time, but, unfortunately, this kind of supervision can be noisy, due to inaccuracies in the camera motion estimation, perspective distortions, occlusions or non- Lambertian effects, among others. As a result, state-of-the-art self-supervised methods typically suffer from significantly larger inaccuracies than supervised ones. Furthermore, many works have explored multi-view integration combined with tracking and simultaneous localization and mapping (SLAM) pipelines. Others propose video-based training schemes. Unfortunately, multi-view self-supervision is even more challenging in endoscopy than in other areas due to the presence of deformations and weak texture.

[0010] Another issue that has been vastly explored regarding endoscopic 3D reconstruction is domain shift, i.e., transferring simulated results into predictions for real scenarios. In this regard, different works propose conditional GANs for depth recovery while integrating SLAM and multi-view inputs. Other frameworks include a depth network trained with synthetic images of a simple colon model and fine-tuned with domain-randomized photorealistic images rendered from CT scans. However, results coming from supervised learning which are then transferred using uncertainty typically use monocular videos and multi-view stereo, and they end up providing weak depth supervision.

[0011] Depth estimation from a single image started with the so-called shape-from-shading (SfS) method, but this technique relies on strong assumptions that do not hold in endoscopic imagery. In SfS, the camera and directional point light model are located at infinity; the reflectance is Lambertian; the albedo is constant, and the surfaces are smooth. Importantly, lights at infinity result in ill-posed problems. By contrast, when the light source is co-located with the camera that is not distant from the target surfaces, there is a 1 / d2 attenuation of pixel intensity, with distance d being the distance from the light source to the surface, which makes the problem well-posed when the albedo is assumed to be constant.

[0012] More recently, the topic was revisited by SIRFS (Shape, Illumination, and Reflectance from Shading) in which the interdependencies between shape, illumination and reflectance are modelled, and in which statistical priors on these quantities are introduced to disentangle their effects. In other works, priors are learned by deep neural networks using supervision, synthetic-to-real or multi-view self-supervision. The use of priors in this context makes the deployment of these models more difficult to be carried out.

[0013] In summary, both single- and multi-view depth estimation methods present significant challenges when applied to endoscopy. The lack of sufficient depth annotated data hinders the use of supervised depth learning. The presence of fluids that either obscure the view or generate reflections, the sudden illumination changes, the paucity of texture and the surface deformations hamper multi-view methods both for self-supervising deep networks and for geometry estimation. Real in-body textures and fluids are hard to simulate realistically, and the synthetic-to-real gap can be large.

[0014] These challenges are not only present in the case of endoscopy, but in many other applications. For instance, pipe inspection both in domestic and industrial environments shares all these characteristics and, therefore, also the same shortcomings regarding reconstruction techniques in the art. The same happens in other cases in which not a cavity is necessarily explored. For instance, inspecting the bottom of the sea with a camera-light source system, inspecting a room or an environment with an infrared camera-light source system, and many other situations share similarities with endoscopic processes that lead to the same challenges when trying to reconstruct the spaces captured by the camera.

[0015] BRIEF DESCRIPTION OF THE INVENTION

[0016] To overcome the above limitations related to depth supervision, multi-view self-supervision and estimation and synthetic-to-real gaps, the invention proposes a novel approach to obtain reconstruction information or 3D reconstructions of a space or cavity in real time specially designed for its use in endoscopic techniques, among other applications, which is based on the use of neural networks that, after trained within an illumination decline framework, estimate depth, albedo and surface orientation for each pixel of a 2D image.

[0017] Advantageously, a first object of the invention relates to a computer implemented method for obtaining data of a space illuminated by a light source, said light source being comprised by a joint camera-light source system, and the method comprising performing the following steps: a) providing a set of training joint camera-light source system 2D image data of at least one space comprising illumination information; b) delivering first training 2D image data to a neural network; c) obtaining, for each pixel of the first training 2D image data, a depth estimate an albedo estimate and a surface orientation estimate corresponding to each pixel of the first training 2D image data; d) training a neural network by using the depth, albedo and surface orientation estimates obtained in the previous step and by applying the illumination decline principle; e) delivering first input 2D image data of a space illuminated by a light source comprised by a joint camera-light source system to the trained neural network; f) obtaining, using the trained neural network, a final depth estimate, a final albedo estimate and a final surface orientation estimate for each pixel of the first input 2D image data.

[0018] By implementing the method of the invention, accurate information of a space depicted in an image can be obtained in a robust manner without the need for annotated or manually supervised large training datasets, which, as previously mentioned, are in some cases difficult to gather. In contrast, only the (non-annotated) information comprised by the training joint camera-light source system 2D image data is needed to carry out the method of the invention. These training joint camera-light source system 2D image data correspond to image data either captured by a real joint camera-light source system or generated synthetically by simulating a joint camera-light source system artificially.

[0019] In the implementation of the method of the invention, depth, albedo, and surface orientation of a space and structures can be estimated using only the illumination decline principle applied to a 2D image as a self-supervision signal, without requiring any external source of depth supervision. It is to be noted that both training and inference are done pixel-wise meaning that both procedures can be carried out for a single image or for a set of images. Since the method requires only the information comprised by the image, test-time refinement can be performed (as explained below) to refine the estimation leading to much more accurate reconstructions. Moreover, by implementing the method of the invention, the domain-shift problem is eliminated, since there is no need for adapting or translating the results built from synthetic scenarios to the cases of real application. Furthermore, it is to be noted that albedo estimation represents very important information in endoscopy, since, for instance, there are colon lesions, such as those associated with Crohn’s disease, which need the albedo information to be identified. This is currently not in the art and represents a great advantage of the method of the invention.

[0020] In another embodiment of the invention, the method comprises performing an additional step after step f): g) obtaining a 2D or 3D image reconstruction of the space depicted in the first input 2D image data by using the depth, albedo and surface orientation estimates obtained in step f).

[0021] This allows for a better interpretation of the depth, albedo, and surface orientation estimates. Even though said estimates represent valuable information of the space shown in an image and can be easily processed to obtain different aspects or characteristics of the mentioned space, a 2D or 3D reconstruction allows for a complete visualization of the information of these three estimates combined. In this way, a representation of a space appearing in an image can be extracted and visualized by a user, so that structures, colors, shapes, etc. can be visually identified and inspected.

[0022] In another embodiment of the invention, the set of single view training 2D image data comprises one or many, monocular or stereo, related or unrelated, consecutive or nonconsecutive, real or simulated images. As opposed to the reconstruction techniques in the art, the method of the invention can be carried out using as training images a set of unrelated image data. This means that there is no need to use image data of the same space captured from different angles, illumination, or perspectives for the method to perform accurately. To obtain the best possible results, the neural network may be trained with images of a space as similar as possible as the one to be reconstructed in the inference phase, but these images do not need to be related. On the other hand, several techniques in the art employ as training images a sequence of video frames in order to reconstruct the properties of a space in a consecutive manner. Although the method of the invention may work under these circumstances, it also works accurately without the images or frames being consecutive. Furthermore, the method of the invention can be performed using one or many images as training 2D image data, and that or those images may be real or synthesized and monocular or stereo images.

[0023] In another embodiment of the invention, the neural network comprises parameters with an initial value and the training of the neural network is carried out by:

[0024] - obtaining, from the depth, albedo, and surface orientation estimates obtained in step c), first synthesized 2D image data by using a rendering system based on illumination decline;

[0025] - updating the parameter values of the neural network by minimizing a loss function that comprises the differences between the first training 2D image data and the first synthesized 2D image data; and

[0026] - repeating steps b)-d) iteratively for different training 2D image data until the neural network is trained;

[0027] This method of training the neural network by comparing training image data with its corresponding synthesized image data leads to very robust results and represents a simple but accurate way of performing the training of the neural network. Specifically, when tailored loss functions are employed, very precise results are obtained. These loss functions may vary or depend on the specific application for which the method is to be used.

[0028] In another embodiment of the invention, the training of the neural network in step d) is further based on geometric and / or photometric calibration parameters of a joint camera-light source system. A higher degree of accuracy is obtained in the reconstruction of a space by implementing this embodiment in which the specific characteristics of the joint camera-light source system to be used for inference are taken into account. Nevertheless, generic calibration parameters obtained for a specific or generic model of the joint camera-light source system may be employed. In another embodiment of the invention steps e) and f) are carried out in real time. Additionally, step g) may be carried out in real time as well. In many applications, the method of the invention may be used as a guide for different instruments working in an inaccessible space. In these cases, the need for real-time representation of the space as directly captured by the camera is of great importance for the efficiency of the task carried out by said instruments.

[0029] In another embodiment of the invention, the method further comprises performing, after step e), the following step: e1) re-training the neural network using as first training 2D image data the input 2D image data delivered in step e).

[0030] As previously mentioned, in the inference phase, test-time refinement can be performed, i.e., the parameters of the neural network can be updated by using as training image an input image. This can be done for every input image, such that the accuracy of the predictions delivered by the neural network improves with each use, thus obtaining much more accurate reconstructions of the space of interest by refining the depth estimates continuously online.

[0031] In another embodiment of the invention, the training of the neural network in steps d) and / or e1) comprises a self-supervised training. This is possible due to the fact that the only information used for the training of the neural network is that comprised by the training 2D image data. Thus, since no annotations or manual supervision are required, the neural network can be trained in a self-supervised manner by using the illumination decline principle as the only self-supervision criterion. This is especially advantageous since, as stated, annotated data is usually difficult to obtain in certain applications such as endoscopy. It is to be noted that even though the neural network can be trained in a selfsupervised way, other types or combinations of types of supervision during the training phase may be employed. For instance, the training may be done in a self-supervised manner and refined by using annotated data.

[0032] In another embodiment of the invention, steps e)-f) are repeated for each of the frames comprised by a monocular or stereo video sequence, using said frames as 2D input image data, and the depth, albedo and surface orientation estimates corresponding to each of said image data are fused to obtain:

[0033] - the joint camera-light source system position and orientation in each image data; and / or

[0034] - a 3D reconstruction of an extended portion of the cavity appearing in said image data.

[0035] Obtaining the position of the camera-light source system within the depicted space appearing in the delivered image data and, even more, reconstructing sections of said space not shown in the data represents a very relevant advantage. In several applications, being able to extract information from unseen portions of a space may be critical, such as in endoscopy where pushing the endoscope further may imply damages for the patient, among other examples. It is to be noted that in any of the two above-mentioned cases, the images can be inputted to the neural network in the same sequence in which they had been taken, or non-consecutively.

[0036] In another embodiment of the invention, after step g), the method further comprises the step of displaying in a display at least one of the following:

[0037] - the final depth estimate;

[0038] - the final albedo estimate;

[0039] - the final surface orientation estimate; and

[0040] - the 2D or 3D reconstruction of the space obtained in step g).

[0041] This display may be any kind of visual display, ranging from a screen to virtual reality or augmented reality displays, among others.

[0042] In another embodiment of the invention, after step f), the resulting depth, albedo and surface orientation estimates are processed by using digital image processing means such that:

[0043] - different structures comprised by the space are identified;

[0044] - measurements of lengths, areas, volumes and / or albedos of structures appearing in the image data are obtained; and / or

[0045] - the percentage of observed structures and / or the percentage of space occluded by dirt, liquids or other obstacles is calculated.

[0046] Several applications benefit from the implementation of this embodiment, such as those related to endoscopy identifying tumors, occlusions, lesions, etc., among others.

[0047] In another embodiment of the invention, the space is one of the following:

[0048] - a hollow organ or a cavity of the body, such as in endoscopic procedures;

[0049] - a region of the seabed and / or other underwater space;

[0050] - a cavity comprised by a rigid structure, such as in the case of inspection of engines or other mechanical components;

[0051] - a pipe or other type of narrow, inaccessible, cylindrical structure;

[0052] - a region comprised by a sewage system; and

[0053] - an underground environment.

[0054] A second object of the invention relates to a system comprising joint camera-light source system adapted to acquire 2D image data, the joint camera-light source system being connected to computing means, and characterized in that the computing means comprise hardware and / or software means adapted to perform a method as disclosed in the various embodiments described above. These computing means may comprise a computer, but may also, or alternatively, comprise distributed computing means such as those allocated in the cloud.

[0055] In another embodiment of the invention, the joint camera-light source system is comprised by an endoscope, a capsule endoscope, a borescope, an augmented / virtual reality device such as augmented / virtual reality glasses comprising a light source, a mobile device such as a smartphone in which the camera flash light acts as the light source, a wearable device such as military helmets or night vision goggles, a robotic device such as a medical robot or an maintenance robot and / or a vehicle.

[0056] Furthermore, the system may comprise, optionally, additional sensors such as inertial units, positioning systems (GPS), electromagnetic pose or proprioceptive encoders, etc. These sensors are adapted to obtain sensor data of a space and the processing means are adapted to provide combined data of said space by combining the sensor data, the final depth, albedo and surface orientation estimates, and, optionally, the joint camera-light source system position and orientation, for instance, as obtained by carrying out a method according to an embodiment of the invention.

[0057] In various applications, this may be especially advantageous, since a reconstruction of the space can be performed much more accurately. Depth, albedo and surface orientation information as obtained by performing the method of the invention is complemented with physical information such as inclination or position of the joint camera-light source system, allowing for a much more precise identification of every structure and element of the space. Moreover, in applications such as endoscopy, this feature, combined with the possibility of having real-time reconstructions of the corresponding cavity or organ, allows the user to move the endoscope (joint-camera light source system) in the most efficient way in order to explore the whole cavity or organ in a non-invasive way.

[0058] A third object of the invention relates to a computer program comprising instructions which, when the program is executed by computing means, cause the computing means to carry out a method as described above in the different embodiments.

[0059] Within the scope of the invention, the term "space illuminated by a light source" shall be interpreted as any volume illuminated by a single light source, or by a light source whose illumination can be efficiently decoupled from the illumination of other light sources. For instance, a hollow cavity of the body illuminated exclusively by the light source of an endoscope (being said light source the only one present in said hollow cavity), a room illuminated by an infrared light source (existing, therefore, the possibility of decoupling the infrared illumination from the visible illumination by using an infrared camera), etc. In this sense, the term “light” may refer to an electromagnetic wave of any frequency, not only those in the (near-)visible spectrum. As long as the camera comprised by the camera-light source system is adapted to capture the light employed to illuminate the space, the method of the invention can be carried out efficiently. In fact, better results may be obtained by using narrow banding techniques, i.e., by using a single frequency to illuminate the space of interest.

[0060] Within the scope of the invention, the term "camera-light source system" shall be interpreted as a device comprising a camera and a light source, the light source positioned contiguously to the camera and rigidly coupled thereto such that both elements move solidly.

[0061] Within the scope of the invention, the term "illumination decline" or “illumination decline principle” shall be interpreted as the fact that, in a space illuminated by a direct light source, the illumination of a surface point comprised by said space decays with a function of the distance from said surface point to the light source.

[0062] In summary, the proposed method allows for reconstructing the structures, shapes and other properties of a space depicted in a 2D image in a fast, robust, precise, simple and domain-shift-free way.

[0063] All the terms and embodiments described anywhere in this document are equally applicable to all aspects of the invention. It should be noted that, as used in the specification and in the appended claims, the singular forms “a”, “an”, and “the” include their plural referents unless the context clearly indicates otherwise. Similarly, the term “comprises” or “comprising” as used herein also describes “consists of” or “consisting of” in accordance with generally accepted patent practice.

[0064] DESCRIPTION OF THE DRAWINGS

[0065] The foregoing and other features and advantages will be more fully understood from the detailed description of the invention, as well as from examples referring to the attached figures, which are described in the following paragraphs, wherein:

[0066] Figure 1 shows the flow chart corresponding to the implementation of a method according to an embodiment of the invention.

[0067] Figure 2 shows an example of the results obtained for the depth, albedo, and surface orientation (normals) estimates of a space from a 2D image as well as the results for the 3D reconstruction of said space, together with the intermediate synthesized image according to an embodiment of the invention.

[0068] Figure 3 shows the reference system and the defined positions, distances and angles used to build up a photometric spotlight illumination model used in a method according to an embodiment of the invention.

[0069] Figure 4A-4C shows a schematic representation of the complete architecture of the neural network used in a method according to an embodiment of the invention, in two different implementations. Fig. 4A shows the overall process performed within the method of the invention, whereas Figs. 4B and 40 show specific implementations of the neural network used in said process. Fig. 4B shows a standard ll-net implementation with two decoding branches, while Fig. 4C shows a DPT-hybrid architecture with a second decoder branch added for the albedo.

[0070] Figure 5 shows the results obtained for the depth, albedo and surface orientation estimates by applying the method of the invention to two different endoscopic images, in both cases displaying the results with and without test-time refinement. Figure 6 shows the qualitative results of the implementation of a method according to an embodiment of the invention using the EndoMapper training dataset. Columns 1-5 correspond to real colonoscopy images, and columns 6-7 to real gastroscopy images.

[0071] Numerical references used in the drawings

[0072] In order to provide a better understanding of the technical features of the invention, the referred Figures 1-6 are accompanied of a series of numerical references which, with an illustrative and non-limiting character, are hereby represented:

[0073] DETAILED DESCRIPTION OF THE INVENTION

[0074] The detailed description of the present invention to be described below refers to the accompanying drawings, which illustrate specific embodiments in which the present invention may be implemented. These embodiments will be described in detail sufficient to enable those skilled in the art to implement the present invention. It should be understood that various embodiments of the present invention are different from each other but need not be mutually exclusive. Accordingly, the detailed description to be described below is not intended to be taken in a limiting meaning, and the scope of the present invention, if properly described, is limited only by the appended claims, in addition to all scopes equivalent to those claimed by the appended claims. In the drawings, reference numerals refer to the same or similar functions over several aspects.

[0075] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings.

[0076] As disclosed above, the invention relates to a method for obtaining data of a space (1) illuminated by a light source comprised by a joint camera-light source system (2), in which a neural network (3) is trained using the illumination decline profile of every pixel in a set of training 2D image data (4). In this way, the neural network (3) can estimate the depth (5), albedo (6) and surface orientation (7) of each pixel of input 2D image data (8) allowing for a posterior 3D reconstruction (9) of the space (1) depicted in said 2D input image data (8), including its structures, shapes and colors. As mentioned previously, although any kind of space (1) illuminated by a light source (10) comprised by a joint camera-light source system (2) can be visually reconstructed using the method of the invention, its application is particularly relevant for exploring or inspecting shallow, inaccessible spaces (1) such as cavities or pipes. Because of this, it will be shown, in the following, and according to Figs. 1-10, a particular embodiment of the method of the invention applied to the field of endoscopic imaging, described for illustrative, but not limiting purposes.

[0077] Fig. 1 shows the flow chart corresponding to the elements and steps of the method of the invention according to a particular embodiment. There, first, the camera-light source system (2) obtains first 2D image data of a space (1) illuminated by the light source (10) comprised by said camera-light source system (2). The first 2D image data are delivered to a neural network (3) which calculates a depth estimate (5) and an albedo estimate (6) for each pixel comprised by the first 2D image data. Then, the depth estimate (5) is delivered to a depth- to-normals algorithm (11), which is in charge of calculating a surface orientation (normal vector or normal) estimate (7) for each pixel comprised by the first 2D image data, based on the previously generated depth estimate (5). In other embodiments of the invention, the calculation of the surface orientation estimate may be performed differently, not necessarily based on the previously obtained depth estimate. Next, these three estimates, (depth (5), albedo (6) and surface orientation (7)) are used to obtain a 3D reconstruction (9) of the space (1) depicted in the first 2D image data, and / or passed onto a rendering system (12) that outputs first synthesized 2D image data (13) based on illumination decline, i.e. , the fact that, in a space (1) illuminated by a direct light source (10), the illumination of a surface point comprised by said space (1) decays with a known function of the distance from said surface point to the light source (10). This first synthesized 2D image data (13) is compared with the first 2D image data obtained by the camera-light source system (2), and the comparison is parameterized by a loss function (14), which is minimized through and optimizer (15) that allows for updating the values of a set or parameters comprised by the neural network (3). This is repeated for second, third, and as many 2D image data captured by the camera-light source system (2), until the neural network (3) is capable of delivering depth (5), albedo (6) and surface orientation (7) estimates such that the synthesize 2D image data generated from said estimates (5, 6, 7) is similar enough to the corresponding one captured by the camera-light source system (2).

[0078] In this way, the 2D image data may comprise training 2D image data (4) or input 2D image data (8). Ideally, according to an embodiment of the method of the invention, first, training 2D image data (4) is used to train the neural network (3). A self-supervised training is possible in this context since only the illumination information comprised by the first 2D image data is used, together with the illumination decline principle, as a self-supervision criterion. No annotated or ground truth data is needed, as the neural network (3) learns, through this setup, to estimate depth (5), albedo (6), and surface orientation (7) very accurately. Then, once the neural network (3) is trained, input 2D image data (8) is inputted to the neural network (3) in order to obtain image data reconstruction of a space (1) depicted in said input 2D image data (8). However, and as seen in Fig. 1 as explained above, this same input 2D image data (8) can be used to keep training the neural network (3) through the bottom loop appearing in Fig. 1. This is known as test-time refinement and implies improving the accuracy of the neural network (3) estimates (5, 6, 7) each time said neural network (3) is used to obtain image data reconstruction of a space (1).

[0079] On the other hand, 2D image data may correspond to a single image or to a batch of images. These images can be processed one at a time since no multi-view or related images are needed in order to carry out the method of the invention. This is a consequence of the fact that only the illumination information comprised by the first 2D image data and the illumination decline principle are used to obtain the output of the method.

[0080] More specifically, according to an embodiment of the invention, to carry out the disclosed method, first, a photometric model is proposed. This allows to model scene illumination as coming from a single spotlight source (10) in the camera reference frame, considering that spotlight sources (10) usually emit with different intensities in each direction. Hence, a spotlight model (SLS) is adopted according to the configuration shown in Fig.2, in which, for surface point Xj with off-axis angle qjj, the radiance reads: where Oo is the maximum radiance and R(qJi) is the radial attenuation controlled by a spread factor p. It is to be noted that the light reaching the surface is subject to the inverse-square law and decays with the propagation distance from xi (light source position) to Xj. In other embodiments of the invention, a different function relating the illumination of a surface point and the distance from said surface point to the light source may be employed.

[0081] In terms of light decline, the fact that in endoscopes and other camera-light source systems (2) the camera and the light source (10) move jointly in a dark environment leads to the attenuation of the illumination being an indirect indicator of scene depth as seen from the camera. Based on this information, according to an embodiment of the invention, a rendering equation is written assuming Lambertian reflections, meaning that the light hitting the surfaces is scattered equally in all directions. Specular reflections, which are prevalent in endoscopic images, are not captured by this model but will be implemented in other steps of the method of the invention. More specifically, for each pixel, the following rendering equation can be written: where dj is the depth of the i-th pixel with image coordinates Uj, n = TT-1(UI) is the camera ray such that Xj = dj n and TT-1is the inverse projection model of the camera. The angle for the light’s incidence with respect to the surface normal n; is denoted as 0j, such that, cos(0j) = L ■ ni. pi represents the albedo of the surface at that point. The parameter g denotes the gain applied by the camera and y is the gamma correction commonly applied by cameras to adapt images to human perception. The resulting l(dj, pi, g) is the color captured by the camera.

[0082] In an embodiment of the invention, the neural network training is improved by supplying to the rendering system (12) information about the specific characteristics of the camera-light source system (2) to be used, as seen in Fig. 1. Therefore, in said embodiment, the method of the invention requires precise photometric and geometric calibration of the camera-light source system (2) or endoscope. This calibration method (16) involves the acquisition of calibration 2D image data (17) from a known calibration pattern as well as solving an optimization problem to obtain the best values for the photometric and geometric calibration parameters (18). Each endoscope has different geometric and photometric parameters, the former affecting the inverse project model TT-1and the latter impacting both the light position xi and spread R. These parameters can be estimated for a particular camera-light source system (2) or endoscope by minimizing the reprojection and photometric errors on images of a calibration target. In the present case, the auto-gain values of the camera-light source system (2) or endoscope are not known, so radiance measurements of the camera are unitless. Thus, g = 1 and Oo = 1 can be arbitrarily set to obtain up-to-scale reconstructions. Calibration errors lie between ±3 gray levels. Other possibilities, such as generic calibrations obtained for an endoscope model or group of models can be used, alternatively.

[0083] Once the calibration parameters (18) have been obtained, to train the neural network (3), a tailored loss function (14) is proposed. According to an embodiment of the invention, this loss function (14), which must be minimized by the neural network (3), contains three main terms: a photometric term (Lp), a regularization term (Ls), and a specular term (Lsp), weighted by the corresponding scalar weights (A):

[0084] L Lp -I-SLS-I-SpLSp , (4)

[0085] The photometric term is taken to be the squared distance between the original image (I) and the rendered one (I): the latter being dependent on the depth, d, albedo, p, and g. It is to be noted that because the rendering model is fully differentiable, end-to-end training can be performed. The regularization term minimizes depth gradients except in areas of high color gradients, which may correspond to depth discontinuities:

[0086] Finally, the specular term is introduced to remedy the non-desired effects that derive from the assumption of Lambertian reflections. Given image location i, the corresponding direction from the surface to the light source and the normal of the surface n, the law of reflection states that Sj = lt- 2 fti (nt■ is the specularly reflected direction. Hence, the specular loss term is chosen as: which minimizes the discrepancy between the expected specular reflection Si and the actual direction (-n) where the camera observes the reflection, resulting in pixel with high intensity, th > 0.8.

[0087] With this setup, the method of the invention takes a single image as input, which makes 3D reconstruction (9) solely from pixel colors an under-constrained problem. A change in the brightness of a pixel can be due to changes in depth, albedo, camera exposure or surface normal. For example, if a given pixel is very bright, it can be because the pixel is close to the camera-light source system (2); the surface has a different albedo, resulting in more light being reflected; the surface normal is aligned to the camera-light source system (2), which increases the reflected light; the camera exposure and digital gain have been increased, which impacts brightness values in the whole image. Given the albedo at each surface point and the camera auto-gain, these ambiguities can be resolved. However, in medical endoscopy, true albedos are unknown, and auto-gain might be not provided by the hardware manufacturer.

[0088] However, endoscopy images, among others, exhibit a limited range of colors, with brighter tones being present in close areas and darker tones in deeper regions. Consequently, significant correlation between albedo and the chromatic attributes is hypothesized, namely Hue and Saturation in the HSV color space, as well as between depth and the Value Channel. In this way, the palette of colors that can be explained by the albedo estimator is constrained and the disentanglement between depth and albedo is enhanced by setting V = 100 for all albedo values. Hence, to predict the albedo map, the neural network (3) predicts just two channels per pixel, for Hue and Saturation, and assumes Value to be one to convert to the RGB space, in which the loss is formulated.

[0089] The proposed neural network (3) outputs depth (5) and albedo (6) estimates thanks to an encoder-decoder architecture (depicted generically in Fig. 4A) that may be implemented in two different ways. The first one (Fig. 4B) is a ll-net with two decoders and skip connections, with a ResNet18 serving as the backbone. The second one (Fig. 4C) relies on using a Resnet-50 (DPT-Hybrid) and two decoders that reassemble the tokens and apply attention heads. In both versions, to compute the surface orientation estimate (7) at any given pixel, a convolution kernel with six-neighborhoods (N, NE, E, S, SW, and W) in the depth estimate (5) is used. Six triangles are defined using the central pixel as reference, with each triangle having its own surface orientation estimate (7). The surface orientation estimate (7) of the central pixel is computed as the average of the surface orientation estimates (7) of the triangles weighted by their area. The use of six neighbors allows for triangles during the convolution pass to speed up computation.

[0090] With the aforementioned setup, the method of the invention has been put to test on three endoscopy datasets:

[0091] Synthetic colon: a simulated colon as captured by an Olympus CF-H190L endoscope consisting of a fish-eye camera and a spot-light source (10), both calibrated has been used. This contrasts with other synthetic datasets that simulate arbitrary camera and illumination configurations, typically pinhole cameras with no or arbitrary distortion and ideal light sources (10) with no radial falloff. Images are rendered using ray-casting techniques, in which the colon’s geometry and albedo are defined by a triangle mesh obtained from a Computerized Tomography (CT) scan of a real colon. Global illumination effects are ignored and Lambertianity is assumed, so there are no specular reflections. The influence of these two effects will be assessed in the two other datasets. This synthetic dataset is composed by 1620 fish-eye RGB frames annotated with per-pixel albedo, depth and surface orientation split into 1168 images for training and 452 images for test.

[0092] Colonoscopy 3D video dataset (C3VD): C3VD contains real images recorded in a phantom with ground-truth depth. The images have been captured by a real Olympus CF-HQ190L endoscope in a phantom silicone model of a human colon. The data is annotated with ground-truth depth and surface orientation by applying 2D-3D registration of the 3D phantom models. It is assumed that the only light source available is in the endoscope. Finally, this dataset includes a geometrical calibration based on the Scaramuzza model. C3VD provides a good compromise between realism (real endoscope, global illumination effects and specular highlights) and ground-truth labels for quantitative evaluation. Of the 10,088 images available, 7,200 are used for training and 2,888 for testing.

[0093] EndoMapper dataset: this represents the most challenging data, as it contains real colonoscopy and gastroscopy procedures inside the human body, performed by endoscopists on a day-to-day basis. Here real textures such as veins, blood and dirt, and other effects such as blur, water and frames very close or even hitting the mucosa can be found. Foam and bubbles are indeed very common in endoscopy images and are usually ignored. The method of the invention is capable of disentangling these as part of the albedo and not of the depth. Before processing the dataset, a manual inspection of the selected sequences is performed and occluded and excessively blurred frames are eliminated. Results are reported using a median-based alignment for all methods, even those supervised with real-scale depth, for fairness. In all experiments, a comparison against models that use depth supervision and multi-view self-supervision is carried out. For depth supervision, the two previously mentioned architectures are used: ll-Net with L1 loss as a representative of convolutional architectures and DPT-Hybrid as a state-of-the-art representative of transformer-based models, learning inverse depth with a scale invariant loss. For a fair comparison, the performance of the method of the invention is also evaluated using the same U-Net and DPT-Hybrid architectures. The ll-Net is pre-trained on ImageNet dataset. For DPT-Hybrid, an initialization with the author-provided weights for encoder and depth decoder is done. The albedo decoder is trained from scratch. During training, a smoothing weight As= 0.1 in Eq. (4) is selected as well as a learning rate of 10"4for the optimizer (15). In the synthetic dataset, the neural network (3) of the invention is trained with Asp = 0, as synthetic dataset has no specular reflections. In C3VD and EndoMapper, Asp = 1 is used.

[0094] As previously stated, the use of the method of the invention enables single-view selfsupervision, so that depth predictions can be continuously refined online, obtaining much more accurate reconstructions. In the results denoted as “TTR”, online test-time refinement (TTR) for each test image separately during N = 20 optimization steps, using the loss function (14) in Eq. (4), as in training time, is performed. To mitigate the risk of catastrophic forgetting, the original model trained in the train split after TTR for each image is loaded again. The results show how TTR significantly improves the metrics with respect to the application of the same method without TTR for ll-Net and DPT-Hybrid architectures. Remarkably, TTR even outperforms the metrics achieved by Depth ground truth (GT) supervision. Fig. 5 shows the improvement given by TTR in the neural network (3) prediction of depth (5), surface orientation (7) and albedo (6) estimates and overall, in the 3D reconstruction (9). Inference time is ~ 5 ms for ll-Net and ~ 22 ms for DPT-Hybrid on a NVIDIA GeForce RTX 3090.

[0095] Regarding the quantitative results on synthetic and phantom synthetic colon, the depth and normal metrics for a ll-Net supervised with Depth GT, and with the self-supervised ll-Net architecture according to the method of the invention are similar. This is notable, as selfsupervision is consistently reported in the art to underperform with respect to depth supervision, and suggests that illumination decline provides a very strong self-supervisory signal in endoscopies. Furthermore, most of the error distribution is lower for light self- supervision and only a small fraction of large errors is better with depth supervision. In terms of surface orientation with light self-supervision there is an overall lower error, due to the lower errors in most pixels.

[0096] Concerning the C3VD phantom results, the invention’s self-supervised architectures ll-Net and DPT with TTR outperform supervision with Depth GT in median absolute error (MedAE), while the rest of the metrics are very close. As in the case of the synthetic dataset, this is a remarkable result because self-supervised architectures typically lag supervised ones in single view depth estimation. The root-mean-square error (RMSE) obtained with the method of the invention is, however, worse, but this fact, together with the fact that the MedAE obtained with the method of the invention is better, suggests that the errors associated to the method of the present invention are better in most of the distribution, with just a few regions with large errors where Depth GT supervision is able to offer an advantage.

[0097] On the other hand, the metrics on the quality of the rendered image suggest the strength of the self-supervision signal. Moreover, there is a significant improvement of these metrics for the TTR case. Multi-view self-supervised baselines have a poor performance, worse in comparison than results in other datasets. This could be due to the weak textures and changing lighting in the colonoscopy images, resulting in noisy estimations for relative motion and uninformative photometric residuals. Being single image, the invention’s approach is impervious to such difficulties.

[0098] Regarding the domain-shift problem, as synthetic-to-real is common in endoscopies to address the lack of ground-truth depth for supervision, XDCycleGAN has also been evaluated as a baseline. It is to be noted that the domain shift is still affecting the results. The invention’s single-view self-supervision enables training in the target domain, and hence removes completely the domain shift, achieving significantly lower errors. Specifically, a ll-Net model with Depth GT supervision and a ll-Net model with light selfsupervision are trained in the synthetic dataset and their performance in the synthetic and C3VD test sets are evaluated. The domain shift affects all metrics significantly. Interestingly, the model trained with light self-supervision and without TTR generalizes significantly better to the C3VD data, as the invention’s self-supervised model is closer to the physical phenomena than Depth GT supervision. Again, it is to be noted that single-view selfsupervision removes completely the domain shift effect, as models can be trained directly in the target domain. Very remarkably, the performance of the method of the invention with domain shift after TTR matches the performance of the models without domain shift.

[0099] Finally, in order to better demonstrate the effects of each term in the loss function (14), an additional analysis has been carried out by ablating the terms of said loss function (14). The smoothness prior (Lsterm) is remarkably beneficial for obtaining both depth (5) and surface orientation (7) estimates. When the information of the specular reflections is not considered (no Lspterm), worse results are obtained. Adding this new loss term, all the depth and normal metrics improve, especially in the median error, which outperforms the supervised and matches that obtained in the simulation experiment. Still, the depth MedAE and RMSE are slightly higher than those of the baseline due to the far spurious points.

[0100] Qualitative results in real endoscopy, i.e., obtained with the EndoMapper dataset, can be seen in Fig. 6. Some details are recovered very accurately, such as the surface orientation estimates (7) showing clearly the tubular shape; the depth estimates (5) reflecting the discontinuities in the Haustras; the albedo estimates (6) capturing the blood vessels, in particular in the 5th column; and the bubbles and fluids colors (gray intensities in Fig. 6) in the 6th and 7th columns, which make the 3D reconstruction (9) of these bubbles and fluids very plausible. Unfortunately, there is no ground-truth data available for this dataset, which prevents from presenting quantitative results. Nevertheless, visual inspection of the available results hints that the strengths of the method of the invention demonstrated quantitatively with other datasets will carry over to truly realistic scenarios like this one.

[0101] As a summary, a single-view self-supervision method for obtaining depth (5), albedo (6) and surface orientation (7) estimates for the case of a single spotlight source (10) co-located with a monocular camera (camera-light source system (2)) is proposed. The specific selfsupervised learning setup that models the quadratic light decline and enables selfsupervised learning is developed in the particular case of endoscopy imaging. The results of applying the method of the invention in said case show that it outperforms multi-view selfsupervision and synthetic-to-real transfer and matches the performance of fully supervised approaches. Not only that, its training and test-time refinement setup is significantly simpler: it only requires a reasonable camera-light source system (2) calibration and does not require camera motion estimation nor ground-truth labels nor realistic simulations, all of them challenging in endoscopies. This unlocks, from a practical point of view, relevant potential applications in the medical domain and in many others with similar illumination characteristics, as previously mentioned.

Claims

CLAIMS1.- Computer implemented method for obtaining data of a space (1) illuminated by a light source (10), said light source (10) being comprised by a joint camera-light source system (2), and the method comprising performing the following steps: a) providing a set of training joint camera-light source system 2D image data (4) of at least one space (1) comprising illumination information; b) delivering first training 2D image data (4) to a neural network (3); and characterized in that the method further comprises performing the following steps: c) obtaining, for each pixel of the first training 2D image data (4), a depth estimate (5) an albedo estimate (6) and a surface orientation estimate (7) corresponding to each pixel of the first training 2D image data (4); d) training a neural network (3) by using the depth (5), albedo (6) and surface orientation (7) estimates obtained in the previous step and by applying the illumination decline principle; e) delivering first input 2D image data (8) of a space (1) illuminated by a light source (10) comprised by a joint camera-light source system (2) to the trained neural network (3); f) obtaining, using the trained neural network (3), a final depth estimate (5’), a final albedo estimate (6’) and a final surface orientation estimate (7’) for each pixel of the first input 2D image data.2.- Method according to the preceding claim, wherein, after step f), the following step is performed: g) obtaining a 2D or 3D reconstruction (9) of the space (1) depicted in the first input 2D image data (8) by using the depth, albedo and surface orientation estimates obtained in step f).3.- Method according to any of the preceding claims, wherein the set of training 2D image data (4) comprises one or many, monocular or stereo, related or unrelated, consecutive or nonconsecutive, real or simulated images.4.- Method according to any of the preceding claims, wherein the neural network (3) comprises parameters with an initial value and wherein the training of the neural network (3) in step d) is carried out by:- obtaining, from the depth (5), albedo (6), and surface orientation (7) estimates obtained in step c), first synthesized 2D image data (13) by using a rendering system (12) based on illumination decline;- updating the parameter values of the neural network by minimizing a loss function (14) that comprises the differences between the first training 2D image data (4) and the first synthesized 2D image data (13); and- repeating steps b)-d) iteratively for different training 2D image data (4) until the neural network is trained;5.- Method according to any of the preceding claims, wherein the training of the neural network (3) in step d) is further based on geometric and / or photometric calibration parameters (18) of a joint camera-light source system (2).6.- Method according to any of the preceding claims, wherein steps e), f) and / or g) are carried out in real time.7.- Method according to any of the preceding claims, further comprising performing, after step e), the following step: e1) re-training the neural network (3) using as first training 2D image data (4) the input 2D image data (8) delivered in step e).8.- Method according to any of the preceding claims, wherein the training of the neural network (3) in steps d) and / or e1) comprises a self-supervised training.9.- Method according to any of the preceding claims, wherein steps e)-f) are repeated for each of the frames comprised by a monocular or stereo video sequence, using said frames as 2D input image data, and wherein the depth (5), albedo (6) and surface orientation (7) estimates corresponding to each of said image data are fused to obtain:- the joint camera-light source system (2) position and orientation in each image data; and / or- a 3D reconstruction (9) of an extended portion of the space (1) appearing in said image data.10.- Method according to any of the preceding claims, further comprising, after step g), displaying in a display at least one of the following:- the final depth estimate (5’);- the final albedo estimate (6’);- the final surface orientation estimate (7’); and- the 2D or 3D reconstruction (9) of the space obtained in step g).11.- Method according to any of the preceding claims, wherein, after step f), the resulting depth (5), albedo (6) and surface orientation (7) estimates are processed by using digital image processing means such that:- different structures comprised by the space (1) are identified;- measurements of lengths, areas, volumes and / or albedos of structures appearing in the image data are obtained; and / or- the percentage of observed structures and / or the percentage of space occluded by dirt, liquids or other obstacles is calculated.12.- Method according to any of the preceding claims, wherein the space (1) is one of the following:- a hollow organ;- a cavity of the body;- a region of the seabed and / or other underwater space;- a cavity comprised by a rigid structure;- a pipe;- a region comprised by a sewage system; and- an underground environment.13.- System comprising a joint camera-light source system (2) adapted to acquire 2D image data, the joint camera-light source system (2) being connected to computing means, and characterized in that the computing means comprise hardware and / or software means adapted to perform a method according to any of claims 1-11.14.- System according to the preceding claim, wherein:- the joint camera-light source system (2) is comprised by an endoscope, a capsule endoscope, a borescope, an augmented / virtual reality device, a mobile device, a wearable device, a robotic device and / or a vehicle;- the system comprises optionally additional sensors adapted to obtain sensor data of a space (1) and / or sensor data of the joint camera-light source system (2);- the processing means are adapted to provide combined data of said space (1) by combining the sensor data, the final depth (5’), albedo (6’) and surface orientation (7’) estimates, and, optionally, the joint camera-light source system (2) position and orientation. 15.- Computer program comprising instructions which, when the program is executed by computing means, cause the computing means to carry out a method according to any of claims 1-12.

Images