A computer vision method and system

Abstract

Three dimensional object reconstruction for computer vision, comprising: estimating an object’s shape based on images captured from different viewing directions using multiple point light sources; inputting the estimated shape to a neural network that computes an object representation; using the neural network to identify points on the object’s surface and sampling the points along one of the viewing directions; rendering each point to generate the 3D reconstruction, wherein the rendering generates loss and the neural network is trained based on the loss. The object representation may be a surface, depth map, height map, neural function, signed distance field or 3D geometric representation. Neural network training may involve loss backpropagation to adjust weights. A rendering neural network may use surface normals and intensity and learn an object material. Two shapes may be estimated for different viewing angles and concatenated. Surface normals may be estimated based on shape estimates and partial differentiation of neural network weights. A rendering neural network may use surface normals and intensity and learn an object material. Loss may be based on difference between the 3D reconstruction and the object. Sampling positions may be points where incident light rays, that pass through pixel centres, intersect the surface representation.

Description

FIELD The present disclosure relates to a computer vision method and system. In particular, the present disclosure relates to performing a three-dimensional reconstruction of an object. BACKGROUND Computer vision tasks include tasks such as acquiring, analysing, and processing image data. Some computer vision systems can perform three-dimensional reconstruction of objects by performing these tasks. However, photometric stereo is a classical problem in computer vision. Photometric stereo is a technique for estimating local geometrical features such as surface normal of the object, depth of the object, and / or the like. Photometric stereo exploits the relation between the orientation of a surface (e.g., in relation to the light source and the observer) on the object and the amount of light reflected by the surface. Photometric stereo may involve retrieving the three-dimensional shape of the object from the light reflected off the object. This can often be challenging, especially, due to the type of reflection and its non-linear dependence on material properties (e.g., material properties of the object). Some existing methods use single view photometric stereo for estimating the shape of the object. The three-dimensional shape estimated using such methods are not always accurate. Although recent advances have been made towards three-dimensional reconstruction of objects, the quality and practical usefulness of the estimated shape have been less convincing. Accordingly, there is a need for accurate three-dimensional reconstruction of objects. Additionally, there is an unmet need for accurate three-dimensional reconstruction of objects and / or scene that have no colour texture or geometric feature. SUMMARY In one aspect, a computer vision method is provided for generating a three dimensional reconstruction of an object, the method comprising: receiving at least one estimated shape of an object, the at least one estimated shape being generated based on a capture of one or more images from a corresponding viewing direction using a plurality of point light sources; providing the at least one estimated shape of the object as input to a first neural network, the first neural network being initialized to compute a representation of the object; sampling one or more points along the corresponding viewing direction using the first neural network to identify a plurality of points on a surface of the object; and rendering each point of the identified plurality of points to generate a three dimensional reconstruction of the object, wherein the rendering produces at least one loss and the first neural network is trained based at least in part on the at least one loss. In one example, the representation is a surface representation of the object. In some variations, the surface representation of the object is a heightmap. Sampling the one or more points may include fitting the heightmap to the at least one estimated shape of the object. In some variations, the surface representation of the object is a neural function, wherein the neural function is represented by the first neural network. In some variations, the at least one estimated shape of the object comprises at least one estimated surface of the object, and wherein first neural network maps a point on a plane to a height from the surface of the object. In some variations, the first neural network may map a point on a surface to a surface depth of the object. In another example, the representation is a three-dimensional geometrical representation of the object. In some variations, the at least one estimated shape of the object comprises at least one estimated three-dimensional geometry of the object, and wherein the first neural network maps a point on a three-dimensional space to a Signed Distance Field of a point on the object. The method may further comprise computing a surface representation of the object from the computed three-dimensional geometrical representation of the object. In some variations, the three-dimensional geometrical representation of the object is a neural function, wherein the neural function is represented by the first neural network. The method may further comprise performing backpropagation of the at least one loss to adjust one or more weights of the first neural network. In one example, the rendering is implemented by a second neural network. The method may further comprise performing backpropagation of the at least one loss to adjust one or more weights of the second neural network. The second neural network is further configured to learn a material of the object. In one example, the rendering may further comprise rendering at least one of a surface normal and an intensity at each point of the identified plurality of points. In one example, the at least one estimated shape is generated using a third neural network. In one example, receiving the at least one estimated shape may comprise receiving a first estimated shape and a second estimated shape, the first estimated shape being generated based on a first set of images captured from a first viewing direction and the second estimated shape being generated based on a second set of images captured from a second viewing direction. The method may further comprise: concatenating the first estimated shape and the second estimated shape to generate a concatenated estimated shape; and providing the concatenated estimated shape as input to the first neural network. In examples in which the representation of the object is the surface representation of the object, the method may further comprise computing at least one surface normal of the object based on a partial differentiation of one or more weights of the first neural network and the at least one estimated shape of the object. The rendering may include rendering the computed at least one surface normal at each point of the identified plurality of points. In one example, sampling the one or more points along the corresponding viewing direction includes determining points at an intersection of a ray of incident light from an image capturing device that captures the one or more images and the surface of the object, and sampling the determined points to identify the plurality of points on the surface of the object, wherein the ray of incident light passes through a centre of each pixel. The method may also comprise determining a rendering loss based on a difference between the three dimensional reconstruction of the object and the actual object. A carrier medium may be provided carrying computer readable instructions adapted to cause a computer to perform the method described above. In one aspect, a system for three dimensional reconstruction of an object may comprise an interface and a processor: the interface having an image input and configured to receive one or more images of an object from a corresponding viewing direction captured using a plurality of point light sources; the processor configured to: receive at least one estimated shape of the object, the at least one estimated shape being generated based on the one or more images of the object; provide the at least one estimated shape of the object as input to a first neural network, the first neural network being initialized to compute a representation of the object; sample one or more points along the corresponding viewing direction using the first neural network to identify a plurality of points on a surface of the object; and render each point of the identified plurality of points to generate a three dimensional reconstruction of the object, wherein the rendering produces at least one loss and the first neural network is trained based at least in part on the at least one loss. BRIEF DESCRIPTION OF FIGURES Systems and methods in accordance with non-limiting examples will now be described with reference to the accompanying figures in which: Figure 1 is a schematic of an example system for three-dimensional reconstruction of an object; Figure 2 is a schematic showing an arrangement of a pair of cameras and light sources for performing three-dimensional imaging of an object; Figure 3 shows an example of the binocular photometric stereo data capture set up discussed in Figure 2; Figure 4 is a flow diagram of the method for 3D Reconstruction according to an example; Figure 5 shows an example implementation of a portion of the method described in Figure 4; Figure 6 shown an example implementation of a portion of the method described in Figure 4; Figure 7 illustrates an evaluation of the methods and systems disclosed herein on DiLiGenT dataset; Figure 8 depicts plots of the mean and median shape error for objects from the DiLiGenT dataset; Figure 9 illustrates an evaluation of the methods and systems disclosed herein on LUCES-Stereo dataset. DETAILED DESCRIPTION Disclosed herein are methods and systems of computer vision. More specifically, disclosed herein are computer vision methods and systems for three-dimensional reconstruction of an object. As used herein, the term “object” may denote what is being imaged. However, it should be readily understood that the term “object” may refer to a plurality of objects, a scene, a combination thereof, and / or the like. “Accurate three-dimensional reconstruction” as used herein may refer to an accurate reconstruction of the shape of the object. For example, “accurate three-dimensional reconstruction” may refer to realistic reconstruction of the shape of the object. Accurate three-dimensional reconstruction of an object can be challenging. Such challenges are often rooted in problems relating to photometric stereo. Some existing methods leverage multiple views (e.g., by capturing images from multiple views using multiple cameras) to generate a three-dimensional reconstruction of the object. For example, some existing methods include image-based models that retrieve depth between the object and a camera of an image capturing apparatus. The colour texture at each pixel may be projected depending on the retrieved depth so as to generate a three-dimensional reconstruction of the object. However, textures are dependent on light reflection, and therefore, realistic three-dimensional reconstruction of the object may not always be possible. Furthermore, colour texture relies on material properties of the object. That is, a same image of the object may appear substantially different from different views. Some existing methods combine three-dimensional scanning techniques such as for example, laser scanner and structured light one. However, these approaches may require merging of data from different sensors. Merging data can lead to a number of challenges such as loss of accuracy, limited resolution of the three-dimensional reconstruction, noisy and incomplete image data (e.g., scans) of shiny and metallic objects, etc. Additionally, merging is often performed at a per-pixel level which can be an arduous task. Until recently, methods that leverage multiple views did not include a learnable component (e.g., neural network). Such methods often lead to errors. Recent advances have been made to leveraging multiple views by including a learnable component. But, such learnable component may be configured to learn textured volume of the object. Thus, these methods are not efficient and are not conditioned to perform surface optimization procedure. More specifically, the technology described herein includes a learnable component to perform surface parameterization. This enables the technology described herein to implement another neural network to render the three-dimensional reconstruction of the object and backpropagate losses from data, thereby making the technology described herein efficient. This is especially possible because the neural surface is fully smooth and differentiable unlike the existing methods with learnable components. Additionally, such methods are often not feasible when implemented for applications such as for example, robotic interaction, conveyor belt scanning, etc. In particular, implementing these methods may not be feasible owing to long capture times of images and precise camera pose calibration. Furthermore, such methods are more fragile, less elegant, and significantly more complex than the technology disclosed herein. The technology disclosed herein can generate an accurate (e.g., realistic) three-dimensional reconstruction of an object having no colour texture or geometric feature. In contrast to the existing methods, the technology disclosed herein allows for better conditioned and efficient surface optimization procedure. In one example, the technology disclosed herein may comprise receiving at least one estimated shape of an object. The estimated shape may be generated based on a capture of one or more images from a corresponding viewing direction using a plurality of point light sources. For example, to generate the estimated shape of the object, images of the object may be captured in a controlled manner. For instance, a plurality of point light sources (e.g., LEDs) may be used in a calibrated fashion. The images may be captured using a binocular photometric set up. Put differently, an image capturing apparatus may include two cameras to capture images of the object under controlled lighting conditions (e.g., using the plurality of point light sources). Capturing images in such a controlled and calibrated manner enables the technology disclosed herein to generate an accurate estimated shape of the object with general reflectance. The at least one estimated shape of the object is provided as input to a first neural network. In some variations, the first neural network is a deep neural network. The first neural network may be initialized to compute a representation (e.g., a surface representation of the object and / or a three-dimensional geometrical representation of the object) of the object. More specifically, the technology disclosed herein formulates the task of generating a three-dimensional reconstruction of the object as learning of a differentiable surface representation and texture representation and / or three-dimensional geometrical representation and texture representation. This may minimise discrepancies between local geometrical features (e.g., surface normal) that are estimated for two viewing directions using varying lighting conditions. This may also minimise discrepancies between rendered surface intensity (e.g., rendered as discussed below) and observed images. As an example, the surface of the object may be represented as a neural heightmap where heights of the points on a surface of the object may be computed using the first neural network. In comparison to existing methods that utilize volume-based representations, representing the surface as neural heightmap may allow for better conditioned and efficient surface optimization procedure. In some variations, the first neural network may be initialized to compute a three-dimensional geometrical representation of the object. For example, the geometry of the object may be represented as neural Signed Distance Field where the Signed Distance Fields of the points on a geometry of the object may be computed using the first neural network. In such variations, a surface representation of the object may be derived from the three-dimensional geometrical representation of the object. The technology disclosed herein may further comprise sampling one or more points along the corresponding viewing direction using the first neural network to identify a plurality of points on a surface of the object. Each point of the identified plurality of points may be rendered to generate a three-dimensional reconstruction of the object (e.g., by rendering surface normal, surface depth, a three-dimensional representation of the object, a combination thereof, and / or the like). Rendering the identified plurality of points may produce one or more losses as further described herein. In one example, these losses may be backpropagated to the first neural network. The first neural network may be trained based on these losses. In some examples, the identified plurality of points may be rendered by implementing a second neural network. This second neural network may be configured to learn a material of the object (e.g., based on rendered loss of intensity). Put differently, instead of predicting individual per point intensity, the second neural network is used to render the identified plurality of points based on a learned material of the object and on albedo parameters. Since the second neural network learns the material of the object based on robust intensity loss, the technology disclosed herein is less susceptible to intensity overfitting than direct intensity matching loss. In particular, robust intensity loss preserves the peak of the specular lobe, and hence is less susceptible to intensity overfitting and limitations of the renderer quality than direct intensity matching loss. Accordingly, in some examples, the technology described herein may include estimating perview based shape of the object by using per view estimated photometric stereo normals, using the estimated per-view based shape of the object to initialize a neural heightmap network (and / or a neural signed distance field network) guided by estimated pixel-wise normal and depth, and follow by robustly fitting of the initialised neural heightmap to image intensity and estimated normal information. The technology disclosed herein may be used in various applications. For example, the technology disclosed herein may be used for accurate quality control measurement of shiny object. As another example, the technology disclosed herein may also be used for depth perception of robotic applications. As yet another example, the technology disclosed herein may be used for biometric three-dimensional feature extraction (e.g., fingerprint patterns). Figure 1 shows a schematic of a system which may be used for capturing three-dimensional (3D) image data of an object, and reconstructing the object. The 3D image data of object 10 is captured using apparatus 11. Further details of apparatus 11 will be provided in reference to Figure 2. The 3D image captured by the apparatus 11 may be provided to a computing device 12, where it is processed. In some variations, the computing device 12 may include an interface to receive the 3D image captured by the apparatus 11. In some examples, the apparatus 11 may be communicatively coupled to the computing device 12 via a network (e.g., Internet, Local Area Network (LAN), Wider Area Network (WAN), and / or the like). Some non-limiting examples of computing device 12 include computers (e.g., desktops, personal computers, laptops, etc.), tablets and e-readers (e.g., Apple iPad®, Samsung Galaxy® Tab, Microsoft Surface®, Amazon Kindle®, etc.), mobile devices and smart phones (e.g., Apple iPhone®, Samsung Galaxy®, Google Pixel®, etc.), etc. The computing device 12 comprises at least one processor. The processor may be any suitable processing device configured to run and / or execute a set of instructions or code, and may include one or more data processors, image processors, graphics processing units, digital signal processors, and / or central processing units. The processor may be, for example, a general purpose processor, a Field Programmable Gate Array (FGPA), an Application Specific Integrated Circuit (ASIC), and / or like. The processor may be include instructions that when executed, may cause the processor to perform one or more of the following: (1) generating at least one estimated shape of the object based on a capture of one or more images from one or more viewing directions using a plurality of point light sources; (2) initializing a first neural network to compute a representation (e.g., surface representation and / or three-dimensional geometrical representation) of the object; (3) using the at least one estimated shape as input to the first neural network to compute the representation (e.g., surface representation and / or three-dimensional geometrical representation) of the object; (4) sampling the one or more points along the one or more viewing directions using the first neural network to identify a plurality of points on a surface of the object; or (5) rendering each point of the identified plurality of points to generate a 3D reconstruction of the object (e.g., rendering may be based on implementing a second neural network). Figure 2 shows an exemplary arrangement of apparatus 11 of Figure 1 which can be used for binocular photometric stereo. More specifically, systems and methods described herein generate an estimated shape of the object using image data (e.g., images of the object) that was captured in a controlled manner from two different views. Figure 2 shows an exemplary variation of the controlled set up for capturing the image data for generating the estimated shape. Figure 2 shows a mount that holds a first camera 21 and a second camera 22. The mount further holds a plurality of light sources 23 in a fixed relationship with one another and the first camera 21 and the second camera 22. This arrangement of apparatus 20 allows the first camera 21, the second camera 22 and the plurality of light sources 23 to be moved together maintaining constant separation from each other. Although in Figure 2, the light sources 23 are provided surrounding cameras 21 and 22, it should be readily understood that the light sources 23 and the cameras 21 and 22 may be provided in any suitable arrangement. Cameras 21 and 22 are used together with light sources 23 to obtain binocular photometric stereo data of object 10. Individual light sources of the plurality of light sources 23 may be activated one after another to allow cameras 21 and 22 to capture binocular photometric stereo data. In a particular arrangement of apparatus 11, a FLEA 3.2 Megapixel camera provided with a 8mm lens may be used as cameras 21 and 22. The cameras 21 and 22 may have a 8mm lens and may be rigidly attached to a printed circuit board. The printed circuit board may further comprise 15 white bright LEDs, used as light sources 23, arranged in a co-planar fashion with the image plane and provided surrounding cameras 21 and 22 at a maximum difference of 6.5 centimetres. Apparatus 11 may be used to capture binocular photometric stereo images of an object 10. The object 10 is positioned in front of the cameras 21 and 22 within the depth range of the cameras 21 and 22. The depth range, also referred to as the depth of field (DOF), is a term used to denote the range of distances between each of the cameras 21 and 22 and the object 10 at which the object 10 is in focus. If an object 10 is too close or too far to the cameras 21 and 22 outside the respective depth range of the cameras 21 and 22, the object gets out of focus and details cannot be resolved. For example, the cameras 21 and 22, equipped with 8mm lens, may have depth range of between 5 and 30 centimetres. Light sources 23 are individually activated one after the other to allow each of the cameras 21 and 22 to capture photometric stereo data of the object under different lighting conditions. This is achieved by switching on just one light source at a time. For example, the apparatus 11 may comprise 15 LEDs and thus a set of 15 photometric stereo images may be captured by each of camera 21 and 22. In photometric stereo, each of the light sources 23 are individually activated upon the object 10 to capture photometric stereo data of the object under different lighting conditions. This is achieved by switching on just one light source at a time. The amount of light reflected from the known surface or object onto each pixel of the camera 21 and 22 is then measured. Figure 3 shows an example of the binocular photometric stereo data capture set up discussed in Figure 2. The mount 30 holds camera 31 and camera 32. The mount 30 also holds a plurality of point light sources (e.g., LEDs) that are fixed as described in Figure 2. The cameras 31 and 32 capture images of the object 10 under different light conditions. Figure 4 is a flow chart showing an outline of an example method 400 of generating a 3D reconstruction of an object. At step 401, the method 400 may include receiving at least estimated shape of the object. The estimated shape of the object may be generated based on capture of one or more images from a corresponding viewing direction using a plurality of point light sources. For example, the (image) data capturing set up shown in Figure 2 and Figure 3 may be used to capture the one or more images. In particular, the images may be captured using two cameras (e.g., camera 21 and camera 22) from two viewing directions. Plurality of point light sources (e.g., bright LEDs) (e.g., plurality of point light sources 23) may be used to capture the one or more images in a controlled manner in varying lighting conditions. For instance, if the data capturing set up comprises 15 point light sources, then each point light source may be switched on one at a time. At each time when a point light source is switched on, a first image may be captured by a first camera (e.g., camera 21) from a first viewing direction and a second image may be captured by a second camera (e.g., camera 22) from a second viewing direction. Therefore, with 15 point light sources, 15 images of the object may be captured by a first camera (e.g., camera 21) from a first viewing direction and 15 images of the object may be captured by a second camera (e.g., camera 22) from a second viewing direction. These images may be used to estimate local geometrical features such as for example, surface normals. The estimated local geometrical features may be used to generate the at least one estimated shape of the object. Local geometrical features may be estimated in any suitable manner. As an example, the local geometrical features may be determined by implementing a convolutional neural network (CNN) based approach as described in Fotios Logothetis, Roberto Mecca, Ignas Budvytis, and Roberto Cipolla, A cnn based approach for the point-light photometric stereo problem, Int. Journal of Computer Vision, 2022 (referred to herein as “PX-Net approach”). In this approach, per view normal maps may be computed using the CNN. This approach offers a general near-field network that obtains quality normal maps for the calibrated near (and far) field photometric stereo setting. In addition, an initial estimate of the geometry (as a depth map per view) may also be obtained using this approach. The normal maps are fused into a unified surface by using normal guidance as well as feedback from the one or more images captured at step 401. For instance, in this approach, a per-pixel CNN is trained to predict surface normals from reflectance samples. The depth may be computed by integrating the normal field in order to iteratively estimate light directions and attenuation which may be used to compensate the input images to compute reflectance samples for the next iteration. Although, the PX-Net approach is described for generating the estimated shape, it should be readily understood that any suitable approach that uses an image capturing set up with point light source such as for example, the image capturing setup (e.g., set up in Figure 2 and Figure 3) described herein may be used to generate the estimated shape of the object. For example, the estimated shape may be generated using a Computer Aided Design (CAD) model. In examples in which more than one estimated shape is generated (e.g., PX-Net approach), the estimated shapes may be concatenated together to produce a single estimated shape. For instance, estimated shapes from two or more viewing directions may be concatenated together by fusing the local geometrical features into a unified surface. At step 403, the method includes providing the at least one estimated shape (received at step 401) as input to a first neural network. The first neural network may be initialize to compute a representation (e.g., surface representation and / or three-dimensional geometric representation) of the object. In some examples, the first neural network may be initialized to compute a surface representation of the object. In other examples, the first neural network may be initialized to compute a three-dimensional geometrical representation (e.g., 3D geometrical representation such as a signed distance field) of the object. Put differently, the surface representation and / or three-dimensional geometrical representation may be a learned representation. The estimated shape (e.g., estimated surface and / or estimated three-dimensional geometry) may be provided as input to the learnable component (e.g., the first neural network). Thus, the surface representation and / or three-dimensional geometrical representation may be defined as a function of the estimated shape where the function may be a learnable function configured to optimize its weights. For examples in which the first neural network is initialized to compute a surface representation, a surface of the object may be expressed as a continuous depth map zs = F{xs,ys\ wherein the subscript s denotes surface coordinates. More specifically, the first neural network may map a point (e.g., any point) on a plane (e.g., x, y plane) to a surface depth of the object. Surface depth and / or surface height as used herein may signify a height from a point on the surface of the object. Put differently, surface depth and / or surface height may be a perpendicular distance from a point on the surface of the object. Therefore, the surface depth of a plurality of points on the object may be represented as a neural function at step 403. The neural function is represented by the first neural network. For the image capturing set up shown in Figure 2 and Figure 3 ( / .e., binocular stereo cameras), this coordinate system may be chosen as the average between the two cameras (i.e., the rectified stereo system). A roto-translation Rctc may be used to convert the surface coordinates (e.g., denoted by the subscript s) to the original camera coordinates (e.g., denoted by the subscript c ). For instance as: [xs,ys,zs]T = Rc • [xc,yc,zc]T + tc (1) The unknown function F is the first neural network and the objective is to optimise its weights. In some variations, the surface normal of the object may be computed based on a partial differentiation of one or more weights of the first neural network and the at least one estimated shape. For instance, in some variations, the first neural network may comprise SIREN architecture (e.g., as described in Vincent Sitzmann, Julien N.P. Martel, Alexander W.Bergman, David B. Lindell, and Gordon Wetzstein, Implicit neural representations with periodic activation functions, In Proc. NeurlPS, 2020) which is a multilayer perceptron with sinusoidal activation functions. Such an architecture may guarantee that the surface is infinitely differentiable and thus the normal integration problem may correspond to solving a partial differential equation. Put differently, the surface normal may be computed as: JdF dF . i7 ns °c —,-1 (2) Ldxs dys J As seen in Equation 2, the proportionality may be due to the unit magnitude constraint of normal vector. In addition, partial derivatives may be computed with automatic differentiation and thus are a function of the network weights. In some variations, the first neural network may be trained based on rendering loss (as described below) using backpropagation. The rendering loss may be determined using the one or more images captured at step 401. In some variations, the surface representation may be a heightmap (e.g., a map of a surface depth) of the object. For examples in which the first neural network is initialized to compute a three-dimensional geometrical representation, a three-dimensional geometrical representation of the object may be expressed as signed distance field of the object. For example, the geometrical representation of the object may be parameterised as the zeroth level set of an implicit function (e.g., points in space where the implicit function is zero - either exactly zero and / or approximated to zero using marching cubes technique, ray sampling, and / or averaging procedure described in Equations 7, 8, and 9 described herein). The geometrical meaning of a signed distance field is that it assigns a scalar value d to each point in space [xw,yw,zw]T corresponding to a distance to the closest surface (with a negative sign indicating a point in the interior of the surface): d = F(xw,yw,zw) (3) The unknown function F is the first neural network and the objective is to optimise its weights. The subscript w denotes world coordinates. More specifically, the first neural network may map a point (e.g., any point) on a three-dimensional space (e.g., x, y, z space) to a signed distance field of a point on the object. Put differently, the signed distance field of a plurality of points on the object may be represented as a neural function. The neural function is represented by the first neural network. For the image capturing set up shown in Figure 2 and Figure 3 ( / .e., binocular stereo cameras), this coordinate system may be chosen as the average between the two cameras ( / .e., the rectified stereo system). A roto-translation Rctc may be used to convert the world coordinates (e.g., denoted by the subscript w) to the original camera coordinates (e.g., denoted by the subscript c ). For instance as: K-,yw>zw\T = Rc • yc>zc\T + tc (4) The signed distance field of any arbitrary geometry may always be continuous and may be differentiable and may satisfy the Eikonal equation of unit magnitude gradient: ||VF(xw, yw, zw)|| = 1, V(xw, yw, zw) 6 (5) For surface points s where F(xs,ys,zs) = 0, the surface normal may be the gradient of the signed distance field: f dF dF n. = VF= —(6) s Wil's dys dzs] ' ' The above equation provides a direct relationship between the signed distance field and the surface normal. Therefore, in these examples, the first neural network may be trained based on rendering loss (as described below) from the one or more images captured at step 401 (e.g., using backpropagation). After the weights of the first neural network are optimised, the computed signed distance fields may be sampled in a regular grid of points and a triangular mesh surface may be recovered using standard marching cubes technique. In this manner the surface representation of the object may be computed from the three-dimensional geometrical representation of the object. Figure 5 illustrates example implementation of step 401 and step 403 of method 400 described in Figure 4. As seen in Figure 5, image 58a and image 58b depict images captured from a pair of cameras using the image capturing setup described herein (e.g., image capturing setup shown in Figure 2 and Figure 3). Put differently, image 58a may be a first image captured by a first camera (e.g., camera 21 in Figure 2) from a first viewing direction. Image 58b may be a second image captured by a second camera (e.g., camera 22 in Figure 2) from a second viewing direction. These images 58a and 58b may be used to generate an estimated shape of the object. For example, 62a and 62b depict per-view normal maps (e.g., normal maps for each of the first viewing direction and the second viewing direction). 64a and 64b depict per-view estimated shape of the object (.e.g., estimated shape for each of the first viewing direction and the second viewing direction). These per-view estimated shapes 64a and 64b may be received by a processor (e.g., processor included in computing device 12 in Figure 1). The per-view estimated shapes 64a and 64b may be provided to a CNN (e.g., first neural network). The first neural network may be randomly initialized as shown at 68. At 70, the first neural network may be initialized based on the per-view estimated shapes 64a and 64b and per-view estimated normal 62a and 62b. As discussed above, in some examples, the first neural network is configured to compute a surface representation of the object. For example, the first neural network may compute a heightmap of the object. The first neural network may be trained to optimize for its weights. In some variations, the first neural network may be trained (e.g., at 72) based on a comparison of the neural heightmap generated at 70 and the first image 58a and second image 58b captured by the image capturing set up. Referring back to Figure 4, at step 405, the method includes sampling one or more points along the corresponding viewing direction using the first neural network to identify a plurality of points on a surface of the object. For instance, sampling may include determining one or more points at an intersection of a ray of incident light from the cameras (e.g., camera 21 and camera 22 that capture the images at step 401) that passes through the centre of each pixel and the surface representation computed at step 403. These determined points may be sampled to identify the plurality of points on the surface of the object. For examples in which the first neural network is initialized to compute a surface representation at step 403, exact conversion between the surface coordinate system and the camera coordinate system might be impossible because the projection depth zc in Equation 1 is unknown. To overcome this, the method 400 may consider tentative depth samples zcl, zc2, zc3,... to generate points i along the viewing direction v as {vzci}. The method may include applying the coordinate transfer Equation 1 to convert the generated points into the world coordinate system as {[xri,yri,zri]}. The learnable function F may be queried in the position to get the surface depth estimates {zsi = F[xri,yri]}. In order to get the ‘actual’ depth estimate ze, the set of surface depth estimates may be reduced as a weighted average, with averaging weight computed through the disparity zsi and zri. In addition, to stop propagating the ray after the first surface intersection, the cumprod operation may be used. More specifically, this is computed as: ze =Hi(zSiSM(-f(zsi - zri)2)) (7) where SM denotes the softmax operator (e.g., exponential divided by the sum of exponentials). The -f scaling factor may be used to get a minimum (softmax of negative values is softmiri) of the correct scale (e.g., convert between millimetres and normalised units inside the sampling interval). The surface normal ne may be estimated with a similar softmax averaging. The surface normal ne may then be projected back to the camera coordinates (e.g., to compare with the original normal and apply losses (described below)) with the transpose rotation RT -ne. In the case of multiple intersection of the ray with the surface ( / .e., occlusions) the softmax distribution may be wide. The method 400 may detect these points and may discard these points (e.g., from a loss application point of view described below). For examples in which the first neural network is initialized to compute a three-dimensional geometrical representation at step 403, the signed distance field may be optimized by identifying the intersection of each ray from each camera (e.g., camera 21 and camera 22) pixel with the surface representation computed at step 403. To achieve this, the method 400 may consider tentative depth samples zcl, zc2, zc3,... to generate points i along the viewing direction v as {yzci}. The method may include applying the coordinate transfer Equation 4 to convert the generated points into the world coordinate system as = {[xri,yri,zri]}. The learnable function F may be queried in these position to obtain signed distance field values drt. In some variations, signed distance field values drt may be converted to differentiable volume transparencies <x using Equation 8 (with p being a trainable parameter and f being a scaling factor to convert between millimetres and normalised units inside the sampling interval). Thus, the intersection p may be computed as a weighted sum of ray samples p = Si(wiri)> with the sample weights WiCorresponding to the accumulated opacity / .e. / = (9) The surface normal and albedo may be computed in a similar manner to the position p and may be projected back to the camera coordinates with the transpose rotation RT ■ ne. At step 407, the method 500 includes rendering each point that was identified at step 405 to generate a 3D reconstruction of the object. Rendering each point that was identified at step 405 may produce one or more losses as further described herein. The points identified at step 405 may be rendered to any suitable representation of the object (e.g., surface normal, surface depth, a three dimensional representation of the object, a combination thereof, and / or the like). In some examples, the rendering may be implemented by a second neural network. The second neural network may be trained on the one or more losses that are produced due to rendering the identified points. Put differently, the second neural network may be trained to render each pixel based on the plurality of points that were identified at step 405. Since the second neural network is trained to render pixels, the second neural network may implicitly learn the material of the object In some examples, the second neural network may also learn other properties of the object based on the loss that the second neural network is trained on. For instance, the second neural network may learn properties such as surface heat of the object, gray scale representation of the object, RGB representation of the object, etc. Accordingly, the second neural network may be configured to learn a material of the object. In some variations, the second neural network may be trained based on robust intensity loss (further detailed below), thereby learning the material of the object. The second neural network may be different from the first neural network described above. To render light intensities, first, near-field lighting vectors lm and light attenuation am may be computed for a point light source m. A near lighting model (such as the model described in R. Mecca, A. Wetzler, A. Bruckstein, and R. Kimmel, Near Field Photometric Stereo with Point Light Sources, SIAM Journal on Imaging Sciences, 2014) may be implemented to do so. The lighting vector for each surface point (e.g., points identified at step 405) p may be computed as plm = sm—p, where sm represents a position of the point light source m. The attenuation factor may be computed as (£m(X) ■ = (lo) where tm = is the lighting direction, Am is the principle orientation of point light source, and pm is an angular dissipation factor. Then, the total intensity im may be computed as im = sm • am • BRDF(n, lm, v, mat). sm is a ‘soft’ indicator variable that is 0 for shaded points and 1 otherwise. To approximate the BRDF (i.e., Bidirectional Reflectance Distribution Function), a multilayer perceptron with 3 X 128 hidden layers with gelu activation and relu activation at the final layer (as m >0) may be used. In some variations, the material vector mat may be 10 dimensional (e.g., based on the number of parameter of the BRDF). For example, this may include single albedo, excluding anisotropy. In some variations, the multilayer perceptron renderer may be trained on the synthetic data from PX-Net approach described above. The trained renderer should be approximating a Disney shader but with added robustness to noise (e.g., due to data augmentation with the synthetic data from PX-Net approach). Even though the surface point may be computed as a weight sum over the incident ray (as described above), the render may be only computed on a single sample. To estimate shadows, each surface point (e.g., points identified at step 405) p may be traced back to the point light source m by following the direction of the lighting vector tm computed above. For example, for each ray 8 samples h may be taken every 1.5mm starting 3mm away from the start. The depth of the surface representation (e.g., heightmap) may be queried and the difference may be computed for example, dh = zh — zs. If at least one of the difference is negative, then that may indicate the presence of a shadow. It’s differentiable approximation may be computed as SM^-sigmoid^d^y Figure 6 illustrates example implementation of step 405 and step 407 of method 400 in Figure 4. The intersection of the ray of incident light from the cameras (e.g., camera 21 and camera 22) that passes through the centre of each pixel and the estimated surface representation may be represented by zr4, zr 2„ zr 3,and the respective surface values are zs^,, zs2,, zs3,which are used to compute the intersection point ze, (Equation 9). Then the surface point (point ze, is shown to avoid clatter) is traced back to the light source though points zM, zh s, zh b and the surface is queried again to obtain zsA, zSi5, zSi6 and compute shadows with above equation.In this manner, the surface normal, depth, and shadow may be estimated from the computed surface representation of the object. The method 400 may also include determining one or more losses (e.g., angular normal loss, rendering loss, robust rendering loss, normal loss, depth loss, depth mean loss, combined loss, Eikonal loss, Free space regulariser, a combination thereof, and / or the like) based on the 3D reconstruction of the object. These losses may be used to train the first neural network and / or the second neural network described herein. More specifically, these losses may be used to optimize the performance of the first neural network and / or the second neural network described herein. In some examples, these losses may be used to perform backpropagation to adjust the weights of the first neural network and / or the second neural network described herein. Angular normal loss - Since the methods and systems described herein integrate surface normals into a surface, computing angular loss may be important. Therefore, for network normal nn and surface normals ns, the angular loss may be computed as: Ln = |atan2(||nn x ns||, nn • ns)|max(nn • v, 0) (11) The angles are measured in degrees. In addition, the loss at each pixel may be weighted by max (nn- v, 0). This may minimize the impact of predicted normal on oblique surface points that are usually less accurate. Rendering loss - The error on the rendered intensities (for all point light sources m) may be computed as: Lr = I R t,m — A,mll(12) Robust rendering loss - The peak (over all point light sources) for both rendered and actual intensities may have to be aligned. This property should be albedo invariant and roughly material invariant (as for all specular materials the peak of the specular lobe align with the half vector). Therefore, for rendered intensities irm and true intensities itm the robust rendering loss is: Lrr relu (If,argmax ^r,m^ (13) Normal regulariser - As normal loss is heavily penalised on oblique points, it may be necessary to include a regulariser: Ln + Yl'W’z') (14) Here nw z is the z component of the surface normal (on world coordinates) and is encouraged to be close to -1 (with the addition of +1 ensuring the loss is always positive). Depth Loss - This may be used only for initialisation of the first neural network (e.g., SIREN architecture) described herein. For surface zs and for target surface zt (target surface is considered since there is no ground truth available yet): Lz = *z 1¾ - Ztl (15) Depth Mean Loss - Numerical integration is inherently ill posed as even under mathematically perfect conditions the integration constant is always ambiguous. The stereo setup theory eliminates this problem (as in principle there is a unique surface consistent with both views). However, the numerical instability remains. To overcome this problem, a mean depth regulariser may be employed. Therefore, a loss to the mean depth may be added to the world coordinates as: Lzr = Az \mean(zw) — z0\ (16) The mean may be computed on every mini-batch of data points and z0 is the mean of the world coordinate system and Az = 0.1 is the regularisation weight. Combined Loss - Combining the Losses above, the total loss is shown as: L = aLn + p Lrr + yLr + 6Lnr + r / Lzr + 4>LZ (17) In some variations, the following may be empirically set as: a = 1.0,(3 = 1000, / = 100,8 = 0.1,77 = 0.1. Also, during the initialisation of the first neural network (e.g., neural heightmap), 0 may be set as 0 = 1. However, the final and main phase of training may not include the depth loss. For examples in which the first neural network is initialized to compute a three-dimensional geometrical representation at step 403, the following losses may also be computed to train the first neural network and / or the second neural network described herein. Eikonal Loss - To enforce Equation 5, for all rays Eikonal loss Le may be applied. Le = |||Vd||-l| (18) Free Space Regulariser- To encourage the network to recover the minimal surface, positive Signed Distance field values are encouraged to increase and discouraged to decease according to: Lr = exp(—fd2) (19) This loss may also be applied to all ray samples. Training For examples in which the first neural network is initialized to compute a surface representation at step 403, the first neural network may be trained in the following way. The surface sampling procedure described at step 405 in Figure 4 may require that the surface representation to be appropriately initialised as the softmax distributions need not be very wide. Therefore, the initial estimates of the surface representations (e.g., depth maps) may be unified into a single coordinate system (e.g., world coordinate system). The learnable function (e.g., SIREN function) may be pre-trained with normal loss and depth loss. The initial depth maps may be inconsistent and the neural network may at best converge to an estimate of their average. Data augmentation of ±1mm on world coordinate points may be applied for the initial surface to be smooth (e.g., instead of the first neural network generating 2 copies of the surface representation). The pre-initialisation stage may be trained for 10 epoch and the main stage may be trained for 250 epochs with the total training time being around a day per object. For examples in which the first neural network is initialized to compute a three-dimensional geometrical representation at step 403, the first neural network may be trained in the following way. Initialisation stage - in order to speed up convergence, the first neural network may be initialised with an initial surface estimate. To achieve this, per view initial surface estimates may be computed. This may be projected into the world coordinate system and may be fused with Poisson reconstruction to obtain an initial surface representation. Then, a distance transform of the surface may be computed in a regular grid of 128 x 128 x 128 points (Pg around the object (i.e., for each of these grid points, the minimum distance to surface points may be computed with brute force search to have a signed distance field estimate d(p5)). The network may be trained to match the signed distance field on these points directly, i.e:. F^Pg) = d(pg) (using simple absolute value loss). Datasets The methods and systems described herein were evaluated using two datasets - DiLiGenT dataset and LUCES dataset. DiLiGenT dataset- The DiLiGeNT dataset contains 5 objects Bear, Buddha, Cow, Pot2, and Reading captured from 20 views, with 96 light images of 612x512 resolution. The objects are around 1.5m away from all cameras (turntable capture setup) and the camera focal length of 50mm approximates orthographic viewing. The views that were chosen were views 3, 4. LUCES-stereo dataset - This dataset was generated using the image capturing set up described herein (e.g., in Figure 2 and Figure 3). In this example, the LUCES-stereo dataset includes a subset of 6 objects (Bell, Bunny, Hippo, Owl, Queen, Squirrel) out of the original 14 objects. A image capturing apparatus (e.g., apparatus 11) includes 2x - Flea3 FL3-U3-32S2C-CS 1 / 2.8” Color USB 3.0 Camera pointgray cameras (2080x1552px) and 15 LED lights as shown in Figure 3. The lenses for the cameras were 8mm. The objects were places 15-20cm away from the camera in order for near lighting effects to be significant (as opposed to DiLiGeNT). To measure performance, since a stereo pair of views are used to compute the 3D reconstructions, full object Haustdorf distances may not be informative. Therefore, in order to have a fair comparison, a cropped ground trough may be computed (e.g., though back-projection of the ground truth depth maps). The Haustdorf distances may be computed from the ground truth to the reconstructed objects. That may ensure that the metric is fair regardless of the reconstructed surface sampling (e.g., since neural surfaces may be arbitrarily sampled). Therefore, the error maps may be computed and shown and different approaches may be compared. In one example, the methods and systems disclosed herein were implemented by disabling various configurations of rendering and robust rendering losses. The surface normal estimated using the PX-Net approach was used for generating the estimated shape and for performing surface fitting. The quantitative results are shown in Table 1. Method Bear Buddha Cow Pot2 Reading Mean ZE Median ZE Method herein [normals only] 0.57 1.04 0.42 1.25 4.68 1.88 0.53 Method herein [normals + rendering] 0.68 0.94 0.54 1.28 4.49 1.58 0.57 Method herein - [normals + robust] 0.86 0.98 0.39 1.24 4.76 1.65 0.53 DiLiGeNT [all views] 0.74 0.53 0.83 0.57 1.39 0.81 0.23 PS-Nerf [all views] 0.45 0.40 0.58 0.40 0.61 0.49 0.31 PS-Nerf [2 views] 3.31 0.57 1.08 1.39 3.15 1.90 1.87 Method herein - [normals + rendering + robust] 0.81 0.75 0.59 1.25 4.75 1.64 0.52 TABLE 1 As seen in Table 1, using both rendering loss and robust rendering loss significantly improves over the use of normal which is consistent with PS-Nerf approach. Using rendering shows significantly better performance on Bear and Reading objects while robust loss is better on Cow object. Better performance on the Cow object may be explained by the fact that it has non-uniform albedo which may prevent rendering loss based network from achieving better accuracy. Figure 7 illustrates an evaluation of the methods and systems disclosed herein on DiLiGenT dataset. The results from the methods disclosed herein are compared to a single view photometric stereo approach (Logothetis et al.) and PS-Nerf volumetric multi-view photometric stereo approach. For each of these methods a predicted shape and predicted error maps are visualised in Figure 7. As seen in Figure 7, the methods disclosed herein outperforms the single view method. Since the visualised error is computed by finding the average of closest distances from ground truth mesh to predicted mesh PS-Nerf, the error maps of the methods disclosed herein look relatively good despite wrongly hallucinated shapes for Pot and Buddha objects. This highlights the strength of the learnable surface representation (e.g., neural heightmap) over the volume approach (e.g., PS- Nerf) as such hallucinations of copies of parts of the object are not possible for the volume approach (e.g., PS-Nerf). Additionally, as seen in Figure 7, the PS-Nerf approach struggles to prevent itself from inventing copies of parts of the object due to very sparse supervision (e.g., see Pot2 and Reading objects). Figure 8 depicts plots of the mean and median shape error for objects from the DiLiGenT dataset. These plots show the mean and median shape error for all objects as well as average of corresponding metrics over all objects. As seen in Figure 8, most of the median shape error metrics are below 1mm. Figure 9 illustrates an evaluation of the methods and systems disclosed herein on LUCES-stereo dataset. The first three rows in Figure 9 show the cropped images and corresponding normal estimated from PX-Net approach and the ground truth shape. The final two rows show the shape predicted by the methods and systems disclosed herein and corresponding error maps. Similarly, table 2 shows quantitative evaluation of the methods and systems disclosed herein on LUCES-stereo dataset. Object Bell Bunny Hippo Owl Queen Squirrel Mean ZE Median ZE Method herein 2.72 1.37 0.86 2.62 0.60 0.90 1.51 1.45 Method herein [with alignment] 1.3 0.57 0.31 0.18 0.46 0.67 0.59 0.36 Table 2 Table 2 also includes a result after scaled rigid iterative closest point alignment (row 2). This is shown since the image planes of the cameras in the LUCES-Stereo are in a parallel configuration. This makes recovering the true depth values an ambiguous task. After scaling, as seen in Table 2, the error for all objects except Bell is below 1mm. Accordingly, the technology disclosed herein can generate 3D reconstruction of objects in an accurate manner from extremely sparse views. The technology disclosed herein is significantly better than single view photometric stereo approach. Additionally, the technology disclosed herein does not suffer from duplication of the parts of an object which is bound to happen for volumetric approaches such as PS-Nerf. Whilst certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel devices, and methods described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the devices, methods 5 and products described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions.

Claims

1. A computer vision method for generating a three dimensional reconstruction of an object, the method comprising:receiving at least one estimated shape of an object, the at least one estimated shape being generated based on a capture of one or more images from a corresponding viewing direction using a plurality of point light sources;providing the at least one estimated shape of the object as input to a first neural network, the first neural network being initialized to compute a representation of the object;sampling one or more points along the corresponding viewing direction using the first neural network to identify a plurality of points on a surface of the object; andrendering each point of the identified plurality of points to generate a three dimensional reconstruction of the object, wherein the rendering produces at least one loss and the first neural network is trained based at least in part on the at least one loss.

2. The computer vision method of claim 1, wherein the representation is a surface representation of the object.

3. The computer vision method of claim 1, wherein the representation is a three-dimensional geometrical representation of the object.

4. The computer vision method of any of claims 1 to 3, further comprising performing backpropagation of the at least one loss to adjust one or more weights of the first neural network.

5. The computer vision method of any of claims 1 to 4, wherein the rendering is implemented by a second neural network.

6. The computer vision method of claim 5, further comprising performing backpropagation of the at least one loss to adjust one or more weights of the second neural network.

7. The computer vision method of any of claims 5 to 6, wherein the second neural network is further configured to learn a material of the object.

8. The computer vision method of claim 1, wherein rendering further comprises rendering at least one of a surface normal and an intensity at each point of the identified plurality of points.

9. The computer vision method of any of claims 1 to 3, wherein the at least one estimated shape is generated using a third neural network.

10. The computer vision method of any of claims 1 to 9, wherein receiving the at least one estimated shape comprises receiving a first estimated shape and a second estimated shape, the first estimated shape being generated based on a first set of images captured from a first viewing direction and the second estimated shape being generated based on a second set of images captured from a second viewing direction.

11. The computer vision method of claim 10, further comprising:concatenating the first estimated shape and the second estimated shape to generate a concatenated estimated shape; andproviding the concatenated estimated shape as input to the first neural network.

12. The computer vision method of claim 2, further comprising computing at least one surface normal of the object based on a partial differentiation of one or more weights of the first neural network and the at least one estimated shape of the object.

13. The computer vision method of claim 12, wherein rendering includes rendering the computed at least one surface normal at each point of the identified plurality of points.

14. The computer vision method of claim 1, wherein sampling the one or more points along the corresponding viewing direction includes determining points at an intersection of a ray of incident light from an image capturing device that captures the one or more images and the surface of the object, and sampling the determined points to identify the plurality of points on the surface of the object, wherein the ray of incident light passes through a centre of each pixel.

15. The computer vision method of any of claims 1 to 3, further comprising determining a rendering loss based on a difference between the three dimensional reconstruction of the object and the actual object.

16. The computer vision method of claim 2, wherein the surface representation of the object is a heightmap.

17. The computer vision method of claim 16, wherein sampling the one or more points includes fitting the heightmap to the at least one estimated shape of the object.

18. The computer vision method of claim 2, wherein the surface representation of the object is a neural function, wherein the neural function is represented by the first neural network.

19. The computer vision method of claim 2, wherein the at least one estimated shape of the object comprises at least one estimated surface of the object, and wherein first neural network maps a point on a plane to a height from the surface of the object.

20. The computer vision method of claim 2, wherein the first neural network maps a point on a surface to a surface depth of the object.

21. The computer vision method of claim 3, wherein the at least one estimated shape of the object comprises at least one estimated three-dimensional geometry of the object, and wherein the first neural network maps a point on a three-dimensional space to a Signed Distance Field of a point on the object.

22. The computer vision method of claim 3, further comprising computing a surface representation of the object from the computed three-dimensional geometrical representation of the object.

23. The computer vision method of claim 3, wherein the three-dimensional geometrical representation of the object is a neural function, wherein the neural function is represented by the first neural network.

24. A carrier medium carrying computer readable instructions adapted to cause a computer to perform the method of any of claims 1 to 23.

25. A system for three dimensional reconstruction of an object, the system comprising an interface and a processor:the interface having an image input and configured to receive one or more images of an object from a corresponding viewing direction captured using a plurality of point light sources;the processor configured to:5 receive at least one estimated shape of the object, the at least one estimated shapebeing generated based on the one or more images of the object;provide the at least one estimated shape of the object as input to a first neural network, the first neural network being initialized to compute a representation of the object;sample one or more points along the corresponding viewing direction using the first 10 neural network to identify a plurality of points on a surface of the object; andrender each point of the identified plurality of points to generate a three dimensional reconstruction of the object, wherein the rendering produces at least one loss and the first neural network is trained based at least in part on the at least one loss.

Images