Method and system for generating a 3D representation of an object

Abstract

A method and system for generating a 3D representation of a target object enclosed in a transparent container is disclosed. The method and system comprise receiving, by at least one processor, captured images of a scene comprising the target object enclosed in the transparent container having a container geometry, each of the captured images of the target object having an associated viewing position and direction and then training, by the at least one processor, a neural radiance field model configured to represent the geometry and appearance of the scene based on evaluating a loss function comparing captured images with corresponding synthesised images. Finally, a 3D representation of the target object from the trained neural radiance field model is generated.

Description

METHOD AND SYSTEM FOR GENERATING A 3D REPRESENTATION OF AN OBJECT PRIORITY DOCUMENTS

[0001] The present application claims priority from Australian Patent Application No.2023222899 titled “METHOD AND SYSTEM FOR GENERATING A 3D REPRESENTATION OF AN OBJECT” and filed on 30 August 2023, the content of which is hereby incorporated by reference in its entirety. INCORPORATION BY REFERENCE

[0002] The following publications are referred to in the present application and their contents are hereby incorporated by reference in their entirety: • “Seeing Through the Glass: Neural 3D Reconstruction of Object Inside a Transparent Container”, Jinguang Tong, Sundaram Muthu, Fahira Afzal Maken, Chuong Nguyen and Hongdong Li, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp.12555-12564, (https: / / openaccess.thecvf.com / content / CVPR2023 / papers / Tong_Seeing_Through_the_Glass_Ne ural_3D_Reconstruction_of_Object_Inside_CVPR_2023_paper.pdf), see also arXiv:2303.13805 [cs.CV]. • “Seeing Through the Glass: Neural 3D Reconstruction of Object Inside a Transparent Container Supplementary Material”, Jinguang Tong, Sundaram Muthu, Fahira Afzal Maken, Chuong Nguyen and Hongdong Li, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, (https: / / openaccess.thecvf.com / content / CVPR2023 / supplemental / Tong_Seeing_Through_the_CV PR_2023_supplemental.pdf) TECHNICAL FIELD

[0003] The present disclosure relates to image processing. In a particular form, the present disclosure relates to generating a 3D representation of an object from a number of images adopting neural network techniques. BACKGROUND

[0004] Forming a 3D representation of an object from a number of two-dimensional images of the object is a common task in machine vision applications. In some examples, the object may be located in a first region having an associated first refractive index and the images are captured by a camera located in a second region having a second refractive interface. In these circumstances, the images of the object willbe captured through an interface or boundary surface defined between the regions of different refractive index. As a result of the differing refractive indices of the two regions, the resulting image will be distorted because of the potential repeated reflections and refractions of light. This greatly complicates any attempt to recover a three-dimensional representation of the object from the images as the 3D geometry of an object cannot be reliably reconstructed using existing structure from motion or neural reconstruction methods.

[0005] In one non-limiting example of a requirement to form of a 3D representation of an object in the circumstances described above, consider the case of an object such as an insect, human tissue or other fragile specimen that forms part of a collection which has been preserved in a hard, but transparent material, such as resin or glass which contains the object. Digitisation of collections of this nature has become an emerging requirement, especially with the advancement of virtual reality techniques where a viewer may remotely view objects in the collection. It follows that digitising or generating 3D representations of the preserved objects would be desirable to allow for their remote viewing and inspection. SUMMARY

[0006] In a first aspect, the present disclosure provides a method comprising: receiving, by at least one processor, captured images of a scene comprising a target object enclosed in a transparent container having a container geometry, each of the captured images of the target object having an associated viewing position and direction; training, by the at least one processor, a neural radiance field model configured to represent the geometry and appearance of the scene based on evaluating a loss function comparing captured images with corresponding synthesised images; and generating, by the at least one processor, a 3D representation of the target object from the trained neural radiance field model.

[0007] In another form, the neural radiance field model comprises a first neural network to represent the geometry of the scene and a second neural network to represent the appearance of the scene.

[0008] In another form, the method further comprises generating, by the at least one processor, a corresponding synthesised image for a captured image by: tracing rays, by the at least one processor, based on the associated camera viewing position and direction of the captured image and generating, by the at least one processor, the corresponding synthesised image based on the traced rays.

[0009] In another form, tracing rays by the at least one processor based on the associated camera viewing position and direction of the captured image comprises: generating, by the at least one processor, for a selected ray originating from a pixel of the captured image and intersecting the container, an associated set of ray paths arising from optical interactions of the selected ray with the container, wherein a ray path corresponds to a collection of ray segments commencing with an initial ray segment from the camera to the container surface and terminating with a final ray segment exiting, or reflected from, the container surface.

[0010] In another form, generating by the at least one processor, the corresponding synthesised image based on the traced rays comprises: determining, by the at least one processor, an irradiance contribution for each ray segment of the associated set of ray paths of the selected ray; and determining, by the at least one processor, the colour of the selected ray by accumulating the irradiance contribution from each ray segment.

[0011] In another form, determining by the at least one processor the irradiance contribution for each ray segment comprises: on determining by the at least one processor that a ray segment is within the container then determining, by the at least one processor, the irradiance contribution based on a weighted integration of radiance along the ray segment based on the neural radiance field model of the scene; or on determining by the at least one processor that a ray segment is outside of the container then determining the irradiance contribution based on an ambient irradiance of the scene.

[0012] In another form, determining by the at least one processor the colour of the selected ray by accumulating the irradiance contribution from each ray segment comprises: determining, by the at least one processor, a ray path irradiance contribution for each ray path of the set of ray paths associated with the selected ray by summing together a weighted irradiance contribution for each ray segment of the ray path starting with the final ray segment and finishing with the initial ray segment, and summing together, by the at least one processor, the ray path irradiance contributions from each ray path to determine the colour of the selected ray.

[0013] In another form, summing together by the at least one processor the weighted irradiance contribution for each ray segment of the ray path starting with the final ray segment and finishing with the initial ray segment comprises: initialising, by the at least one processor, the ray path irradiance contribution with the irradiance contribution of the final ray segment corresponding to outside the container;successively adding, by the at least one processor, weighted irradiance contributions for each of the ray segments in the ray path before the initial ray segment, wherein each weighted irradiance contribution corresponds to the irradiance contribution of the ray segment weighted by an optical interaction factor for the ray segment, a colour correction factor for the ray segment, an occupancy factor for the ray segment, and a transmission factor for the parent ray segment; and adding, by the at least one processor, a weighted irradiance contribution for the initial ray segment corresponding to outside the container, wherein the weighted irradiance contribution corresponds to the irradiance contribution of the ray segment weighted by an optical interaction factor for the ray segment, and a transmission factor for the parent ray segment.

[0014] In another form, the weighted integration is based on a hierarchical sampling process.

[0015] In another form, the first neural network is a multi-layer perceptron that represents the geometry of the scene as a signed distance function.

[0016] In another form, the second neural network is a multi-layer perceptron that represents the appearance of the scene as an irradiance for input point, view direction and normal vector.

[0017] In another form, the method further comprises determining, by the at least one processor, the container geometry from one or more of the captured images.

[0018] In another form, the loss function comprises a photometric loss function.

[0019] In another form, the container comprises a transparent container wall that contains a transparent medium, wherein the transparent container wall and the contained transparent medium have different refractive indices.

[0020] In a second aspect, the present disclosure provides a system comprising: at least one processor; and a storage operatively coupled to the at least one processor and for storing instructions that when executed by the at least one processor cause the at least one processor to: receive captured images of a scene comprising a target object enclosed in a transparent container having a container geometry, each of the captured images of the target object having an associated viewing position and direction; train a neural radiance field model configured to represent the geometry and appearance of the scene based on evaluating a loss function comparing captured images with corresponding synthesised images; and generate a 3D representation of the target object from the trained neural radiance field model.

[0021] In another form, the neural radiance field model comprises a first neural network to represent the geometry of the scene and a second neural network to represent the appearance of the scene.

[0022] In another form, a corresponding synthesised image for a captured image is generated by: tracing rays based on the associated camera viewing position and direction of the captured image and generating the corresponding synthesised image based on the traced rays.

[0023] In another form, tracing rays based on the associated camera viewing position and direction of the captured image comprises: generating for a selected ray originating from a pixel of the captured image and intersecting the container, an associated set of ray paths arising from optical interactions of the selected ray with the container, wherein a ray path corresponds to a collection of ray segments commencing with an initial ray segment from the camera to the container surface and terminating with a final ray segment exiting, or reflected from, the container surface.

[0024] In another form, generating the corresponding synthesised image based on the traced rays comprises: determining an irradiance contribution for each ray segment of the associated set of ray paths of the selected ray; and determining the colour of the selected ray by accumulating the irradiance contribution from each ray segment.

[0025] In another form, determining the irradiance contribution for each ray segment comprises: on determining that a ray segment is within the container then determining the irradiance contribution based on a weighted integration of radiance along the ray segment based on the neural radiance field model of the scene; or on determining that a ray segment is outside of the container then determining the irradiance contribution based on an ambient irradiance of the scene.

[0026] In another form, determining the colour of the selected ray by accumulating the irradiance contribution from each ray segment comprises: determining a ray path irradiance contribution for each ray path of the set of ray paths associated with the selected ray by summing together a weighted irradiance contribution for each ray segment of the ray path starting with the final ray segment and finishing with the initial ray segment, and summing together the ray path irradiance contributions from each ray path to determine the colour of the selected ray.

[0027] In another form, summing together the weighted irradiance contribution for each ray segment of the ray path starting with the final ray segment and finishing with the initial ray segment comprises: initialising the ray path irradiance contribution with the irradiance contribution of the final ray segment corresponding to outside the container; successively adding weighted irradiance contributions for each of the ray segments in the ray path before the initial ray segment, wherein each weighted irradiance contribution corresponds to the irradiance contribution of the ray segment weighted by an optical interaction factor for the ray segment, a colour correction factor for the ray segment, an occupancy factor for the ray segment, and a transmission factor for the parent ray segment; and adding a weighted irradiance contribution for the initial ray segment corresponding to outside the container, wherein the weighted irradiance contribution corresponds to the irradiance contribution of the ray segment weighted by an optical interaction factor for the ray segment, and a transmission factor for the parent ray segment.

[0028] In another form, the weighted integration is based on a hierarchical sampling process.

[0029] In another form, the first neural network is a multi-layer perceptron that represents the geometry of the scene as a signed distance function.

[0030] In another form, the second neural network is a multi-layer perceptron that represents the appearance of the scene as an irradiance for input point, view direction and normal vector.

[0031] In another form, the system further comprises determining the container geometry from one or more of the captured images.

[0032] In another form, the loss function comprises a photometric loss function.

[0033] In another form, the container comprises a transparent container wall that contains a transparent medium, wherein the transparent container wall and the contained transparent medium have different refractive indices.

[0034] In another form, the system further comprises a camera to capture the captured images of the scene.

[0035] In a third aspect, the present disclosure provides a computer program product including one or more non-transitory machine readable mediums having instructions encoded thereon that when executed by at least one processor causes a process for generation of a 3D representation of a target object to be carried out, the process comprising:receiving, by the at least one processor, captured images of a scene comprising the target object enclosed in a transparent container having a container geometry, each of the captured images of the target object having an associated viewing position and direction; training, by the at least one processor, a neural radiance field model configured to represent the geometry and appearance of the scene based on evaluating a loss function comparing captured images with corresponding synthesised images; and generating, by the at least one processor, the 3D representation of the target object from the trained neural radiance field model.

[0036] In another form, the neural radiance field model comprises a first neural network to represent the geometry of the scene and a second neural network to represent the appearance of the scene.

[0037] In another form, the computer program product further comprises generating, by the at least one processor, a corresponding synthesised image for a captured image by: tracing rays, by the at least one processor, based on the associated camera viewing position and direction of the captured image and generating, by the at least one processor, the corresponding synthesised image based on the traced rays.

[0038] In another form, tracing rays by the at least one processor based on the associated camera viewing position and direction of the captured image comprises: generating, by the at least one processor, for a selected ray originating from a pixel of the captured image and intersecting the container, an associated set of ray paths arising from optical interactions of the selected ray with the container, wherein a ray path corresponds to a collection of ray segments commencing with an initial ray segment from the camera to the container surface and terminating with a final ray segment exiting, or reflected from, the container surface.

[0039] In another form, generating by the at least one processor, the corresponding synthesised image based on the traced rays comprises: determining, by the at least one processor, an irradiance contribution for each ray segment of the associated set of ray paths of the selected ray; and determining, by the at least one processor, the colour of the selected ray by accumulating the irradiance contribution from each ray segment.

[0040] In another form, determining by the at least one processor the irradiance contribution for each ray segment comprises:on determining by the at least one processor that a ray segment is within the container then determining, by the at least one processor, the irradiance contribution based on a weighted integration of radiance along the ray segment based on the neural radiance field model of the scene; or on determining by the at least one processor that a ray segment is outside of the container then determining the irradiance contribution based on an ambient irradiance of the scene.

[0041] In another form, determining by the at least one processor the colour of the selected ray by accumulating the irradiance contribution from each ray segment comprises: determining, by the at least one processor, a ray path irradiance contribution for each ray path of the set of ray paths associated with the selected ray by summing together a weighted irradiance contribution for each ray segment of the ray path starting with the final ray segment and finishing with the initial ray segment, and summing together, by the at least one processor, the ray path irradiance contributions from each ray path to determine the colour of the selected ray.

[0042] In another form, summing together by the at least one processor the weighted irradiance contribution for each ray segment of the ray path starting with the final ray segment and finishing with the initial ray segment comprises: initialising, by the at least one processor, the ray path irradiance contribution with the irradiance contribution of the final ray segment corresponding to outside the container; successively adding, by the at least one processor, weighted irradiance contributions for each of the ray segments in the ray path before the initial ray segment, wherein each weighted irradiance contribution corresponds to the irradiance contribution of the ray segment weighted by an optical interaction factor for the ray segment, a colour correction factor for the ray segment, an occupancy factor for the ray segment, and a transmission factor for the parent ray segment; and adding, by the at least one processor, a weighted irradiance contribution for the initial ray segment corresponding to outside the container, wherein the weighted irradiance contribution corresponds to the irradiance contribution of the ray segment weighted by an optical interaction factor for the ray segment, and a transmission factor for the parent ray segment.

[0043] In another form, the weighted integration is based on a hierarchical sampling process.

[0044] In another form, the first neural network is a multi-layer perceptron that represents the geometry of the scene as a signed distance function.

[0045] In another form, the second neural network is a multi-layer perceptron that represents the appearance of the scene as an irradiance for input point, view direction and normal vector.

[0046] In another form, the computer program product further comprises determining, by the at least one processor, the container geometry from one or more of the captured images.

[0047] In another form, loss function comprises a photometric loss function.

[0048] In another form, the container comprises a transparent container wall that contains a transparent medium, wherein the transparent container wall and the contained transparent medium have different refractive indices.

[0049] In a fourth aspect, the present disclosure provides a system comprising: at least one processor; and a storage operatively coupled to the at least one processor and for storing instructions that when executed by the at least one processor cause the at least one processor to carry out the method of the first aspect.

[0050] In another form, the method further comprises image capture means to capture the captured images of the scene.

[0051] In a fifth aspect, the present disclosure provides a computer program product including one or more non-transitory machine readable mediums having instructions encoded thereon that when executed by at least one processor causes the method of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0052] Embodiments of the present disclosure will be discussed with reference to the accompanying drawings wherein:

[0053] FIG.1 shows an example general overview of a scene comprising a target object enclosed in a transparent container as viewed by a number of cameras each having their own viewing position and direction in accordance with an embodiment of the present disclosure;

[0054] FIG.2 is a flow diagram of an example method for generating a 3D representation of a target object based on a trained neural radiance field model in accordance with an embodiment of the present disclosure;

[0055] FIG.3 is an example system for generating a 3D representation of a target object based on a trained neural radiance field model in accordance with an embodiment of the present disclosure;

[0056] FIG.4 is a flow diagram of an example method for generating a corresponding synthesised image for a captured image in accordance with an embodiment of the present disclosure;

[0057] FIG.5 is a diagram similar to FIG.1 showing the generation of ray segments arising from the optical interactions a selected ray will have with the container in accordance with an embodiment of the disclosure;

[0058] FIG.6 is a flow diagram of an example method for determining the container geometry from one or more captured images in accordance with an embodiment of the disclosure;

[0059] FIG.7 is flow diagram of an example method for generating the corresponding synthesised image in accordance with an embodiment of the disclosure;

[0060] FIG.8 is a flow diagram of an example method for determining the irradiance contribution for each ray segment in accordance with an embodiment of the disclosure;

[0061] FIG.9 is a flow diagram of an example method for determining the colour of the selected ray by accumulating the irradiance contribution from each ray segment according to an illustrative embodiment;

[0062] FIG.10 is a flow diagram of an example method for summing together the weighted irradiance contribution for each ray segment of the ray path starting with the final ray segment and finishing with the initial ray segment in accordance with an illustrative embodiment;

[0063] FIG.11 is a flow diagram of an example method for training a neural radiance field model in accordance with an illustrative embodiment;

[0064] FIG.12 is a flow diagram of an example method for generating a synthesised image in accordance with an illustrative embodiment;

[0065] FIG.13 is a flow diagram of an example method for determining the accumulated irradiance of ray segments according to an embodiment of the present disclosure;

[0066] FIG.14 is a flow diagram of an example method 1400 for determining the colour of a ray according to an embodiment of the present disclosure. In one example, method 1400 may correspond to step 1240 of method 1400 illustrated in FIG.12;

[0067] FIGS.15A and 15B show respectively an image of a target object enclosed in a transparent container and a 3D representation of the target object generated in accordance with the present disclosure; and

[0068] FIGS.16A and 16B show respectively an image of a target object enclosed in a transparent container and a 3D representation of the target object generated in accordance with the present disclosure.

[0069] In the following description, like reference characters designate like or corresponding parts throughout the figures. DESCRIPTION OF EMBODIMENTS

[0070] Techniques are disclosed for generating a 3D representation of a target object that is enclosed in a transparent container using neural network techniques. Images of the scene that comprise the container are used to train a neural radiance field model that represents both the geometry and appearance of the scene. The disclosed techniques are particularly beneficial, as they allow for the recovery of the shape and configuration of an object that would otherwise be enclosed in a container where direct access to the object is not possible.

[0071] Referring now to FIG.1, there is shown an example general overview of a scene 100 comprising a target object 130 enclosed in a transparent container 110 having a container geometry 115 as viewed by multiple cameras 120 each having their own respective viewing position and associated viewing direction. While in this example, the transparent container is shown as having a general rectangular prism shape, it would be appreciated that the container may have any suitable regular or non-regular geometry having an outer container surface defining the extent of the container. In this example, the transparent container 110 is formed of a solid transparent material such as resin or glass and the target object 130 is a preserved insect.

[0072] Throughout this specification, the term “transparent” when describing a container is taken to mean that light is able to pass through the container to allow an object enclosed in the container to be discernible from outside of the container. Examples of transparent containers include, but are not limited to, solid containers formed of glass, clear plastic (eg, acrylics, polycarbonates or other suitable thermoplastics), or resin (eg, amber or other plant resins, artificial resins). In some examples, the transparent container may be tinted or otherwise colorised.

[0073] Referring now to FIG.2, there is shown a flow diagram of a computer-implemented method 200 for generating a 3D representation of a target object according to an illustrative embodiment. In one example, method 200 may operate to process images of the scene 100 having the same general form as illustrated in FIG.1 to generate a 3D representation of a target object that is enclosed in a transparent container.

[0074] By way of overview, method 200 comprises (at step 210) receiving (by at least one processor) captured images 205 of a scene comprising a target object enclosed in a transparent container having a container geometry where each of the captured images of the target object has an associated viewing position and direction. A neural radiance field model configured to represent the geometry and appearance of the scene is then trained (by the at least one processor) based on evaluating a loss function comparing captured images with corresponding synthesised images (at step 220). Finally (at step 230), a 3D representation 295 of the target object is generated (by the at least one processor) from the trained neural radiance field model.

[0075] Referring now to FIG.3, there is shown an example system 300 for generating a 3D representation of a target object according to an illustrative embodiment. In various examples, system 300 may be configured to implement or carry out method 200 as illustrated in FIG.2 (and also FIGS.4 and 6- 10 referred to below).

[0076] System 300 includes a computing device 310 having at least one processor 315 configured to execute the following modules, each of which is described in further detail below: Image Receiving Module 320, Neural Network Training Module 330 and 3D Representation Generation Module 340. Computing device 310 may include an optional display 380 and input device 370 to allow an operator to interact with computing device 310 where required.

[0077] In this example, the computing device 310 is further configured to receive, as input, at least two captured images 305 of a scene comprising a target object enclosed in a transparent container having a container geometry and their associated viewing positions and directions to be processed by system 300. As would be appreciated, this information may be received by a network (not shown).

[0078] The computing device 310 is further configured to produce, as an output, a 3D representation of the target object 395 generated from the captured images 305 of the scene that may be used to generate one or more visual images of the target object in 3D or 3D printings of the target object. Each of the modules 320, 330, and 340 can be used in conjunction with each other to generate the 3D representation of the target 395 based on a trained neural radiance field model of the scene.

[0079] The computing device 310 may be any computer system, such as a workstation, desktop computer, server, laptop, handheld computer, tablet computer, mobile computing or communication device, VR device or VR component (eg, headset, camera, etc) or other form of computing or telecommunications device that is capable of communication and that has sufficient processor power and memory capacity to perform the operations described in this disclosure.

[0080] A distributed computational system may be provided including a plurality of such computing devices. The computing device 310 includes storage 360 that may include one or more storage devices or non-transitory computer-readable media having encoded on the media one or more computer-executable instructions or software for implementing techniques as variously described in this disclosure. The storage 360 may include a computer system memory or random access memory, such as a durable disk storage (which may include any suitable optical or magnetic durable storage device, eg, RAM, ROM, Flash, USB drive, or other semiconductor-based storage medium), a hard-drive, CD-ROM, or other computer readable media, for storing data and computer-readable instructions or software that implement various embodiments as taught in this disclosure.

[0081] Storage 360 may include other types of memory as well, or combinations. The storage device may be provided on the computing device 310 or provided separately or remotely from the computing device 310. The non-transitory computer-readable media may include, but are not limited to, one or more types of hardware memory, non-transitory tangible media (for example, one or more magnetic storage disks, one or more optical disks, one or more USB flash drives), and the like. The non transitory computer-readable media included in the computing device 310 may store computer-readable and computer-executable instructions or software for implementing various embodiments. The computer- readable media may be provided on the computing device 310 or provided separately or remotely from the computing device.

[0082] The computing device 310 also includes at least one processor 315 for executing computer- readable and computer-executable instructions or software stored in the storage 360 and other programs for controlling system hardware. Virtualisation may be employed in the computing device 310 so that infrastructure and resources in the computing device 310 may be shared dynamically. For example, a virtual machine may be provided to handle a process running on multiple processors so that the process appears to be using only one computing resource rather than multiple computing resources. Multiple virtual machines may also be used with one processor.

[0083] A user may interact with the computing device 310 through a display 380, such as a screen or monitor, including an augmented reality display device, which may display one or more user interfaces provided in accordance with some embodiments. The output device 380 may also display other aspects, elements or information or data associated with some embodiments. The computing device 310 may include input or input / output devices 370 for receiving input from a user, for example, a keyboard, a joystick, a game controller, a pointing device (eg, a mouse, a user's finger interfacing directly with a touch-sensitive display device, etc.), or any suitable user interface, including an AR headset. The computing device 310 may include other suitable conventional I / O peripherals.

[0084] The computing device 310 includes or is operatively coupled to various suitable devices for performing one or more of the aspects as variously described in this disclosure. The computing device may run any suitable operating system including, but not limited to, any embedded operating system, any real-time operating system, any open source operating system, any proprietary operating system, any operating systems for mobile computing devices, or any other operating system capable of running on the computing device 310 and performing the operations described in this disclosure. In an embodiment, the operating system may be run on one or more cloud machine instances.

[0085] In other embodiments, the functional components / modules may be implemented with hardware, such as gate level logic (eg, FPGA) or a purpose-built semiconductor (eg, ASIC). Still other embodiments may be implemented with a microcontroller having several input / output ports for receiving and outputting data, and several embedded routines for carrying out the functionality described in this disclosure. In a more general sense, any suitable combination of hardware, software, and firmware can be used, as will be apparent.

[0086] As will be appreciated in light of this disclosure, the various modules and components of the system 300, such as the modules 320, 330, 340, are implemented in software, such as a set of instructions (eg, HTML, XML, C, C++, object oriented C, BASIC, Python, etc.) encoded on any computer readable medium or computer program product (eg, hard drive, server, disc, or other suitable non-transitory memory or set of memories), that when executed by one or more processors, cause the various methodologies provided in this disclosure to be carried out.

[0087] It will be appreciated that, in some embodiments, various functions and data transformations performed by the user computing system, as described in this disclosure, can be performed by similar processors or databases in different configurations and arrangements, and that the depicted embodiments are not intended to be limiting. Various components of this example embodiment, including the computing device 310, may be integrated into, for example, one or more desktop or laptop computers, workstations, tablets, smart phones, game consoles, set-top boxes, or other such computing devices. Other componentry and modules typical of a computing system, such as processors (eg, central processing unit and co-processor, graphics processor, etc.), input devices (eg, keyboard, mouse, touch pad, touch screen, etc), and operating systems, are not shown but will be readily apparent.

[0088] In one example, system 300 further comprises image capture means in the form of one or more imaging devices or cameras that are configured to capture the multiple captured images of the scene comprising the target object enclosed in the transparent container.

[0089] Referring back to FIG.2, at step 210 method 200 comprises receiving captured images of a scene comprising a target object enclosed in a transparent container having a container geometry, each of thecaptured images of the target object having a respective viewing position and associated viewing direction. In one example. the functionality of step 210 may be carried out by Image Receiving Module 320 of system 300 illustrated in FIG.3.

[0090] As would be appreciated, any number of standard or proprietary digital image formats (eg, JPEG, bitmap, PNG, TIFF, QuickTime VR, and PANO) or customised image format capable of storing colour information for each pixel (eg, RGB format) may be used for captured images 205.

[0091] At step 220, method 200 comprises training a neural radiance field model configured to represent the geometry and appearance of the scene based on evaluating a loss function comparing captured images 205 with corresponding synthesised images. In one example, the functionality of step 220 may be carried out by Neural Network Training Module 330 of system 300 illustrated in FIG.3.

[0092] In one example, the neural radiance field model comprises a first neural network to represent the geometry of the scene. In one example, the first neural network comprises a multi-layer perceptron (MLP) defined as:

[0093] with θ as parameters where gθ represent the geometry of the scene as a signed distance function(SDF) which outputs for an input point coordinate X the signed distance value s. From ^^ఏ( ^^), theof an object ^^ i the scene may be extracted from a zero-level set of its signed distance as:

[0094] In one example, ^^ఏnitialised to be an ellipsoid-like SDF field prior to training.

[0095] In one example, the neural radiance field model comprises a second neural network to represent the appearance of the scene. In one example, the second neural network comprises a MLP defined as: ^^థ

[0096] with ^^ asparameters where ^^థsents the appearance of the scene by outputting theirradiance ^^ for an input point coordinate ^^, view direction ^^ and the normal vector ^^.

[0097] Referring now to FIG.4, there is shown a flow diagram of an example method 400 for generating a corresponding synthesised image for a captured image according to an embodiment of the present disclosure.

[0098] At step 410, method 400 comprises tracing rays from the associated camera viewing position and direction of the captured image and at step 420 the corresponding synthesised image is generated based on the traced rays.

[0099] In one example, tracing rays from the associated camera viewing position and direction of the captured image comprises generating for a selected ray originating from a pixel of the captured image an associated set of ray segments arising from any optical interactions of the selected ray with the container. The following discussion will be in the context of a selected ray corresponding to a pixel of the captured image and the generation of a corresponding pixel in the corresponding synthesised image. As will be discussed below, generation of the corresponding synthesised image will involve the process described with respect to a single pixel and selected ray being carried out multiple times with respect to multiple pixels of the captured image to generate the corresponding pixels of the synthesised image.

[0100] Referring now to FIG.5, there is shown a diagram similar to FIG.1 depicting the generation of ray segments (eg, ℓ^^, ℓ௧^arising from the optical interactions that a selected ray (eg, ℓ^) will have with the container 110 in accordance with an embodiment of the disclosure. Considering selected ray ℓ^, then as depicted, ℓ^^ is a reflected ray segment reflected from first container surface 116 and ℓ௧^ is a refracted ray segment that is transmitted through first container surface 116 that are generated in accordance with the Law of Reflection and Snell’s Law respectively. As can be seen from FIG.5, further ongoing ray segments will be generated corresponding to the reflected and refracted ray segments of original refracted ray ℓ௧^ arising from the optical interaction with container surface 117 and then further ray segments arising from further optical interactions with the first and second container surfaces 116, 117 defined by the container geometry (eg, a box as shown in FIG.5).

[0101] The generation of a reflected ray ℓ^^ and arefracted ray ℓ௧^ from the original ray ℓ^maybe represented as follows:

[0102] where ^^^is the surface normal at the point of intersection of the ray with the container surface defined by the container geometry (eg, container surfaces 116, 117) and ℛℰ represes the optical interactions of reflection and refraction in accordance with the Law of Reflection and Snell’s Law. Ray tracing process is performed recursively on ℓ^^ and ℓ௧^ and in one example the extent of recursion isconstrained by a defined depth of recursion ^^^^. In one example, a recursion depth ^^^^= 2 is adopted as a trade-off between accuracy and efficiency.

[0103] In this manner, the original camera ray ℓ^following intersection with the transparent container 110 will have a set of associated resulting traced rays formed as a result of the initial intersection of original camera ray ℓ^with the transparent container and potentially further interactions that subsequent reflected and refracted rays may have with the surface of the container. Considering FIG. 5 again, the resulting traced rays would comprise an associated set of ray paths where each ray path corresponds to a collection of ray segments that commence with an initial ray segment from the camera to the container surface (ie, initial ray segment located outside of the container) and terminates with a final ray segment exiting, or reflected from, the container surface (ie, final ray segment also located outside of the camera). As an example, FIG.5 depicts four ray paths Ray Paths 1-4 where in each case the ray path corresponds to a collection of ray segments that commence with a ray segment from the camera to the container surface and terminates with a ray segment exiting, or reflected from (eg, Ray Path 1), the container surface.

[0104] As would be appreciated, the optical interactions referred to in Equation 4 are dependent on the container surface normal ^^^at the point of intersection of the ray with the container surface as defined by the container geometry. In one example, the container geometry (and hence the dependence of the surface normal as a function of location on the container surface) is information included with the captured image. As will be seen below, in some examples, the container geometry may also be employed to generate a binary mask indicating which pixels of a captured image may be employed to generate the corresponding synthesised image.

[0105] In another example, the geometry of the container surface is determined from one or more of the captured images.

[0106] Referring now to FIG.6, there is shown a flow diagram of an example method 600 for determining the container geometry from one or more captured images 605 according to an embodiment of the disclosure. In one example, the method of determining the container geometry is based on a known base geometric primitive for the container. In one example, the base geometric primitive for the container is determined to be of a cuboid or rectangular prism configuration. In another example, the container may have a cylindrical or spherical configuration.

[0107] At step 610, an image is selected from the one or more captured images. In one example, the image with the lowest re-projection error during calibration is selected. In another example, the image showing the greatest number of discernible vertices or edge features of the container may be selected.

[0108] At step 620, the edges of the container in the selected image are labelled with lines. In one example, this may be carried out by an edge detection analysis. In another example, the edges may be labelled by an operator.

[0109] At step 630, the 2D coordinates of the line intersections are determined.

[0110] At step 640, the 3D coordinates of the line intersections are determined by ray-plane intersection based on the camera viewing position and direction (ie, camera pose) 645 associated with the selected image.

[0111] At step 650, the geometry (and pose) of the container is determined on the basis that the base geometric primitive 655 of the container is known. In one example, the base geometric primitive is a cuboid configuration 655 and this allows the base geometric primitive in the form of a general cuboid to be fitted to the 3D coordinates determined at step 640 to determine the container geometry.

[0112] As would be appreciated, any other techniques capable of determining the 3D shape or configuration of the container may be employed. While determining the geometry of the container from an image employing machine vision techniques may be convenient in some applications, in other applications the physical geometry of the container could be determined using a 3D scanner (optical based or otherwise). The container enclosing the target object could then be placed with a predetermined orientation and location with respect to a camera (also having a predetermined pose) as a result allowing the container geometry for any captured image to be precisely defined. This approach may have applications where the container has a non-regular or complicated shape which may be difficult to discern from captured images of the container. As mentioned above, the container geometry data for each captured image may then form part of or be associated with each captured image when it is being processed in accordance with the present disclosure.

[0113] Referring back to FIG.4, after the ray tracing process has been completed for the selected ray ℓ^, this ray is effectively extended to a set ^^^of ray segments ^^^such that ^^^∈ ^^^where the set ^^^contains the original camera ray ℓ^in addition to the set of associated traced ray segments. As would be appreciated, for some rays there will be no interaction with the container and as a result there is no supplementing of the original ray with additional ray segments.

[0114] At step 420 of FIG.4, the corresponding synthesised image is generated based on the traced rays.

[0115] Referring now to FIG.7, there is shown a flow diagram of an example method 700 for generating the corresponding synthesised image according to an embodiment of the disclosure. Referringagain to the context of a selected ray which has been extended to a set ^^^of ray segments ^^^, at step 710 an irradiance contribution is determined for each ray segment of the associated set of ray segments of the selected ray.

[0116] Referring now to FIG.8, there is shown a flow diagram of an example method 800 for determining the irradiance contribution for each ray segment according to an embodiment of the disclosure.

[0117] At step 810, an initial determination is made whether a ray segment is within the container or not. At step 820, where the ray segment is outside of the container, the irradiance contribution is based on an ambient irradiance of the scene. Formally, this can be written as ^^^௨௧= ^^^^^^^^௧.

[0118] At step 830, where the ray segment is determined to be within the container then determining the irradiance contribution is based on a weighted integration of radiance along the ray segment based on the neural radiance field model of the scene.

[0119] Consider the volume rendering equation. For a general light ray parameterised as follows ^^( ^^) = ^^ + ^^ ^^, the color ^^(ℓ) is calculated by the weighted integration of radiance along the ray by:

[0120] where ^^( ^^) is the weight at point ^^( ^^) and ^^( ^^( ^^), ^^) is the radiance emitted from point ^^( ^^) along the direction ^^.

[0121] An unbiased and occlusion-aware weight function may be defined as:

[0122] with the aid of an opaque density function:

[0123] wh here:^^

[0124] As can be seen from above, the weighted integration of radiance defined by Equation 5 will be based on the neural radiance field model of the scene through the determination of the SDF (ie, the representation of the geometry corresponding to ^^( ^^( ^^)) from Equation 1) and irradiance (ie, the representation of appearance corresponding to ^^( ^^( ^^), ^^) from ^^( ^^, ^^, ^^) of Equation 2).

[0125] Integration of Equation 5 is estimated by using numerical quadrature. In one example, hierarchical sampling process is adopted that separates the sampling process into coarse and fine parts. For the coarse part, a fixed number of points (eg, 64) is sampled uniformly along the ray segment and the density ^^( ^^) for each point is determined according to Equation 7 based on the values obtained from the geometric first neural network (eg, see Equation 1). This density is also used in fine sampling. In the fine sampling process, another fixed number of points (eg, 64) may be sampled around a point based on the initial determination of a point’s density. Accordingly, in the volume rendering more points will be sampled in those regions of higher density and less around lower density regions which will assist in characterising the target object. Based on this approach, when tracing a ray segment through the container, there is no attempt to explicitly model the interaction of a ray segment with the target object as the ray segment effectively passes straight through the container space. The Applicant has found that this allows much better control of the scene as only light interactions that result from the optical interaction of a ray segment with the surface of the container (ie, reflection and refraction) need to be considered.

[0126] As would be appreciated, the underlying neural radiance field model defined by Equations 1-3 (eg, see also “NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi- view Reconstruction”, Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, Wenping Wangar, Advances in Neural Information Processing Systems, 2021) is but one example of an underlying architecture that may be enhanced to determine a 3D representation of a target object in a transparent container in accordance with the present disclosure.

[0127] In another example, the underlying neural radiance field model is based on the NeRF architecture (eg, see “Nerf: Representing scenes as neural radiance fields for view synthesis”, Mildenhall, Ben, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng, Communications of the ACM, 65, no.1 (2021): 99-106). This architecture employs a single neural network (eg, MLP) to represent a scene where the neural network predicts density and color at a given point and calculates the irradiance of a ray segment by volume rendering. After training the neural network, the geometry of the target object can be extracted from the predicted density.

[0128] In a further example, the underlying neural radiance field model is based on the UNISURF architecture (eg, see “Unisurf: Unifying neural implicit surfaces and radiance fields for multi- view reconstruction”, Oechsle, M., Peng, S. and Geiger, A., 2021, Proceedings of the IEEE / CVF International Conference on Computer Vision, pp.5589-5599). This architecture employs two neuralnetworks in the form of a geometric neural network and an appearance neural network. An occupancy field is used to represent the geometry (occupancy 1 means inside the object while 0 means outside the object). In this example, rendering involves first trying to find the surface from the occupancy network and then sampling points near to the surface. Volume rendering is employed to obtain the color of the ray segment.

[0129] In yet another example, the underlying neural radiance field model is based on the VolSDF architecture (eg, see “"Volume rendering of neural implicit surfaces”, Yariv, Lior, Jiatao Gu, Yoni Kasten, and Yaron Lipman, Advances in Neural Information Processing Systems 34 (2021): 4805- 4815) which is similar to the NeuS architecture referred to above except that the density ^^ (eg, for comparison see Equation 7) is determined based on the SDF value ^^ by:

[0130] where ^^ and ^^ are learnable parameters. ^^ఉis the Cumulative Distribution Function (CDF) of the Laplace distribution with zero mean and ^^ scale.

[0131] Referring back to FIG.7, at step 720 the colour of the selected ray is determined by accumulating the irradiance contribution from each ray segment.

[0132] Referring now to FIG.9, there is shown a flow diagram of an example method 900 for determining the colour of the selected ray by accumulating the irradiance contribution from each ray segment according to an illustrative embodiment.

[0133] Recalling that the ray segments of a selected ray may be grouped in accordance with ray paths that each correspond to a collection of ray segments that commence with an initial ray segment from the camera to the container surface (ie, initial ray segment located outside of the container) and terminates with a final ray segment exiting, or reflected from, the container surface, then step 910 of method comprises first determining the ray path irradiance contributions for each ray path of the set of ray paths that are associated with the selected ray. This is carried out by in turn summing together a ray segment irradiance contribution for each ray segment of the ray path starting with the final ray segment and finishing with the initial ray segment. Once the individual ray path irradiance contributions have been determined for each ray path corresponding to the selected ray then at step 920 these can be summed together to determine the colour of the selected ray.

[0134] Referring now to FIG.10, there is shown a flow diagram of an example method 1000 for summing together the weighted irradiance contribution for each ray segment of the ray path starting with the final ray segment and finishing with the initial ray segment according to an illustrative embodiment.

[0135] At step 1010, method 1000 comprises initialising the ray path irradiance contribution with the irradiance contribution of the final ray segment corresponding to outside the container. As would be appreciated, in accordance with step 820 of method 800 (see FIG.8), this irradiance contribution will correspond to the background irradiance.

[0136] At step 1020, method 1000 comprises successively adding weighted irradiance contributions for each of the ray segments in the ray path following the final ray segment and before the initial ray segment.

[0137] Each of the weighted irradiance contributions corresponds to the irradiance contribution of the ray segment (eg, calculated in accordance with step 810 of method 800 in FIG.8) weighted by an optical interaction factor for the ray segment, a colour correction factor for the ray segment, an occupancy factor for the ray segment, and a transmission factor for the parent ray segment.

[0138] Consider where a ray segment results from an interaction of parent ray segment with the container surface. The ray segment may result from either a reflection of the parent ray segment characterised by reflectance ^^^which is calculated according to the Fresnel equations under a “natural light” assumption or from a refraction of the parent ray segment resulting in transmission through the container surface. The refraction may be characterised by the transmittance and is defined as follows:

[0139] In one example, an optical interaction factor may be determined for the ray segment in accordance with:

[0140] where Equation 10 characterises the optical correction factor for a ray segment arising as a result of a reflection of parent ray segment and Equation 11 characterises the optical correction factor for a ray segment arising as a result of a refraction (and transmission) of a parent ray segment.

[0141] As the second neural network of the neural radiance field model (see Equation 3) predicts colour in linear space, in one example a colour correction factor is adopted to compensate for the expected non-linear camera response function (CRF) that describes the relationship between image intensity and scene irradiance. In one example, a gamma correction function is applied as follows:

[0142] where ^^( ^^ gamma correction function.

[0143] For a ray segment, a transmission factor ^^ℓcorresponding to the accumulated transmittance along the ray segment, may be computed as:

[0144] where ^^^and ^^^indicate the start and end of the ray segment (ie, see Equation 8 where ^^^is substituted for variable ^^). It is important to note that the transmittance defined by Equation 13 which represents the transmittance of the space along the ray segment is different to ^^^^^referred to in Equation 11 which relates to the transmittance of refracted light.

[0145] Furthermore, an occupancy factor ^^ℓfor a ray segment may be defined as

[0146] As referred to above, in one example the irradiance contribution of the ray segment may be weighted by an occupancy factor ^^ℓfor the ray segment, and a transmission factor ^^ℓthat has been calculated for the parent ray segment.

[0147] Referring back to FIG.10, at step 1030, method 1000 comprises adding a weighted irradiance contribution for the initial ray segment corresponding to outside the container. In one example, the weighted irradiance contribution corresponds to the irradiance contribution of the ray segment as determined in accordance with step 820 of method 800 (see FIG.8) weighted by an optical interaction factor for the ray segment (see Equation 10 or 11), and a transmission factor for the parent ray segment (see Equation 13).

[0148] The above discussion has been in the context of generating the colour of a pixel of a synthesised image corresponding to the pixel of the capture image. As would be appreciated this approach may be repeated for a batch of pixels from the captured image to form the equivalent batch of synthesised pixels of the corresponding synthesised image.

[0149] In one example, a binary mask ^^^^of the container is applied to the captured image to select only those pixels that correspond to an image of the container.

[0150] In one example, a batch of pixels is randomly sampled from the masked set of pixels corresponding to the container and the corresponding synthesised pixels are generated based on the batch of pixels and their associated “selected” rays.

[0151] Referring back to FIG.2, at step 220 method 200 involves evaluating a loss function. As would be appreciated, the training of the neural radiance field model will involve the generation of multiple synthesised images and evaluating the loss function until the loss function converges and the parameters of the neural radiance field model are determined (see also FIGS.11-14 and associated discussion below).

[0152] In one example, the loss function comprises a photometric loss function ℒ^^^^^defined as follows:

[0153] comparing the rendered pixel colour ^^^^of the synthesised image with the pixel colour of the capture image ^^^. In this example, L1 loss function has been adopted.

[0154] In another example, the loss function further comprises a transparency based loss function ℒ௧^^^^that exploits a sparsity prior that a considerable part of the container ^^^^(except the area occupied by the object) is transparent to the light. This is characterised by regularising the transmittance of ray segments in the container ( ^^^^) as follows:

[0155] The Applicant has found that adoption of ℒ௧^^^^is of assistance where the target object takes up a limited area of the container. Experimental results show that ℒ௧^^^^helps to get cleaner geometry.

[0156] In another example, the loss function further comprises an Eikonal regularisation based loss function ℒ^^^applicable to all the sampled points (ie, ^^) from ray segments in the container to regularise the first neural network ^^ఏ(see Equation 1) of the neural radiance field model which is defined as follows:

[0157] Accordingly, in one example the loss function may be expressed as follows:

[0158] In one example, ^^^and ^^ଶare set to 0.1

[0159] Referring back to FIG.2, at step 230, method 200 comprises generating, a 3D representation of the target object from the trained neural radiance field model. In one sense, the trained first neural network is already a representation of the target object as it predicts the signed distance value corresponding to the surface of the target object (ie, see Equation 1) from which a further more convenient representation may be generated. In one example. the functionality of step 230 may be carried out by 3D Representation Generation Module 340 of system 300 illustrated in FIG.3.

[0160] In one example, a 3D representation in the form of a mesh is determined from the zero- level set of the neural radiance field model. In one embodiment, the mesh is determined by adopting the Marching Cubes algorithm. In yet another example, a 3D representation in the form of a set of voxels is determined from the zero-level set of the neural radiance field model. In yet another example, a 3D representation in the form of an occupancy field is determined from the zero-level set of the neural radiance field model. As would be appreciated, the various 3D representations may be interchangeable in the sense that a mesh representation may be used to generate a point cloud representation by sampling from the mesh surface or in another example an occupancy field or set of voxels could be generated by voxelising by sampling the mesh in volume space.

[0161] In another example, the 3D representation in the form of a point cloud characterising the surface of the target object is determined from the zero-level set of the trained first neural network neural radiance field model by randomly sampling points in a region of interest corresponding to the target object and processing them through the first neural network to obtain corresponding SDF values. A SDF threshold may be set and an assumption made that a point whose distance to the surface is less than the threshold is then on the surface of the target object. This process may be repeated to a set of points, ie a point cloud corresponding to the surface of the target object.

[0162] Numerous embodiments will be apparent in light of the present disclosure, and the features functionality described may be combined in any number of configurations. Referring now to FIG.11, there is shown a flow diagram of an example method 1100 for training a neural radiance field model according to an embodiment of the present disclosure. In one example, method 1100 may correspond to step 220 of method 200 for generating a 3D representation of a target object as shown in FIG.2. In another example, the functionality of step 220 may be carried out by Neural Network Training Module 330 of system 300 illustrated in FIG.3.

[0163] At step 1110, the first and second neural network models are initialised. In one example, the first neural network is a first MLP defined in accordance with Equation 1 to represent the geometry of the scene comprising a target object enclosed in a transparent container and the second neural network is a second MLP defined in accordance with Equation 3 to represent the appearance of the scene.

[0164] At step 1120, the next captured image for the training process is selected.

[0165] At step 1130, the captured image characteristics of the captured image are identified or determined. In one example, the captured image characteristics comprise the associated viewing position and direction (ie, camera pose) for the captured image. In another example, the captured image characteristics may comprise the background lighting corresponding to the captured image.

[0166] As step 1140, rays are traced based on the camera viewing position and direction that correspond to those of the camera that took the captured image.

[0167] At step 1150, a synthesised image is generated adopting the neural radiance field model which in this example comprises a first neural network to represent the geometry of the scene and a second neural network to represent the appearance of the scene.

[0168] At step 1160, a loss function is evaluated based on the difference between the captured image and the corresponding synthesised image (eg, see Equation 18).

[0169] At step 1170, a determination is made as to whether the loss function converges. If the loss function converges then 1170A (ie, condition YES) the training method 1100 is completed at step 1190. If on the other hand the loss function has not converged 1170B (ie, condition NO) then at step 1180 the parameters of first and second neural network models are updated by back propagating the loss function.

[0170] Referring now to FIG.12, there is shown a flow diagram of an example method 1200 for generating a synthesised image according to an embodiment of the present disclosure. In one example, method 1200 may correspond to step 1150 of method 1100 illustrated in FIG.11.

[0171] At step 1210, the next camera ray for generating the synthesised image is selected.

[0172] At step 1220, a hierarchical set of ray segments corresponding to the selected ray is determined by computing the reflections and refractions of the selected ray as it interacts with the surface and volume of the container (eg, see Equation 4). In one example, the hierarchical set of ray segments comprises a set of ray paths arising from optical interactions of the selected ray with the container andwherein each ray path corresponds to a collection of ray segments commencing with an initial ray segment from the camera to the container surface and terminating with a final ray segment exiting, or reflected from, the container surface (eg, see FIG.4).

[0173] At step 1230, the irradiance of each ray segment in the hierarchical set of ray segments is determined. An example implementation of this determination is described in more detail with reference to FIG.13.

[0174] At step 1240, the colour of the selected ray is determined by accumulating the irradiances of each of the ray segments in the hierarchical set of ray segments corresponding to the selected ray.

[0175] At step 1250, this determined colour is stored.

[0176] At step 1260, a determination is then made as to whether there are any further rays to process corresponding to the captured image. If there are no further camera rays 1260B (ie, condition NO) then method 1200 terminates. If there are further camera rays 1260A (ie, condition YES) then the next ray is selected at step 1210 and method 1200 is repeated.

[0177] Referring now to FIG.13, there is shown a flow diagram of an example method 1300 for determining the accumulated irradiance of ray segments according to an embodiment of the present disclosure. In one example, method 1300 may correspond to step 1230 of method 1200 illustrated in FIG. 12.

[0178] At step 1310, the next ray segment of the current ray is selected.

[0179] A determination is then made at step 1320 whether the selected ray segment is inside the container 1320A (ie, condition YES) or outside the container 1320B (ie, condition NO) (see also FIG.8 and method 800). Taking the example of where the ray segment is located outside of the container, then at step 1330 the irradiance of the ray segment is determined based on the background lighting and the irradiance of the ray segment is stored at step 1380. In the case of the ray segment being located in the container then at step 1340 points are selected along the ray segment.

[0180] At step 1350 the geometry of the object is predicted by the first trained neural network model and at step 1355 the appearance of each of the sample points is predicted by the second trained neural network model. At step 1360, based on the outputs of steps 1350 and 1355 an integration along the ray segment is carried out by volume rendering to determine the irradiance of the ray segment (eg, see Equation 5). At step 1370, the integrated weights (eg, see Equation 6) are stored as the occupancy of theselected ray segment and at step 1380 the irradiance as determined by the integration process at step 1360 is stored for the selected ray segment.

[0181] At step 1390, a determination is then made as to whether there are any further ray segments to process corresponding to the ray being processed. If there are no further camera ray segments 1390B (ie, condition NO) then method 1300 terminates at step 1395. If there are further ray segments 1390A (ie, condition YES), then the next ray is selected at step 1310 and method 1300 is repeated.

[0182] Referring now to FIG.14, there is shown a flow diagram of an example method 1400 for determining the colour of a ray according to an embodiment of the present disclosure. In one example, method 1400 may correspond to step 1240 of method 1200 illustrated in FIG.12.

[0183] At step 1410, for the hierarchical set of ray segments corresponding to the current ray, the next segment is selected in inverse order from the ray tracing. In the example where the hierarchical set of ray segments have been arranged in ray paths that each corresponds to a collection of ray segments commencing with an initial ray segment from the camera to the container surface and terminating with a final ray segment exiting, or reflected from, the container surface, the selection process will start with the final ray segment for each ray path and finish with the initial ray segment to the camera position.

[0184] At step 1420, the weighted irradiance is determined for the selected ray segment based on the reflectance / transmittance of the surface (eg. see Equation 13) and occupancy of the selected ray segment and any parent segment (eg, see Equation 14).

[0185] At step 1430, the computed weighted irradiance of the current ray segment is added to the accumulated irradiance of the parent ray segment.

[0186] At step 1440, a determination is then made as to whether there are any further ray segments to process corresponding to the ray. In one example, this will be equivalent to determining whether all of the ray segments of the different ray paths have been traversed. If there are no further ray segments 1440B (ie, condition NO) then the colour of the current ray is based on accumulated radiance of the ray segments at step 1450. In one example, the colour is determined based on a gamma mapping (eg, see Equation 12). If there are further ray segments 1440A (ie, condition YES), then the next ray segment is selected at step 1410 and method 1400 is repeated.

[0187] Referring now to FIG.15A and FIG.15B there is shown respectively an image 1500 of a target object 1530 enclosed in a transparent container 1510 and a 3D representation 1550 of the target object 1530 generated according to the present disclosure. In this example, image 1500 is one of 60 artificially rendered images generated for testing purposes where each image is generated for a differentcamera pose and having a resolution 800×800 pixels. As may be determined from inspection, the 3D representation 1550 of target object reproduces the detailed features of target object.

[0188] Referring now to FIG.16A and FIG.16B there is shown respectively an image 1600 of a target object 1630 enclosed in a transparent container 1610 and a 3D representation 1650 of the target object 1530 generated according to the present disclosure. In this example, image 1600 is one of a number of images captured of in this case a target object in the form of an insect preserved in a clear plastic material. In this example, the 3D representation is based on a potential set of 60 captured images each having a resolution of 1728×1152 pixels and corresponding to a different camera pose.

[0189] In another example, the transparent container may be in the form of a container comprising a transparent container wall that contains a transparent medium in which the target object is enclosed where the container wall and the contained transparent medium have different refractive indices. In one example, the container wall is made of a solid material such as glass or plastic and the transparent medium is a transparent liquid. In one example, the container of transparent liquid may comprise a tank of water and the target object is an object located in the tank of water. In another example, the container wall is again made of a solid material such as glass or plastic and the transparent medium is air.

[0190] In accordance with the present disclosure, a 3D representation of the target object enclosed in the container, where the container comprises a transparent container wall that contains a transparent medium, may be generated based on a knowledge of the container geometry including the thickness of the container walls. In this example, the ray tracing (eg, see FIG.4 and / or FIG.12) to generate the synthesised image for training of the neural radiance field model will also incorporate the refraction and reflection of traced rays at the air / container interface surface and the container wall / transparent medium interface surface (eg, see Equations 4, 10 and 11) caused by the different refractive indices between air and the container wall (eg, glass or plastic) or between the container wall and the contained transparent medium (eg, water or air) respectively as well as the traversal of any ray through the container wall.

[0191] Other aspects of a method and system to generate a 3D representation of a target object enclosed in a transparent container in accordance with the present disclosure may be found in “Seeing Through the Glass: Neural 3D Reconstruction of Object Inside a Transparent Container”, Tong et al; and “Seeing Through the Glass: Neural 3D Reconstruction of Object Inside a Transparent Container Supplementary Material”, Tong et al, the disclosures of which are herein incorporated by reference in their entireties.

[0192] As can be seen from inspection, methods and systems in accordance with embodiments of the present disclosure are capable of generating a 3D representation of an object enclosed in atransparent container that compensates for the optical distortion caused by the interaction of light with a material (ie, the container) with a different refractive index.

[0193] The reference to any prior art in this specification is not, and should not be taken as, an acknowledgement or any form of suggestion that such prior art forms part of the common general knowledge.

[0194] It will be understood that the terms “comprise” and “include” and any of their derivatives (e.g. comprises, comprising, includes, including) as used in this specification, and the claims that follow, is to be taken to be inclusive of features to which the term refers, and is not meant to exclude the presence of any additional features unless otherwise stated or implied.

[0195] In some cases, a single embodiment may, for succinctness and / or to assist in understanding the scope of the disclosure, combine multiple features. It is to be understood that in such a case, these multiple features may be provided separately (in separate embodiments), or in any other suitable combination. Alternatively, where separate features are described in separate embodiments, these separate features may be combined into a single embodiment unless otherwise stated or implied. This also applies to the claims which can be recombined in any combination. That is a claim may be amended to include a feature defined in any other claim. Further a phrase referring to “at least one of” a list of items refers to any combination of those items, including single members. As an example, “at least one of: a, b, or c” is intended to cover: a, b, c, a-b, a-c, b-c, and a-b-c.

[0196] It will be appreciated by those skilled in the art that the disclosure is not restricted in its use to the particular application or applications described. Neither is the present disclosure restricted in its preferred embodiment with regard to the particular elements and / or features described or depicted herein. It will be appreciated that the disclosure is not limited to the embodiment or embodiments disclosed, but is capable of numerous rearrangements, modifications and substitutions without departing from the scope as set forth and defined by the following claims.

Claims

CLAIMS:

1. A method comprising: receiving, by at least one processor, captured images of a scene comprising a target object enclosed in a transparent container having a container geometry, each of the captured images of the target object having an associated viewing position and direction; training, by the at least one processor, a neural radiance field model configured to represent the geometry and appearance of the scene based on evaluating a loss function comparing captured images with corresponding synthesised images; and generating, by the at least one processor, a 3D representation of the target object from the trained neural radiance field model.

2. The method of claim 1, wherein the neural radiance field model comprises a first neural network to represent the geometry of the scene and a second neural network to represent the appearance of the scene.

3. The method of claim 1 or 2, further comprising generating, by the at least one processor, a corresponding synthesised image for a captured image by: tracing rays, by the at least one processor, based on the associated camera viewing position and direction of the captured image and generating, by the at least one processor, the corresponding synthesised image based on the traced rays.

4. The method of claim 3, wherein tracing rays by the at least one processor based on the associated camera viewing position and direction of the captured image comprises: generating, by the at least one processor, for a selected ray originating from a pixel of the captured image and intersecting the container, an associated set of ray paths arising from optical interactions of the selected ray with the container, wherein a ray path corresponds to a collection of ray segments commencing with an initial ray segment from the camera to the container surface and terminating with a final ray segment exiting, or reflected from, the container surface.

5. The method of claim 4, wherein generating by the at least one processor, the corresponding synthesised image based on the traced rays comprises: determining, by the at least one processor, an irradiance contribution for each ray segment of the associated set of ray paths of the selected ray; and determining, by the at least one processor, the colour of the selected ray by accumulating the irradiance contribution from each ray segment.

6. The method of claim 5, wherein determining by the at least one processor the irradiance contribution for each ray segment comprises: on determining by the at least one processor that a ray segment is within the container then determining, by the at least one processor, the irradiance contribution based on a weighted integration of radiance along the ray segment based on the neural radiance field model of the scene; or on determining by the at least one processor that a ray segment is outside of the container then determining the irradiance contribution based on an ambient irradiance of the scene.

7. The method of claim 5 or 6, wherein determining by the at least one processor the colour of the selected ray by accumulating the irradiance contribution from each ray segment comprises: determining, by the at least one processor, a ray path irradiance contribution for each ray path of the set of ray paths associated with the selected ray by summing together a weighted irradiance contribution for each ray segment of the ray path starting with the final ray segment and finishing with the initial ray segment, and summing together, by the at least one processor, the ray path irradiance contributions from each ray path to determine the colour of the selected ray.

8. The method of claim 7, wherein summing together by the at least one processor the weighted irradiance contribution for each ray segment of the ray path starting with the final ray segment and finishing with the initial ray segment comprises: initialising, by the at least one processor, the ray path irradiance contribution with the irradiance contribution of the final ray segment corresponding to outside the container; successively adding, by the at least one processor, weighted irradiance contributions for each of the ray segments in the ray path before the initial ray segment, wherein each weighted irradiance contribution corresponds to the irradiance contribution of the ray segment weighted by an optical interaction factor for the ray segment, a colour correction factor for the ray segment, an occupancy factor for the ray segment, and a transmission factor for the parent ray segment; and adding, by the at least one processor, a weighted irradiance contribution for the initial ray segment corresponding to outside the container, wherein the weighted irradiance contribution corresponds to the irradiance contribution of the ray segment weighted by an optical interaction factor for the ray segment, and a transmission factor for the parent ray segment.

9. The method of any one of claims 6 to 8, wherein the weighted integration is based on a hierarchical sampling process.

10. The method of any one of claims 2 to 9, wherein the first neural network is a multi-layer perceptron that represents the geometry of the scene as a signed distance function.

11. The method of any one of claims 2 to 10, wherein the second neural network is a multi-layer perceptron that represents the appearance of the scene as an irradiance for input point, view direction and normal vector.

12. The method of any one of the preceding claims, further comprising determining, by the at least one processor, the container geometry from one or more of the captured images.

13. The method of any one of the preceding claims, wherein the loss function comprises a photometric loss function.

14. The method of any one of the preceding claims, wherein the container comprises a transparent container wall that contains a transparent medium, wherein the transparent container wall and the contained transparent medium have different refractive indices.

15. A system comprising: at least one processor; and a storage operatively coupled to the at least one processor and for storing instructions that when executed by the at least one processor cause the at least one processor to: receive captured images of a scene comprising a target object enclosed in a transparent container having a container geometry, each of the captured images of the target object having an associated viewing position and direction; train a neural radiance field model configured to represent the geometry and appearance of the scene based on evaluating a loss function comparing captured images with corresponding synthesised images; and generate a 3D representation of the target object from the trained neural radiance field model.

16. The system of claim 15, wherein the neural radiance field model comprises a first neural network to represent the geometry of the scene and a second neural network to represent the appearance of the scene.

17. The system of claim 15 or 16, wherein a corresponding synthesised image for a captured image is generated by: tracing rays based on the associated camera viewing position and direction of the captured image and generating the corresponding synthesised image based on the traced rays.

18. The system of claim 17, wherein tracing rays based on the associated camera viewing position and direction of the captured image comprises: generating for a selected ray originating from a pixel of the captured image and intersecting the container, an associated set of ray paths arising from optical interactions of the selected ray with the container, wherein a ray path corresponds to a collection of ray segments commencing with an initial ray segment from the camera to the container surface and terminating with a final ray segment exiting, or reflected from, the container surface.

19. The system of claim 18, wherein generating the corresponding synthesised image based on the traced rays comprises: determining an irradiance contribution for each ray segment of the associated set of ray paths of the selected ray; and determining the colour of the selected ray by accumulating the irradiance contribution from each ray segment.

20. The system of claim 19, wherein determining the irradiance contribution for each ray segment comprises: on determining that a ray segment is within the container then determining the irradiance contribution based on a weighted integration of radiance along the ray segment based on the neural radiance field model of the scene; or on determining that a ray segment is outside of the container then determining the irradiance contribution based on an ambient irradiance of the scene.

21. The system of claim 19 or 20, wherein determining the colour of the selected ray by accumulating the irradiance contribution from each ray segment comprises: determining a ray path irradiance contribution for each ray path of the set of ray paths associated with the selected ray by summing together a weighted irradiance contribution for each ray segment of the ray path starting with the final ray segment and finishing with the initial ray segment, and summing together the ray path irradiance contributions from each ray path to determine the colour of the selected ray.

22. The system of claim 21, wherein summing together the weighted irradiance contribution for each ray segment of the ray path starting with the final ray segment and finishing with the initial ray segment comprises: initialising the ray path irradiance contribution with the irradiance contribution of the final ray segment corresponding to outside the container; successively adding weighted irradiance contributions for each of the ray segments in the ray path before the initial ray segment, wherein each weighted irradiance contribution corresponds to theirradiance contribution of the ray segment weighted by an optical interaction factor for the ray segment, a colour correction factor for the ray segment, an occupancy factor for the ray segment, and a transmission factor for the parent ray segment; and adding a weighted irradiance contribution for the initial ray segment corresponding to outside the container, wherein the weighted irradiance contribution corresponds to the irradiance contribution of the ray segment weighted by an optical interaction factor for the ray segment, and a transmission factor for the parent ray segment.

23. The system of any one of claims 20 to 22, wherein the weighted integration is based on a hierarchical sampling process.

24. The system of any one of claims 16 to 23, wherein the first neural network is a multi-layer perceptron that represents the geometry of the scene as a signed distance function.

25. The system of any one of claims 16 to 24, wherein the second neural network is a multi-layer perceptron that represents the appearance of the scene as an irradiance for input point, view direction and normal vector.

26. The system of any one of claims 15 to 25, further comprising determining the container geometry from one or more of the captured images.

27. The system of any one of claims 15 to 26, wherein the loss function comprises a photometric loss function.

28. The method of any one of claims 15 to 27, wherein the container comprises a transparent container wall that contains a transparent medium, wherein the transparent container wall and the contained transparent medium have different refractive indices.

29. The system of any one of claims 15 to 28, further comprising a camera to capture the captured images of the scene.

30. A computer program product including one or more non-transitory machine readable mediums having instructions encoded thereon that when executed by at least one processor causes a process for generation of a 3D representation of a target object to be carried out, the process comprising: receiving, by the at least one processor, captured images of a scene comprising the target object enclosed in a transparent container having a container geometry, each of the captured images of the target object having an associated viewing position and direction;training, by the at least one processor, a neural radiance field model configured to represent the geometry and appearance of the scene based on evaluating a loss function comparing captured images with corresponding synthesised images; and generating, by the at least one processor, the 3D representation of the target object from the trained neural radiance field model.

31. The computer program product of claim 30, wherein the neural radiance field model comprises a first neural network to represent the geometry of the scene and a second neural network to represent the appearance of the scene.

32. The computer program product of claim 30 or 31, further comprising generating, by the at least one processor, a corresponding synthesised image for a captured image by: tracing rays, by the at least one processor, based on the associated camera viewing position and direction of the captured image and generating, by the at least one processor, the corresponding synthesised image based on the traced rays.

33. The computer program product of claim 32, wherein tracing rays by the at least one processor based on the associated camera viewing position and direction of the captured image comprises: generating, by the at least one processor, for a selected ray originating from a pixel of the captured image and intersecting the container, an associated set of ray paths arising from optical interactions of the selected ray with the container, wherein a ray path corresponds to a collection of ray segments commencing with an initial ray segment from the camera to the container surface and terminating with a final ray segment exiting, or reflected from, the container surface.

34. The computer program product of claim 33, wherein generating by the at least one processor, the corresponding synthesised image based on the traced rays comprises: determining, by the at least one processor, an irradiance contribution for each ray segment of the associated set of ray paths of the selected ray; and determining, by the at least one processor, the colour of the selected ray by accumulating the irradiance contribution from each ray segment.

35. The computer program product of claim 34, wherein determining by the at least one processor the irradiance contribution for each ray segment comprises: on determining by the at least one processor that a ray segment is within the container then determining, by the at least one processor, the irradiance contribution based on a weighted integration of radiance along the ray segment based on the neural radiance field model of the scene; oron determining by the at least one processor that a ray segment is outside of the container then determining the irradiance contribution based on an ambient irradiance of the scene.

36. The computer program product of claim 34 or 35, wherein determining by the at least one processor the colour of the selected ray by accumulating the irradiance contribution from each ray segment comprises: determining, by the at least one processor, a ray path irradiance contribution for each ray path of the set of ray paths associated with the selected ray by summing together a weighted irradiance contribution for each ray segment of the ray path starting with the final ray segment and finishing with the initial ray segment, and summing together, by the at least one processor, the ray path irradiance contributions from each ray path to determine the colour of the selected ray.

37. The computer program product of claim 36, wherein summing together by the at least one processor the weighted irradiance contribution for each ray segment of the ray path starting with the final ray segment and finishing with the initial ray segment comprises: initialising, by the at least one processor, the ray path irradiance contribution with the irradiance contribution of the final ray segment corresponding to outside the container; successively adding, by the at least one processor, weighted irradiance contributions for each of the ray segments in the ray path before the initial ray segment, wherein each weighted irradiance contribution corresponds to the irradiance contribution of the ray segment weighted by an optical interaction factor for the ray segment, a colour correction factor for the ray segment, an occupancy factor for the ray segment, and a transmission factor for the parent ray segment; and adding, by the at least one processor, a weighted irradiance contribution for the initial ray segment corresponding to outside the container, wherein the weighted irradiance contribution corresponds to the irradiance contribution of the ray segment weighted by an optical interaction factor for the ray segment, and a transmission factor for the parent ray segment.

38. The computer program product of any one of claims 35 to 37, wherein the weighted integration is based on a hierarchical sampling process.

39. The computer program product of any one of claims 31 to 38, wherein the first neural network is a multi-layer perceptron that represents the geometry of the scene as a signed distance function.

40. The computer program product of any one of claims 31 to 39, wherein the second neural network is a multi-layer perceptron that represents the appearance of the scene as an irradiance for input point, view direction and normal vector.

41. The computer program product of any one of claims 30 to 40, further comprising determining, by the at least one processor, the container geometry from one or more of the captured images.

42. The computer program product of any one of claims 30 to 41, wherein the loss function comprises a photometric loss function.

43. The computer program product of any one of claims 30 to 42, wherein the container comprises a transparent container wall that contains a transparent medium, wherein the transparent container wall and the contained transparent medium have different refractive indices.

44. A system comprising: at least one processor; and a storage operatively coupled to the at least one processor and for storing instructions that when executed by the at least one processor cause the at least one processor to carry out the method of any one of claims 1 to 14.

45. The system of claim 44, further comprising image capture means to capture the captured images of the scene.

46. A computer program product including one or more non-transitory machine readable mediums having instructions encoded thereon that when executed by at least one processor causes the method of any one of claims 1 to 14 to be carried out.

Images