Computer-implemented method and system for retrieving multispectral BRDF parameters

Abstract

The present invention generally relates to a computer-implemented method and apparatus for retrieving multispectral bidirectional reflectance distribution function (BRDF) parameters by using red-green-blue-depth (RGB-D) data. The method comprises the following steps: capturing (S1) at least one image of one or more objects in a scene by an RGB-D camera, wherein the captured at least one image of the one or more objects comprises RGB-D data, the RGB-D data comprising color and geometry information of the objects; reconstructing (S2) the captured at least one image of the one or more objects into one or more 3D reconstructions by a processing unit by using the RGB-D data; classifying (S3) the BRDF of the surface of the one or more objects based on the 3D reconstruction by a deep neural network, wherein the deep neural network comprises an input layer, an output layer and at least one hidden layer between the input layer and the output layer; and retrieving (S4) multispectral BRDF parameters by approximating the classified BRDF using an iterative optimization method. Thanks to the present invention, multispectral BRDF parameters of objects in a specific environment can be retrieved.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method, system, handheld device and program for retrieving multispectral BRDF parameters. Background Art

[0002] In modern development processes, a large number of different simulations are performed before the actual development process and / or prototyping process begins. For the simulation and to achieve realistic simulation results, it is necessary to simulate the materials to be used in the simulation as realistically as possible. Thus, materials play an important role in rendering realistic 3D objects in simulations, such as computer graphics. In particular, the materials used can define how light interacts with the object. This interaction should be identified and described in a way that can be used for further use.

[0003] Some material groups are defined by a function called a bidirectional reflectance distribution function (BRDF). BRDF defines how light or other electromagnetic radiation interacts with the surface of an object. In addition, BRDF represents a function for the reflection behavior of the material surface at any angle of incidence. For each light beam incident on the material at a given angle of incidence, it provides a quotient of the radiation density and irradiance for each light beam emitted. For example, the incident light may not be transmitted under the surface, or it is approximated. For photo-realistic renderings, the BRDF function is considered to be independent of wavelength and only the R, G, and B channels are retrieved. For sensor-real rendering, wavelength can play an important role (such as lidar, infrared sensors, etc.). Due to the unknown behavior of the material in different states and therefore its BRDF is unknown, it is necessary to measure this information.

[0004] State-of-the-art techniques require complex and extensive setups and expensive equipment. Moreover, to measure e.g. automotive materials, a large number of iterations with state-of-the-art techniques in different local illumination models are required, taking a lot of time and hardware for such data collection and making it less efficient. Furthermore, in cases where large asset databases of these materials are used (geometry, materials, textures belonging to the simulation world), the option to collect and measure materials is expensive, as simple storage of finely divided samples is memory-intensive. Additionally, multispectral properties are generally not available.

[0005] A method for image recording and image processing is known from US 2018 / 0047208 A1, for example, recording a three-dimensional scan of an object and rendering the three-dimensional scan in a virtual scene or environment.

[0006] There is a need for more efficient methods and systems to measure the BRDF of materials for simulation. Summary of the invention

[0007] The object of the present invention is to improve the retrieval of multispectral BRDF parameters. In particular, the above-mentioned disadvantages should be eliminated.

[0008] This object is achieved by the subject matter according to the independent claims with the corresponding claimed features. Advantageous embodiments are the subject matter of the dependent claims, the description and the figures.

[0009] According to a first aspect, the present invention relates to a computer-implemented method for retrieving multispectral bidirectional reflectance distribution function BRDF parameters by using red, green, and blue depth RGB-D data. The method comprises the following steps:

[0010] - At least one image of one or more objects in the scene is captured by an RGB-D camera, wherein the at least one image captured of the one or more objects comprises RGB-D data, and the RGB-D data comprises color and geometric shape information of the object.

[0011] The use of RGB-D cameras provides images that include highly accurate information about objects. An RGB-D image is a combination of an RGB image and its corresponding depth image. The depth image is an image channel where each pixel is associated with the distance between the image plane and the corresponding object in the RGB image.

[0012] Color and geometry information may include information of points on the surface of one or more objects. This information may include RGB values ​​and depth values, in particular the distances of these points taken from a specific direction of an RGB-D camera. The RGB values ​​include red, green and blue components for each individual point. The respective colors of each point of each pixel in the image are determined by the combination of the red, green and blue intensities stored in each color plane at the pixel position. The image may be stored in a graphic file format having, for example, 24 bits, with each red, green and blue component being 8 bits. This produces 16 million color possibilities. Further, a graphic file format such as 8 or 16 bits may be used, in which a bit distribution similar to 24 bits is applied.

[0013] - reconstructing, by the processing unit, the at least one image taken of the one or more objects into one or more 3D reconstructions by using the RGB-D data. The reconstruction step may comprise a digital creation of the surface of the object, which digital creation corresponds to some extent to the points taken by the RGB-D camera.

[0014] - Classifying the BRDF of the surface of one or more objects based on the 3D reconstruction by a deep neural network, wherein the deep neural network includes an input layer, an output layer, and at least one hidden layer between the input layer and the output layer.

[0015] - Retrieve multispectral BRDF parameters by using an iterative optimization method to approximate the classified BRDF.

[0016] Using the method and system according to the present invention, multispectral properties of a material can be retrieved based on the material type which is unknown but required for sensor-realistic rendering.

[0017] Furthermore, the optimization process includes classification and multispectral parameter estimation, which makes the retrieval of multispectral BRDF parameters more effective, fast and accurate.

[0018] Advantageously, the retrieved BRDF parameters of an object can be used to verify the correct operation of sensors, for example in a car, and / or to calibrate the sensors to correctly measure the surface of an object and thus also the object (e.g. car, road sign) itself. For example, identifying objects is important for autonomous driving. The car's sensors must detect every object located around the car, regardless of the material and in every lighting condition (i.e. reflection, intensity, etc.). By using the retrieved BRDF parameters for simulation, it is more efficient to verify the sensors in simulation, since verification and calibration for all materials under varying lighting conditions no longer needs to be performed in the outside real environment. Every lighting condition and material behavior described by a BRDF function is available.

[0019] Furthermore, the retrieved multispectral BRDF parameters of a material may be stored in a material asset database and may be used for simulation and testing purposes, as well as for creating digital environments and objects in computer and film scenes.

[0020] In a preferred embodiment of the present invention, the step of retrieving multispectral BRDF parameters further comprises estimating spectral parameters of the BRDF.

[0021] In a preferred embodiment of the present invention, the step of retrieving multispectral BRDF parameters further comprises estimating spectral properties of an environment map. The spectral properties of the environment map describe the light arriving at the point where the BRDF parameters are estimated from all directions.

[0022] Typically, the BRDF is written as a function of two unit vectors and a function of wavelength. These two vectors encode the incident and outgoing directions of the reflection relative to the surface normal. When arriving in the first direction, the BRDF measures how much light is scattered in the second direction. In general, the BRDF is modeled as a function of a parameter p that is independent of the wavelength λ. By using a function based on spherical harmonics per λ, the environment map can be modeled. When an RGB-D camera is used to reconstruct an object, the RGB-D camera integrates the spectral curve (camera spectral curve) per RGB channel per point about the object's reflectivity, which is a combination of the BRDF and the environment. In this way, the spectral BRDF parameters can be retrieved.

[0023] In a preferred embodiment of the present invention, a 3D reconstruction of one or more objects includes RGB values ​​and surface normals of one or more objects. A surface normal (also described as a normal) can be a line or vector of an object such as being perpendicular to a given object. For example, in two-dimensional space, the normal of a curve at a given point is a line perpendicular to the tangent of the curve at that point. In three-dimensional space, the normal of a surface at a point of an object is a vector perpendicular to a tangent to the surface at that point. Normals can be used to determine the orientation of a surface toward a light source, such as a flat shadow, or the orientation of each corner to simulate a curved surface.

[0024] In a preferred embodiment of the present invention, the classified BRDF includes spectral parameters of the BRDF and spectral properties of an environment map of one or more objects. The spectral parameters of the BRDF can be measured using the camera RGB-D specifications. With the spectral curve of the known camera, the RGB-D model of the surface and the outgoing light of the surface can be measured. In order to render an image with a spectral BRDF, it is necessary to provide an environment for the material to interact with it. Advantageously, the environment map describes the given ambient lighting. The appearance of the surface, especially the reflective surface can be approximated. In addition, by providing the spectral properties of the environment map, retrieving multi-spectral BRDF parameters is independent of the environment and its light conditions. Therefore, it is not required to provide a darkroom or laboratory with specific and adjustable lighting settings.

[0025] In a preferred embodiment of the present invention, the deep neural network comprises a convolutional neural network. In another preferred embodiment of the present invention, the deep neural network comprises a recursive neural network. In another preferred embodiment of the present invention, the deep neural network comprises a feedforward neural network.

[0026] A deep neural network is a neural network of a certain complexity consisting of more than two layers. A deep neural network uses mathematical models to process data in a complex way. Deep neural networks generally include techniques that simulate human brain activity, especially pattern recognition. A neural network includes an input layer, an output layer, and at least one hidden layer between the input layer and the output layer. Each layer performs a specific type of data sorting and filing. In particular, neural networks are suitable for processing unlabeled and / or unstructured data. In particular, neural networks represent a type of machine learning in which artificial intelligence is used to classify and sort information.

[0027] The network is a deep learning (also known as deep structured learning or hierarchical learning) artificial neural network (ANN). It may contain more than one hidden layer. The network uses a machine learning method based on learning data representations, rather than task-specific algorithms. Preferably, the learning is supervised or partially supervised. The network may also consist of one or more convolutional layers. In particular, it may consist of a fully convolutional network, which is trained in an end-to-end manner, optionally with skip connections. It uses pooling layers and deconvolution layers. The network takes an image as input and provides one or more images as output. The network is trained in an end-to-end manner, for example by directly comparing the entire generated output with the entire ground truth, such as the entire segmentation mask (rather than considering patches and labels). It may be trained using deep supervision, where the loss function consists of a combination of costs calculated at each resolution.

[0028] Training can include providing different light directions and different RGB-D models for different materials, resulting in different BRDFs. The combination of input and expected output can be used to train a deep neural network.

[0029] According to a preferred embodiment, the iterative optimization method includes L2 norm optimization. L2 norm optimization is equivalent to least squares. Least squares is a method of approximating the solution of an overdetermined system of equations in regression analysis. It basically minimizes the sum of squares of the difference between the target value and the estimated value. Since the L2 norm squares the error (if the error is > 1, it increases a lot), the model will see a much larger error than, for example, the L1 norm, so the model is more sensitive to this example, and the model is adjusted to minimize this error.

[0030] The L2 norm includes an analytical solution, which computationally allows the L2 norm solution to be computed efficiently.

[0031] According to another preferred embodiment, the iterative optimization method comprises Levenberg-Marquandt optimization. The Levenberg-Marquardt algorithm is a numerical optimization algorithm for solving nonlinear equilibrium problems using the method of least squares. The method combines the Gauss-Newton method with a regularization technique that forces the function value to decrease. The Levenberg-Marquardt algorithm is much more robust than the Gauss-Newton method, i.e., it converges with a high probability even under poor starting conditions.

[0032] In this way, by using iterative optimization, errors that occur when estimating BRDF parameters due to different and unpredictable light incidence on the object surface can be minimized. Thus, the present invention allows to retrieve the BRDF in unknown environments without previously defined light conditions. The object can be placed under different lighting conditions, and the material of the object will behave according to these different lighting conditions.

[0033] In a further possible embodiment, the iterative optimization minimizes the approximation error until the amount of error in the estimate is lower than the amount of error in the manual measurement. In this way, the error of the light and the error about the BRDF can be minimized. If the amount of error in the estimate is lower than the error in the manual measurement, it indicates that the environment corresponding to the spectral parameters of the BRDF has been found. In this case, these parameters correspond to the BRDF parameters of the material that has been photographed. The iteration can be stopped, otherwise it continues. For example, for the training phase of a deep neural network, manual measurements have been continued. In a further embodiment, a threshold is added to the estimated error. The threshold can be determined experimentally.

[0034] The higher the threshold is chosen, the more accurate the estimate will be. The lower the threshold is chosen, the less time the method takes to calculate and retrieve the multispectral BRDF parameters.

[0035] So far, the present invention has been described with respect to the claimed method. Features, advantages or alternative embodiments of the present invention may be assigned to other claimed objects (e.g., computer programs or systems, i.e., devices or computer program products), and vice versa. In other words, the subject matter claimed or described about the system may be improved with the features described or claimed in the context of the method, and vice versa. In this case, the functional features of the method are embodied by the structural units of the system, respectively, and vice versa. Generally, in computer science, software implementations and corresponding hardware implementations are equivalent. Therefore, for example, the method steps for "storing" data may be performed with a storage unit and corresponding instructions for writing data into storage. In order to avoid redundancy, although the device may also be used in the alternative embodiments described with reference to the method, these embodiments are no longer explicitly described for the device.

[0036] In another aspect, the present invention may relate to a system for retrieving multispectral bidirectional reflectance distribution function BRDF parameters by using red, green, blue, depth RGBD data, the system comprising:

[0037] - an image capturing device adapted to capture at least one image of one or more objects in a scene, wherein the captured at least one image of the one or more objects comprises RGBD data, the RGBD data comprising color and geometry information of the objects;

[0038] -Computer unit, comprising:

[0039] - a memory unit for storing a deep neural network, the deep neural network comprising an input layer, an output layer and at least one hidden layer between the input layer and the output layer;

[0040] - at least one processing unit adapted to store program instructions which, when executed on the at least one processing unit, cause the system to:

[0041] - reconstructing at least one captured image of the one or more objects into one or more 3D reconstructions by using the RGDB data;

[0042] - using a deep neural network, classifying the BRDF type on the surface of one or more objects based on the 3D reconstruction; and

[0043] - Retrieve multispectral BRDF parameters by using an iterative optimization method to approximate the classified BRDF types.

[0044] In another aspect, the present invention may be implemented as a handheld device configured to perform one of the embodiments of the method. The handheld device may include, for example, a mobile phone, a tablet computer, a laptop, a notebook computer and / or a PDA. A camera, in particular an RGB-D camera, may be included in the handheld device or connected to a connection interface of the handheld device via a wired or wireless connection.

[0045] In another aspect, the present invention can be implemented as a computer program product comprising program elements, which, when loaded into the memory of an electronic device, causes the electronic device to perform the steps of the method for retrieving multispectral bidirectional reflectance distribution function BRDF parameters according to any of the preceding method claims. The electronic device includes each device suitable for executing the program.

[0046] As part of the invention, not all steps of the method must be performed on the same component or computer instance, but may also be performed on different computer instances.

[0047] In addition, it is possible that, as a distributed system, individual steps of the above-described method may be performed in one unit, while the remaining components may be performed in another unit.

[0048] The present invention, in particular the aforementioned method, may be provided as a computer program loadable into a processing unit of a network element, such as a (e.g. mobile) device of a physician and / or a server. The computer program comprises code suitable for performing the steps of the method as mentioned above when processed by the processing unit. The computer program may be stored on a non-volatile computer readable medium or data carrier, such as a memory stick. The computer program may also be downloaded from a server entity in a downloadable form. The computer program may be virtualized and / or may be distributed on different network nodes.

[0049] A computer program may be provided, marketed, used, imported and / or stored as a computer program product (and may therefore be part of a computer program product). Thus, a computer program may be provided on a storage medium (a computer readable medium, such as a computer disk or memory stick, etc.). Alternatively, the computer program may be provided via download with the aid of a corresponding network connection to a server, which stores the computer program by providing a link to a server on which the computer program is stored. A "computer readable medium" or "storage medium" may be any medium that can contain, store, communicate, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. A computer readable medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device, equipment or propagation medium. More specific examples (a non-exhaustive list) of computer readable media may include the following: an electrical connection with one or more conductors, a portable computer floppy disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, and a portable compact disk read-only memory (CDROM).

[0050] The term "computer" or "computing unit" refers to any electronic device including a processor, such as a general-purpose central processing unit (CPU), a dedicated processor or a microcontroller. The processor is suitable for performing a specific computing task, i.e., for retrieving multispectral bidirectional reflectance distribution function (BRDF) parameters. The computer or computing unit is capable of receiving data (input), being capable of performing a predetermined sequence of operations thereon, and being capable of generating results therefrom in the form of information or signals (output). Depending on the context, the term "computer" or "computing unit" will mean a processor in particular, or may more generally refer to a processor associated with a collection of related elements contained within a single chassis or housing.

[0051] The systems and methods described herein can be implemented by a computer program or multiple computer programs, which can exist in various forms in both active and inactive states in a single computer system or across multiple computer systems. For example, they can exist as (multiple) software programs, which consist of program instructions in source code, object code, executable code, or other formats for performing some steps. Any of the above can be implemented in compressed or uncompressed form on a computer-readable medium including storage devices and signals.

[0052] In view of the following description and embodiments, the properties, features and advantages of the present invention described above, and the modes of their implementation (which will be described in more detail in the context of the figures) become clearer and more easily understood. The following description does not limit the present invention to the embodiments included. In different figures, the same components or parts may be marked with the same reference numerals. Generally, these figures are not to scale. It should be understood that the preferred embodiments of the present invention may also be any combination of dependent claims or the above-described embodiments with corresponding independent claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In the following, possible embodiments of different aspects of the present invention will be described in more detail with reference to the accompanying drawings.

[0054] Figure 1 A flow chart illustrating a possible embodiment of a method for retrieving multispectral bidirectional reflectance distribution function (BRDF) parameters; and

[0055] Figure 2 A block diagram of a possible embodiment of a system architecture for retrieving multispectral bidirectional reflectance distribution function BRDF parameters by using red, green, and blue depth RGB-D data is shown. DETAILED DESCRIPTION

[0056] Figure 1 A flow chart of a possible embodiment of a method 1 for retrieving multispectral bidirectional reflectance distribution function BRDF parameters is schematically shown. BRDF is a function of the reflective behavior of a material surface at any angle of incidence. For each light beam incident on a material at a given angle of incidence, it provides the quotient of the radiation density and the irradiance for each light beam emitted. For example, BRDF is used in realistic 3D computer graphics, where it represents part of the basic rendering equation and is used to represent surfaces as realistically and physically correctly as possible.

[0057] In the illustrated exemplary embodiment, the method comprises several main steps. In a first step S1, one or more objects 30 in a scene are photographed by an RGB-D camera 11 (see Figure 2 At least one image of one or more objects 30 is captured (and corresponding description). At least one image captured of one or more objects 30 comprises RGB-D data, which comprises color and geometry information of the object 30. In step S1, parameters of the material are derived. This comprises measuring the exit angle for each incident angle by the RGB-D camera 11 to create a model for each individual material. Each material depicts the light on its surface in a different way, which is described by parameters. These parameters can be estimated by processing the images captured by the RGB-D camera 11.

[0058] By using the known spectral curve of the RGB-D camera 11, the multispectral BRDF properties are retrieved and the multispectral environment map can be identified. The material can be measured by using the RGB-D camera 11 only on one light spot. For example, a plate with a unique material surface of a specific size is photographed. The RGB-D camera 11 uses its image to capture points on the surface of the unique material as pixels including RGB-D data. The RGB-D data includes color information of red, green and blue as well as depth information. The image captured by the RGB-D camera 11 includes all light information (reflection) from the points on the surface to which the RGB-D camera 11 is pointed. Further, the light from the environment and the reflection of the surface are added together. Thus, the present invention estimates the direction of light and the parameters of the BRDF.

[0059] In a further step S2, at least one image of one or more objects 30 is reconstructed into one or more 3D reconstructions by a processing unit using the RGB-D data. The RGB-D data provided by the RGB-D camera includes RGB values ​​and depth values. These values ​​describe 3D points on the surface of the object 30 recorded by the image. The RGB values ​​and depth values ​​can be used to create a surface that conforms to these points. The created surface is used to estimate normals. This obtains digital information from points on the surface of the object 30, which includes directional RGB-D data. The digital information includes depth information for the points in each direction.

[0060] In a further step S3, the BRDF of the surface of one or more objects 30 is classified by the deep neural network 20 based on the 3D reconstruction. The deep neural network 20 includes an input layer, an output layer and at least one hidden layer between the input layer and the output layer. The 3D reconstruction of the object defined as the surface normal and the direction of the RGB value are placed in different layers of the deep neural network 20. The deep neural network 20 can be trained by several known input values ​​and output values. Training includes determining the relationship between the provided training input values ​​and the output values. Based on the trained data set, further BRDF classification can be performed by using unknown or new input data. For example, training can be performed for several material models required for testing and simulation purposes.

[0061] In a further step S4, multispectral BRDF parameters are retrieved by using an iterative optimization method to approximate the classified BRDF. In an embodiment, step S4 may include estimating the spectral parameters of the BRDF and estimating the spectral properties of the environment map. Step S4 is iterated It. until a very small error value is reached. The environment map defines the incident light from each direction. The environmental properties include the magnitude of the light from each direction. When a criterion (such as the estimated error amount) is reached, the iterative optimization method stops. The error amount can be determined by practical research, with the aim of achieving the maximum accuracy and / or performance of the method. The optimization method may include L2 norm optimization or Levenberg-Marquandt optimization. Advantageously, using the method of the present invention, the BRDF parameters from the surface of the object 30 can be estimated in each state and are independent of the lighting conditions.

[0062] In equation (1), the surface outgoing radiance from the object 30 can be calculated. Outgoing radiance Lo(x, ω o ,λ) can be obtained by Ω + The incident radiation rate Li(ω i ,λ) and the bidirectional reflectance distribution function ρ(x,ω i ,ω o ,λ,p) is calculated by the integral.

[0063]

[0064] The bidirectional reflectance distribution function includes x as the position on the surface of the object 30 to be scanned. It also includes the incident direction vector ω i and the outgoing direction vector ω o The parameter γ describes the wavelength of the light. The parameter n describes the surface normal.

[0065] In equation (2), the tristimulus representation of the color response (eg, RGB, XYZ) after the reconstruction step is shown. From each point on the surface of the object 30, the color response C k (x,ω o ) . Further, in order to estimate the color response C k (x,ω o ), using the wavelength-based stimulus function ∫ λ CIE k (λ). CIE k Describes standard curves, such as the CIE1931RGB color matching functions.

[0066]

[0067] In equation (3), specifying the classification step S3, the wavelength λ must be discretized. Equation (3) describes the output C of the sensor (RBG-D camera) k (x,ω o ). Parameters x, ω o is the result of reconstruction, and C k is the result of the RGB values. Equation (3) must be modified to obtain the bidirectional reflectance distribution function p and the incident radiance Li.

[0068]

[0069] In equation (5), the incident light is represented by spherical harmonics. Equation (5) is manipulated by inserting equation (4) to achieve better performance in calculating the method and to optimize the processing of the method.

[0070] ∑ m＝0 c m，k，j Y m (ω i ) (4)

[0071] Equation (4) describes Li(ω i ,λ j ) is represented by spherical harmonics. Parameter c m，k，j describes the environment map, and the parameter Y m Spectral parameters are described.

[0072]

[0073]

[0074] In equation (6), the terms are rearranged. For a given Ck ( x,ω o ), the environment map parameters (light) c can be estimated m，k，j and BRDF parameter ρ.

[0075]

[0076] The estimation includes the step of classifying the BRDF type on the surface by using a deep neural network. Further, a suitable initial guess for BRDFp is selected. Subsequently, the estimation is repeated (continued) in a loop until convergence (error metric) is reached. The loop includes estimating the environment map parameters c m，k，j (using non-parametric estimation of the matrix) and using different optimization methods to estimate BRDFp, such as the L2 norm or Levenberg-Marquandt optimization.

[0077] Figure 2A block diagram of a possible embodiment of a system architecture 10 for retrieving multispectral bidirectional reflectance distribution function BRDF parameters by using red, green, and blue depth RGB-D data is schematically shown.

[0078] As in Figure 2 As seen in the block diagram of , the system 10 is used to retrieve multispectral bidirectional reflectance distribution function BRDF parameters by using red, green, and blue depth RGB-D data. In the illustrated embodiment, the system 10 includes an RGB-D camera 11 and a computing unit 12. The RGB-D camera 11 and the computing unit 12 are constructed as two independent entities. In this embodiment, the RGB-D camera 11 and the computing unit 12 are connected via a communication connection 15. The communication connection 15 can be used for bidirectional data exchange between the computing unit 12 and the RGB-D camera 11. The data exchange can include sending and receiving control signals and / or data signals. The communication connection 15 can include a wireless communication connection such as Wi-Fi, Bluetooth, Bluetooth low energy, NFC, infrared, and / or a wired communication connection such as a serial communication connection, including PCI-Express, Gigabit Ethernet, HDMI, USB standard 3.1 and lower, SATA, DVI or Ethernet. The RGB-D camera 11 and the computing unit 12 include a communication connection interface for connecting to the corresponding communication connection 15.

[0079] In the illustrated embodiment, the computing unit 12 includes a memory unit 13 and a processing unit 14. The computing unit 12 in the system 10 for retrieving multispectral bidirectional reflectance distribution function BRDF parameters can be formed as a computer, a personal computer, or a workstation in a computer network, and can include a processing unit 14 (one or more processors), a memory unit 13, and a system bus that couples various system components including the memory unit 13 to the processing unit 14. The system bus can be one of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, and a local bus using any number of bus architectures. The memory unit 13 can include a read-only memory (ROM) and / or a random access memory (RAM). A basic input / output system (BIOS) can be stored in the ROM, which contains basic routines that help transfer information between elements within the PC, such as at startup. The computing unit can also include a hard disk drive for reading from and writing to a hard disk and an optical drive for reading from or writing to a removable (magnetic) optical disk (such as a compact disk (CD) or other (magnetic) optical media). The drive and associated storage media provide non-volatile storage of machine-readable instructions, data structures, program modules and other data for the computer. Although the exemplary environment described herein uses a hard disk and a removable (magnetic) optical disk, it will be appreciated by those skilled in the art that other types of storage media (such as flash memory cards, digital video disks, random access memories (RAMS), read-only memories (ROM), etc.) may be used to replace or supplement the storage devices presented above. Several program modules may be stored on a hard disk, (magnetic) optical disk, ROM or RAM, such as an operating system, one or more application programs, such as methods for calculating outputs and / or other program modules and / or program data.

[0080] A user can enter commands and information into the computer via input devices such as a keyboard and a pointing device. Other input devices may also be included, such as a microphone, a joystick, a scanner, etc. These and other input devices are often connected to the processing unit 14 via a serial interface coupled to the system bus. However, the input devices may also be connected via other interfaces, such as a parallel port, a game port, or a universal serial bus (USB). A monitor (e.g., a GUI) or other type of display device may also be connected to the system bus via an interface such as a video adapter. In addition to the display, the computer may also include other peripheral output devices, such as speakers and printers.

[0081] The computing unit 12 can operate in a network environment that defines logical connections to one or more remote computers. The remote computer can be another personal computer, a server, a router, a network PC, a peer device, or another shared network node, and can contain many or all of the above elements associated with a personal computer. Logical connections include local area networks (LANs) and wide area networks (WANs), intranets, and the Internet.

[0082] Furthermore, the computing unit 12 may be implemented as a system-on-chip design on a microcontroller or a programmable chip such as an ASIC or an FPGA.

[0083] The memory unit 13 may store the deep neural network 20 .

[0084] The processing unit 12 is adapted to store program instructions that, when executed on the processing unit 14, cause the system to reconstruct at least one captured image of one or more objects 30 into one or more 3D reconstructions by using RGB-D data. Further, the processing unit uses the deep neural network 20 to classify the BRDF on the surface of the one or more objects 30 based on the 3D reconstruction. Further, the processing unit 12 retrieves the multispectral BRDF parameters by using iterative optimization to approximate the classified BRDF. The processing unit 12 may include several processing units adapted to store and execute program instructions.

[0085] The RGB-D camera 11 may be a specific type of depth sensing camera device that may work in association with an RGB camera that is capable of enhancing a normal image of an object 30 with depth information (related to the distance to the sensor) on a per-pixel basis. The camera assembly may be an infrared sensor, an infrared camera and / or an RGB camera. The synchronized output stream of depth and color information is converted into spatial information. In the near infrared range, the IR projector emits a coded pattern of small dots visible to the human eye. For example, a CMOS sensor receives an image reflected by an object and / or scene and calculates a depth matrix (to which the RGB image may be assigned) with, for example, a VGA resolution based on the camera distance of the parallax across the corresponding points. The calculations may be performed on a chip in the camera using a parallel algorithm, which greatly reduces the load on the host computer. This describes only an exemplary embodiment of an RGB-D camera 11 used in a system 10 according to the present invention. The present invention is not limited to the above-described embodiments. Further configurations and implementations of the RGB-D camera are conceivable.

[0086] The RGB-D camera 11 is configured to capture at least one image of one or more objects 30 in a scene. In an embodiment, the RGB-D camera 11 captures a series of images of one or more objects 30 in a scene depending on the surface (material) of the object 30. An image and / or a series of images may be processed in the camera or by a processing unit 14 of the computer unit 12. In an embodiment, the RGB-D camera 11 may include a memory for storing an image and / or a series of images before processing or sending the image. The at least one captured image of the one or more objects 30 includes RGB-D data, which includes color and geometry information of the object 30.

[0087] In an alternative embodiment of the system 10 according to the invention, the system 10 comprising the RGB-D camera 11 and the computer unit 12 can be constructed in one single entity. The system 10 can be constructed as a handheld device, such as a mobile phone, a tablet, a laptop and / or a PDA, comprising the RGB-D camera 11, the processing unit 13 and the memory unit 13.

[0088] The scope of protection of the present invention is indicated by the accompanying claims and is not limited by the features explained in the description or shown in the drawings.

[0089] The present invention generally relates to a computer-implemented method and apparatus for retrieving multispectral bidirectional reflectance distribution function (BRDF) parameters using red, green, and blue depth (RGB-D) data. The method comprises the following steps:

[0090] At least one image of one or more objects 30 in the scene is captured by the RGB-D camera 11 S1, wherein the at least one image of the one or more objects 30 captured includes RGB-D data, and the RGB-D data includes color and geometric shape information of the object 30;

[0091] reconstructing S2 , by the processing unit, the at least one captured image of the one or more objects 30 into one or more 3D reconstructions by using the RGB-D data;

[0092] The BRDF of the surface of one or more objects 30 is classified S3 based on the 3D reconstruction by a deep neural network 20, wherein the deep neural network 20 includes an input layer, an output layer, and at least one hidden layer between the input layer and the output layer; and multispectral BRDF parameters are retrieved S4 by using an iterative optimization method to approximate the classified BRDF.

[0093] Thanks to the invention, it is possible to retrieve multispectral BRDF parameters of objects in a specific environment.

Claims

1. A computer-implemented method (1) for retrieving multispectral bidirectional reflectance distribution function (BRDF) parameters by using red, green, and blue depth (RGB-D) data, comprising the following steps: - capturing (S1) at least one image of one or more objects (30) in a scene by an RGB-D camera (11), wherein the captured at least one image of the one or more objects (30) comprises RGB-D data, and the RGB-D data comprises color and geometric shape information of the object (30); - reconstructing (S2) the at least one image taken of the one or more objects (30) into one or more 3D reconstructions by using the RGB-D data by a processing unit; It is characterized in that - classifying (S3) the BRDF of the surface of the one or more objects (30) based on the 3D reconstruction by a deep neural network (20), wherein the deep neural network (20) comprises an input layer, an output layer and at least one hidden layer between the input layer and the output layer; and - Retrieving (S4) multispectral BRDF parameters by approximating the classified BRDF using an iterative optimization method.

2. The method according to claim 1, wherein: The step of retrieving said multispectral BRDF parameters further comprises: - estimating multispectral parameters of said BRDF.

3. The method according to any one of claims 1 and 2, wherein: The step of retrieving the multispectral BRDF parameters also includes: - estimating the spectral properties of the environment map.

4. The method according to any one of claims 1 and 2, wherein: The 3D reconstruction of the one or more objects (30) comprises RGB values ​​and surface normals of the one or more objects.

5. The method according to any one of claims 1 and 2, wherein: The classified BRDF includes spectral parameters of the BRDF and spectral properties of an environment map of the one or more objects (30).

6. The method according to any one of claims 1 and 2, wherein: The deep neural network (20) includes at least a convolutional neural network, a recurrent neural network or a feedforward neural network.

7. The method according to claim 3, wherein: The iterative optimization includes L2 norm optimization or Levenberg-Marquandt optimization.

8. The method according to claim 7, wherein: The iterative optimization minimizes the approximation error until the amount of error in the estimate is lower than the amount of error in the manual measurement.

9. A system (10) for retrieving multispectral bidirectional reflectance distribution function (BRDF) parameters by using red, green, and blue depth (RGB-D) data, comprising: - an RGB-D camera (11) adapted to capture at least one image of one or more objects (30) in a scene, wherein the captured at least one image of the one or more objects (30) comprises RGB-D data, wherein the RGB-D data comprises color and geometry information of the object; - Computer unit (12), comprising: - a memory unit (13) for storing a deep neural network (20), the deep neural network comprising an input layer, an output layer and at least one hidden layer between the input layer and the output layer; - at least one processing unit (14) adapted to store program instructions which, when executed on the at least one processing unit (14), cause the system to: - reconstructing at least one captured image of the one or more objects (30) into one or more 3D reconstructions by using the RGB-D data; - using the deep neural network (20), classifying the BRDF on the surface of the one or more objects based on the 3D reconstruction; and - Retrieve multispectral BRDF parameters by approximating the classified BRDF using iterative optimization.

10. A handheld device (100) configured to perform the method according to any one of claims 1-8.

11. A computer program product comprising program elements which, when loaded into a memory of an electronic device, cause the electronic device to execute the steps of the method according to any one of claims 1 to 8.

Images