Neural segmentation fields for representing 3D scenes

Abstract

According to one example embodiment, a method for training a neural network to render a segmentation map corresponding to an arbitrarily selected view of a 3D scene includes calculating color texture and volume density corresponding to the selected training view of the 3D scene using a 3D representation pre-trained to represent the 3D scene, generating predicted segmentation maps corresponding to the selected training views of the 3D scene using the neural network based on the color texture and volume density, and adjusting configuration parameters of network nodes of the neural network based on a loss function configured to receive as a first input the ground truth segmentation maps corresponding to the selected training views and further receive as a second input the predicted segmentation maps.

Description

[Technical Field]

[0001] 1. Reference to Related Applications This application claims priority to U.S. Provisional Application No. 63 / 410,344, filed September 27, 2022, which is incorporated herein by reference in its entirety.

[0002] 2. Fields of Disclosure Various exemplary embodiments relate to rendering three-dimensional (3D) scenes in two-dimensional (2D) space. [Background technology]

[0003] 3.Background The process of linking each pixel in an image to a class label is called semantic segmentation. The labels can be, for example, "person," "flower," "building," etc. Semantic segmentation can be thought of as image classification at the pixel level. Thus, in semantic segmentation, every pixel in an image is typically associated with a corresponding class label. Semantic segmentation differs from object detection because, unlike object detection, semantic segmentation works at the pixel level to relatively accurately determine the contours of objects in an image.

[0004] Unlike semantic segmentation, which treats multiple objects in a class as a single entity, instance segmentation identifies individual objects within a class. Instance segmentation is sometimes considered a refinement of semantic segmentation. Various computer vision applications use either semantic segmentation or a combination of semantic and instance segmentation. Summary of the Invention

[0005] Overview of specific embodiments Various embodiments of methods and apparatuses for rendering segmentation maps of 3D scenes using machine learning are disclosed herein. According to one exemplary embodiment, a smaller neural network (NN) is trained to render a segmentation map corresponding to an arbitrarily selected view of the 3D scene. Here, the training is performed using a larger neural network pre-trained to represent the 3D scene with color texture and volumetric density. In other words, the smaller neural network is configured to function on top of the pre-trained larger neural network, thereby providing the ability to obtain segmentation information faster and with a relatively low additional burden on computational resources. Also disclosed are several embodiments of entropy-based loss functions and regularization terms therefor. These are configured to facilitate the training process for the smaller neural network, for example, by utilizing auxiliary information available in conjunction with or from previous training of the larger neural network.

[0006] According to an example embodiment, there is provided an image processing method comprising training, by a processor, a first neural network to render a segmentation map corresponding to an arbitrarily selected view of a 3D scene, the training including: calculating, by the processor, a color texture and volume density corresponding to the selected training view of the 3D scene using a 3D representation previously trained to represent the 3D scene; generating, by the processor, a predicted segmentation map corresponding to the selected training view of the 3D scene based on the color texture and the volume density using the first neural network; and adjusting, by the processor, configuration parameters of NN nodes of the first neural network based on a loss function configured to receive as a first input a ground truth segmentation map corresponding to the selected training view and further configured to receive the predicted segmentation map as a second input.

[0007] According to another exemplary embodiment, a non-transitory computer-readable medium is provided that stores instructions that, when executed by the processor, cause the processor to perform operations including the image processing methods described above or below.

[0008] According to yet another exemplary embodiment, there is provided an image processing device comprising at least one processor and at least one memory containing program code, the at least one memory and the program code configured by the at least one processor to cause the device to train at least a first neural network to render segmentation maps corresponding to arbitrarily selected views of a 3D scene. To train the first neural network, the device is configured to: calculate color texture and volume density corresponding to selected training views of the 3D scene, the calculation being performed using a 3D representation pre-trained to represent the 3D scene; generate predicted segmentation maps corresponding to the selected training views of the 3D scene, the generation being performed using the first neural network based on the color texture and the volume density; and adjust configuration parameters of NN nodes of the first neural network based on a loss function, the loss function receiving as a first input a ground truth segmentation map corresponding to the selected training view and the predicted segmentation map as a second input.

[0009] According to yet another exemplary embodiment, there is provided an image processing method including testing, by a processor, a first neural network to render a segmentation map corresponding to an arbitrarily selected view of a 3D scene, the testing step including: calculating, by the processor, a color texture and volume density corresponding to the selected view of the 3D scene, the calculating step being performed using a 3D representation pre-trained to represent the 3D scene; and generating, by the processor, a segmentation map corresponding to the selected view of the 3D scene based on the color texture and the volume density using the first neural network, the first neural network having been trained by the processor using the pre-trained 3D representation.

[0010] According to yet another exemplary embodiment, there is provided an image processing device comprising at least one processor and at least one memory containing program code, wherein the at least one memory and the program code are configured by the at least one processor to cause the device to test a first neural network to render a segmentation map corresponding to an arbitrarily selected view of a 3D scene, and to test the first neural network, the device is configured to: calculate a color texture and volume density corresponding to the selected view of the 3D scene, wherein the calculation is performed using a 3D representation pre-trained to represent the 3D scene; and generate a segmentation map corresponding to the selected view of the 3D scene, wherein the generation is performed using the first neural network based on the color texture and the volume density, and wherein the first neural network has been trained using the pre-trained 3D representation. [Brief explanation of the drawings]

[0011] Other aspects, features, and advantages of various disclosed embodiments will become more fully apparent, by way of example, from the following detailed description and accompanying drawings.

[0012] [Figure 1] 1A-1B are block diagrams illustrating example process flows for training and testing a neural segmentation field, respectively, according to various embodiments.

[0013] [Figure 2] FIG. 2 is a block diagram illustrating a multi-layer perceptron (MLP) that can be used to implement a neural segmentation field, according to one embodiment.

[0014] [Figure 3] FIG. 3 is a flowchart illustrating a process for training the MLP of FIG. 2 according to various embodiments.

[0015] [Figure 4] FIG. 4 is a flowchart illustrating a process for testing the MLP of FIG. 2 according to various embodiments.

[0016] [Figure 5] FIG. 5 is a flowchart illustrating the calculation of a loss function used in the process of FIG. 3 according to some embodiments.

[0017] [Figure 6] FIG. 6 is a flowchart illustrating the calculation of an example regularization term for the loss function used in the process of FIG. 3 according to some embodiments.

[0018] [Figure 7] FIG. 7 is a flowchart illustrating the calculation of another example edge regularization term of the loss function used in the process of FIG. 3 , according to some examples.

[0019] [Figure 8] FIG. 8 is a block diagram illustrating a computing device according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] Detailed Description The present disclosure and aspects thereof may be embodied in various forms, including hardware, devices or circuits controlled by computer-implemented methods, computer program products, computer systems or networks, user interfaces and application programming interfaces, as well as hardware-implemented methods, signal processing circuits, memory arrays, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. The foregoing is intended only to provide a conceptual overview of various aspects of the present disclosure and is not intended to limit the scope of the present disclosure in any way.

[0021] In the following description, numerous details are set forth, such as the configuration, timing, and operation of optical devices, to provide an understanding of one or more aspects of the present disclosure. Those skilled in the art will readily appreciate that these specific details are merely examples and are not intended to limit the scope of the present application.

[0022] Additionally, while this disclosure primarily focuses on examples in which various circuits are used in digital projection systems, it will be understood that this is but one example implementation. Furthermore, it will be understood that the systems and methods of this disclosure can be used in any device that requires projecting light, such as cinema, home and other commercial projection systems, heads-up displays, virtual reality displays, etc.

[0023] A neural network (NN) is a typical nonlinear trainable circuit comprising multiple processing elements (PEs), also called "neurons," "artificial neurons," or "NN nodes." In some implementations, a neural network is an application-specific circuit, with different PEs implemented as respective configurable subcircuits connected by physical links to form a corresponding physical network. In some other implementations, a neural network may be emulated by a computer, in which one or more electronic processors are programmed to perform signal processing similar to the corresponding application-specific circuit.

[0024] Each PE in a neural network typically has connections to one or more other PEs. The multiple connections between PEs (physical or computer-emulated) define the neural network's topology. In some topologies, a collection of PEs may comprise multiple layers. Different layers may have different types of PEs configured to perform different types of transformations on the input. A signal may progress from the first PE layer (typically referred to as the input layer) to the last PE layer (typically referred to as the output layer). In some topologies, a neural network may have one or more intermediate PE layers (typically referred to as hidden layers) located between the input and output PE layers. An example PE may scale, sum, and bias the input signal and use an activation function to generate an output signal that is a static nonlinear function of the biased sum. The resulting PE output may become one of the neural network's outputs or be sent to one or more other PEs via corresponding connections. The respective weights and / or biases applied by individual PEs may be changed (e.g., optimized) during a training (learning) mode of operation and are typically fixed (i.e., constant) during a testing (working) mode of operation. Various embodiments disclosed herein may employ or rely on one or more neural networks.

[0025] Neural Radiance Field Neural radiance fields (NeRFs), for example, implicitly represent 3D scenes using a neural network that takes 3D positions and gaze directions as inputs and produces corresponding predicted color textures and volumetric densities as outputs. The corresponding neural network can be trained using a set of 2D images with known camera poses and associated intrinsic information. After training, the neural network can render the color textures and volumetric densities of any view of the 3D scene by querying the 3D positions and gaze directions corresponding to various pixels in the view.

[0026] In various examples, NeRF is configured to implicitly represent a scene using continuous 3D scene density σ and color c = (r, g, b) as a function of a continuous five-dimensional (5D) input vector of spatial coordinates x = (x, y, z) and viewing direction d = (θ, φ), where x, y, and z are Cartesian coordinates and θ and φ are angles used in a spherical coordinate system. The continuous function that transforms the five-dimensional input (x, d) to a four-dimensional (4D) output (σ, r, g, b) is approximated using a neural network such as a multilayer perceptron (MLP).

[0027] In some NeRF configurations, density σ is a function of only x, while radiance c is a function of both position x and viewing direction d. The color of a pixel in a 2D rendering of a scene is calculated via volumetric rendering. Given a set of images of a 3D scene with known camera parameters, the PE parameters of a neural network can be optimized via gradient descent by minimizing the photometric differences between the rendered images and the ground truth images. A brief description of examples corresponding to MLP, positional coding, and volumetric rendering is provided in the remainder of this section.

[0028] An MLP is a fully connected feedforward neural network with an input layer, one or more hidden layers, and an output layer. The attribute "fully connected" means that there is a respective weighted connection between each PE in the previous layer and each PE in the adjacent subsequent layer. The entries of the corresponding weight matrix W are w ab , which contains the weight for the connection between the a-th PE in the subsequent layer and the b-th PE in the previous layer. For a layer with multiple PEs, its output JPEG2025531498000002.jpg512 is the input value JPEG2025531498000003.jpg511 can be expressed as a function f of the weighted sum of o is the number of PEs in the subsequent layer, and n i is the number of PEs in the previous layer. The function f is the activation function mentioned above. Mathematically,

number

number

[0029] Because deep neural networks may be inherently biased toward preferentially learning lower frequency functions, NeRF may be further configured to map the input to a higher dimensional space using a series of trigonometric functions γ, passing higher dimensional inputs to the neural network to better fit output data with high frequency components. In various examples, the series γ is defined as follows:

number

[0030] An example of NeRF represents a 3D scene as volumetric density and directional radiance at any point in 3D space, and uses volumetric rendering with numerical integration to render the color of any ray passing through the scene: r(t)=o+td r , t n <t<t f is emitted from the camera through any pixel and is divided into the near boundary and the far boundary (t n and t f ) where o is the origin and d r is the unit vector in the direction of the ray, and t n and t f Select K random orthogonal points {t k For |k=1,...,K}, the approximated expected color can be calculated using the following formula:

number

number

[0031] In addition to color and texture, some applications and / or users require and / or expect semantic segmentation of 3D scenes, which the above-mentioned NeRF does not provide. Beneficially, various embodiments disclosed herein address at least this problem in the state of the art by using an initial, pre-trained neural network, such as NeRF, to provide a neural segmentation field implemented as an additional, smaller neural network trained and configured to implicitly represent the semantic segmentation of the same 3D scene. According to one exemplary embodiment, training the smaller neural network exploits the fact that the semantics, color, texture, and geometry of a 3D scene are typically significantly correlated. Thus, given the 3D scene representation of the initial, pre-trained neural network and a set of 2D segmentations with known camera poses and intrinsic parameters, a neural segmentation field can be learned in a relatively fast and efficient manner, and the corresponding small neural network can then be used to render a 2D segmentation map of the 3D scene corresponding to an arbitrarily selected viewpoint. At least some embodiments of the training process implement enhancements, e.g., in the form of one or more regularization terms, that can improve the segmentation quality of at least some 3D scenes. In some instances, certain exemplary embodiments of the disclosed neural segmentation field beneficially achieve relatively high segmentation quality (e.g., mIoU>0.9) for a relatively large number (e.g., >25) of semantic classes with a relatively small model size (e.g., <0.4MB). Various embodiments of the neural segmentation field can be applied, for example, to object editing (e.g., color manipulation or enhancement) and / or 3D scene encoding and compression.

[0032] As used herein, the acronym "mIoU" stands for mean intersection-over-union. The mIoU metric is typically calculated from ground truth and predicted masks and is conventionally used to evaluate the performance of various object detection and semantic segmentation models. Qualitatively, this metric can be understood as measuring the overlap between the ground truth and predicted masks. Thus, the higher the mIoU value, the better the model's performance. An mIoU value greater than 0.9 typically indicates excellent performance.

[0033] Neural Segmentation Fields 1A-1B are block diagrams illustrating exemplary process flows (100, 101), respectively, for training and testing a neural segmentation field (110) according to various embodiments. The training stage (108) of the training process flow (100) generates a neural segmentation field (110), which is an implicit segmentation representation that represents the segmentation of the 3D scene as a neural network, based on a 3D scene representation (102), a set of 2D segmentation maps (104), and corresponding camera parameters (106). In some examples, the 3D scene representation (102) is a NeRF. In some other examples, the 3D scene representation (102) is another suitable 3D scene representation that has been pre-trained to represent 3D scenes. For example, such a 3D scene representation may be a non-NN-based solution, such as spherical harmonics or another suitable implementation thereof. In various examples, the semantic segmentation and instance segmentation can be represented by a neural segmentation field (110). A test stage (114) of the test process flow (101) generates a 2D segmentation map (116) corresponding to a test (e.g., new or arbitrary) viewpoint using the 3D scene representation (102), the neural segmentation field (110) generated in the training process flow (100), and camera parameters (112) for the test viewpoint. Further details about specific, non-limiting examples of the training stage (108) and test stage (114) are described below, for example, with reference to Figures 3 and 4.

[0034] FIG. 2 is a block diagram illustrating an MLP (200) that can be used to implement a neural segmentation field, according to one embodiment. The MLP (200) has four layers (2101-2104). The first layer (2101) is an input layer. The next two layers (2102, 2103) are hidden layers. The fourth layer (2104) is an output layer. In general, the MLP (200) can have N hidden layers, where N is a positive integer. Thus, the example of the MLP (200) shown in FIG. 2 corresponds to N=2. In some examples, the number N is in the range of 1 to 4.

[0035] Each layer (2101-2104) has M PEs (202), where M is a positive integer. In some embodiments, the number M is in the range of 16 to 256. Each PE (202) in the second layer (2102) 21 ~202 2M ) is the input layer (2101) of each PE (202 11 ~202 1M ) in the third layer (2103) are directly connected to receive their respective inputs. 31 ~202 3M ) is the second layer (2102) of each PE (202 21 ~202 2M ) are directly connected to receive their respective inputs. 41 ~202 4M ) is the third layer (2103) of each PE (202 31 ~202 3M ) to receive their respective inputs. Similar PE / layer connections are implemented for further embodiments corresponding to other values ​​of N. In some examples, each PE (202) operates using a ReLU activation function (see equation (2)). In other examples, other suitable activation functions may be used.

[0036] In one specific example, the input to the MLP (200) includes position-encoded 3D coordinates (γ(x), γ(y), γ(z)) (see also Equation (3)) and 4D inputs including radiance c = (r, g, b) and density σ. The output of the MLP (200) is a set of multiple semantic logits: JPEG2025531498000009.jpg511, where n s is the number of segmentation classes.

[0037] FIG. 3 is a flowchart (300) illustrating a process for training an MLP (NN1, 200) according to various embodiments. Such training is performed using camera parameters (302) for a set of training views. Each of the training views is associated (342) with a corresponding ground truth label map (340). The ground truth label map (340) serves as one of the inputs to a loss function (330). Iterative tuning (328) of the PE parameters used in the MLP (200) is performed to optimize (e.g., approximately minimize) the loss function (330) over the set of training views. Once an optimization criterion is met, the training process ends and the corresponding "optimal" PE parameters are fixed and then used in a test (work) mode of operation. Further details about the test (work) mode of operation are described below with reference to FIG. 4.

[0038] For a view selected from the set of training views, the camera parameters (302) are used to obtain 5D inputs (304, 306) for a pre-trained neural network (NN2, 310). In some implementations, the neural network (310) implements a NeRF, e.g., as described above. The 5D inputs (304, 306) include corresponding 3D spatial coordinates (x, y, z) (306) and a 2D gaze direction (θ, φ) (304). In response to the 5D inputs (304, 306), the pre-trained neural network (310) generates a 4D output (r, g, b, σ) (314). A positional encoding (312) is applied to the 3D spatial coordinates (x, y, z) (306) to generate a corresponding sequence γ(p) (318) (see, e.g., Equation (3)). The 4D output (r, g, b, σ) (314) of the neural network (310), along with the sequence γ(p) (318), provides the input to the MLP (200) during training. In response to this input, the MLP (200) generates multiple semantic logits Generates JPEG2025531498000010.jpg511(320).

[0039] Volumetric rendering (322) can be implemented, for example, in a manner consistent with equation (4), to derive from the logits s (320) a corresponding 2D semantic segmentation map of width W and height H: JPEG2025531498000011.jpg524(324) is used to calculate the map (324). An example of a mathematical equation for the calculation of the map (324) is as follows:

number

[0040] FIG. 4 is a flowchart (400) illustrating a process for testing an MLP (NN1, 200) according to various embodiments. As already mentioned above, the term "testing" refers to a working mode of operation capable of generating semantic segmentation maps corresponding to any view of a 3D scene. Prior to entering the working mode of operation, the MLP (200) is trained as described above with reference to FIG. 3. In the testing process (400), the PE parameters of the MLP (200) are constant, e.g., fixed to optimal values ​​determined using the training process (300). The pre-trained neural network (310) remains in the same configuration in the testing process (400) as it was in the training process (300).

[0041] For a selected (e.g., any) new view, the corresponding camera parameters (402) are used to obtain corresponding 5D inputs (304, 306) for the neural network (310). In response to the 5D inputs (304, 306), the neural network (310) generates a 4D output (σ, r, g, b) (314). A positional encoding (312) is applied to the 3D spatial coordinates (x, y, z) (306) to generate a corresponding sequence γ(p) (318). The 4D output (σ, r, g, b) (314) of the neural network (310), along with the sequence γ(p) (318), provide input for the trained MLP (200). In response to this input, the trained MLP (200) generates a plurality of semantic logits ( JPEG2025531498000013.jpg511 (320). Further processing of the output of the MLP involves applying a softmax function (426) to the semantic logits (320) to obtain the probability v of a ray / pixel belonging to class c. For example, for a pixel (h, w), the probability of belonging to class c can be calculated as follows:

number

[0042] Example of a loss function In one specific example, the loss function (330) is defined as the cross-entropy between the predicted semantic segmentation map (324) and the ground truth segmentation map (340), for example, as follows:

number

[0043] In another example, the loss function (330) is defined as the weighted cross-entropy between the predicted semantic segmentation map (324) and the ground truth segmentation map (340), for example:

number

number

[0044] FIG. 5 is a flowchart (500) illustrating the calculation of weighted cross-entropy for the loss function (330) according to some embodiments. Inputs to the calculation (500) include the predicted semantic segmentation map (324), the ground truth segmentation map (340), and the RGB image (332). Edge detection (504) is performed on the RGB image (332) to detect object edges therein. A weighting function w in Equation (11) is then calculated (502), giving object edge pixels a higher weight than object interior (body) pixels. In some examples, the calculation blocks (502, 504) can be implemented using a deep learning-based edge detector. The loss function (330) is then calculated according to Equations (9)-(10).

[0045] In some other implementations, the loss function (330) includes one or more weighted regularization terms added to the cross-entropy of Equation (8) or the weighted cross-entropy of Equation (10). Two illustrative examples of such regularization terms are described below with reference to FIGS. 6-7. In various other embodiments, other suitable regularization terms can be used in addition to or in place of these example regularization terms. Equation (12) provides an example of a loss function (330) employing such regularization terms.

number

[0046] An example of a regularization term implements pixel-wise edge regularization: the corresponding single-term regularization term R(p) at pixel p is defined as follows:

number

number

[0047] 6 is a flowchart (600) illustrating the calculation of a pixel-wise regularization term (610) for a loss function (330) according to some embodiments. Inputs to the calculation (600) include a predicted semantic segmentation map (324) and an RGB image (332). An edge detection (504) is performed on the RGB image (332) to detect object edges therein. An edge map (604) is generated based on the object edges detected in block (504). A segmentation edge map (602) is generated based on the predicted semantic segmentation map (324). The pixel-wise edge regularization term (610) is then calculated based on the edge maps (602, 604) according to Equation (13):

[0048] Another example of a regularization term implements pairwise edge regularization: for pixels p and q, the corresponding pairwise regularization term R(p,q) is defined as:

number

[0049] In one specific example, D rgb is a weighted graph G, which is a 4-neighborhood graph with pixels as nodes and the edge weights as the size of the edges in the RGB image. rgb is the length of the shortest path P between the two pixels p and q above. The corresponding formula is:

number

[0050] 7 is a flowchart (700) illustrating the computation of a pairwise edge regularization term (710) for a loss function (330) in some examples. Inputs to the computation (700) include a predicted semantic segmentation map (324) and an RGB image (332). Edge detection (504) is performed on the RGB image (332) to detect object edges therein. An edge map (604) is generated based on the object edges detected in block (504). A pairwise distance map (702) is generated based on the edge map (604) according to equation (18). Based on the predicted semantic segmentation map (324), D s A pairwise distance map (704) is generated using the above definition of (p, q). A pairwise edge regularization term (710) is then calculated based on the distance maps (702, 704) according to equation (16). In some implementations, the pairwise edge regularization term (710) may be calculated over a local region around a pixel of interest. The local region may have a size of, for example, 8x8 or 16x16 pixels.

[0051] Hardware Example 8 is a block diagram illustrating a computing device (800) according to one embodiment. The device (800) can be used, for example, to implement the process flows (100, 101). The device (800) includes an input / output (I / O) device (810), an image processing engine (IPE, 820), and memory (830). The input / output device (810) can be used to enable the device (800) to receive at least parameters of the 3D representation (102) and output at least parameters of the neural segmentation field (110) and the segmentation map (116). The input / output device (810) can also be used to connect the device (800) to a display.

[0052] The memory (830) may include a buffer for receiving image data corresponding to the 3D scene to be rendered. The image data may be in the form of, for example, one or more image files. Once the image data is received, the memory (830) may provide portions of the data to the IPE (820) for processing. The IPE (820) includes a processor (822) and a memory (824). The memory (824) may store program code therein. When executed by the processor (822), the program code enables the IPE (820) to perform image processing. Such image processing may include, but is not limited to, image processing according to some or all of the flowcharts (100, 101, 300, 400, 500, 600, 700) described above. The program code may include, among other things, program code used to emulate various neural networks, such as NN1 (200) and NN2 (310) described above. Once the IPE (820) has generated the various maps described above by executing the corresponding portions of code, the IPE (820) may perform its rendering process to provide a corresponding viewable image for viewing on a display, which may take the form of, for example, a suitable image file output via the I / O device (810).

[0053] According to the above exemplary embodiments, for example, in the Overview section and / or with reference to any one of Figures 1-8 or any combination of some or all of Figures 1-8, there is provided an image processing method comprising: training, by a processor, a first neural network to render a segmentation map corresponding to an arbitrarily selected view of a 3D scene, the training comprising: calculating, by the processor, a color texture and volume density corresponding to the selected training view of the 3D scene, the calculating step being performed using a second neural network pre-trained to represent the 3D scene; generating, by the processor, a predicted segmentation map corresponding to the selected training view of the 3D scene, the generating step being performed using the first neural network based on the color texture and the volume density; and adjusting, by the processor, configuration parameters of processing elements of the first neural network based on a loss function configured to receive, as a first input, a ground truth segmentation map corresponding to the selected training view, and further configured to receive, as a second input, the predicted segmentation map.

[0054] In some embodiments of the above method, the first neural network is a multi-layer perceptron and the second neural network is another multi-layer perceptron implementing a neural radiance field.

[0055] In some embodiments of any of the above methods, the first neural network is configured to output semantic logits corresponding to the selected training view of the 3D scene in response to the color texture and the volumetric density.

[0056] In some embodiments of any of the above methods, training further comprises calculating a cross-entropy between the predicted segmentation map and the ground truth segmentation map, and using the cross-entropy to obtain a loss function value.

[0057] In some embodiments of any of the above methods, the loss function is configured to receive, as a third input, a color image of the 3D scene corresponding to the selected training view.

[0058] In some embodiments of any of the above methods, the training further comprises detecting object edges in the color image; assigning different weights to object edge pixels and object interior pixels; calculating a cross-entropy between the predicted segmentation map and the ground truth segmentation map using the different weights; and using the cross-entropy to obtain a loss function value.

[0059] In some embodiments of any of the above methods, the loss function includes one or more regularization terms that favor alignment of corresponding edges in the predicted segmentation map and the ground truth segmentation map.

[0060] In some embodiments of any of the above methods, the generating step is performed in response to a position encoding of 3D spatial coordinates corresponding to the selected training view.

[0061] In some embodiments of any of the above methods, the image processing method further includes creating, by the processor, a segmentation map corresponding to an arbitrarily selected view of the 3D scene, the creating being performed using the first neural network after the training.

[0062] In some embodiments of any of the above methods, the image processing method further includes applying, by the processor, a softmax function to the segmentation map corresponding to the arbitrarily selected view of the 3D scene to generate a probability map, and converting, by the processor, the probability map into a semantic label map representing the arbitrarily selected view of the 3D scene.

[0063] According to other exemplary embodiments described above, for example in the Overview section and / or with reference to any one or a combination of some or all of Figures 1-8, there is provided a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the processor to perform operations including any of the methods described above.

[0064] According to yet another exemplary embodiment described above, for example in the Overview section and / or with reference to any one or a combination of some or all of Figures 1-8, there is provided an image processing apparatus comprising at least one processor and at least one memory including program code, wherein said at least one memory and said program code are configured to cause said apparatus to train at least a first neural network to render a segmentation map corresponding to an arbitrarily selected view of a 3D scene, and to train said first neural network, said apparatus is configured to: an image processing device for computing a color texture and volume density corresponding to the selected training view of the 3D scene, the computing being performed using a second neural network pre-trained to represent the 3D scene; generating a predicted segmentation map corresponding to the selected training view of the 3D scene, the generating being performed using the first neural network based on the color texture and the volume density; and adjusting configuration parameters of processing elements of the first neural network based on a loss function, the loss function receiving as a first input the ground truth segmentation map corresponding to the selected training view and further receiving the predicted segmentation map as a second input.

[0065] In some embodiments of the above apparatus, the first neural network is a multi-layer perceptron and the second neural network is another multi-layer perceptron that implements a neural radiance field.

[0066] In some embodiments of any of the above devices, the first neural network is configured to output semantic logits corresponding to the selected training view of the 3D scene in response to the color texture and the volumetric density.

[0067] In some embodiments of any of the above devices, to train the first neural network, the device is further configured to calculate a cross-entropy between the predicted segmentation map and the ground truth segmentation map, and to obtain a loss function value using the cross-entropy.

[0068] In some embodiments of any of the above apparatus, the loss function is configured to receive, as a third input, a color image of the 3D scene corresponding to the selected training view.

[0069] In some embodiments of any of the above devices, to train the first neural network, the device is further configured to detect object edges in the color image, assign different weights to object edge pixels and object interior pixels, calculate a cross-entropy between the predicted segmentation map and the ground truth segmentation map using the different weights, and obtain a loss function value using the cross-entropy.

[0070] In some embodiments of any of the above apparatus, the loss function includes one or more regularization terms that favor matching of corresponding edges in the predicted segmentation map and the ground truth segmentation map.

[0071] In some embodiments of any of the above devices, the at least one memory and the program code are further configured, by the at least one processor, to cause the device to create a segmentation map corresponding to an arbitrarily selected view of the 3D scene, the creation being performed using the first neural network after the training.

[0072] In some embodiments of any of the above devices, the at least one memory and the program code are further configured, by the at least one processor, to cause the device to apply a softmax function to the segmentation map corresponding to the arbitrarily selected view of the 3D scene to generate a probability map, and to convert the probability map into a semantic label map representing the arbitrarily selected view of the 3D scene.

[0073] According to yet another exemplary embodiment described above, for example in the Overview section and / or with reference to any one or a combination of some or all of Figures 1-8, there is provided an image processing method comprising: testing, by a processor, a first neural network to render a segmentation map corresponding to an arbitrarily selected view of a 3D scene, wherein the testing comprises: calculating, by the processor, a color texture and volume density corresponding to the selected view of the 3D scene, the calculating being performed using a 3D representation pre-trained to represent the 3D scene; and generating, by the processor, a segmentation map corresponding to the selected view of the 3D scene based on the color texture and the volume density using the first neural network, wherein the first neural network has been trained by the processor using the pre-trained 3D representation.

[0074] According to yet another exemplary embodiment described above, for example in the Overview section and / or with reference to any one of Figures 1-8 or a combination of some or all of Figures 1-8, there is provided an image processing device comprising at least one processor and at least one memory including program code, wherein the at least one memory and the program code are configured by the at least one processor to cause the device to test a first neural network to render a segmentation map corresponding to an arbitrarily selected view of a 3D scene, wherein to test the first neural network, the device is configured to: calculate a color texture and volume density corresponding to the selected view of the 3D scene, wherein the calculation is performed using a 3D representation pre-trained to represent the 3D scene; and generate a segmentation map corresponding to the selected view of the 3D scene, wherein the generation is performed using the first neural network based on the color texture and the volume density, wherein the first neural network has been trained using the pre-trained 3D representation.

[0075] With respect to processes, systems, methods, heuristics, etc. described herein, steps of such processes, etc. have been described as occurring according to a certain ordered sequence, but it will be understood that such processes can be practiced with the described steps performed in an order other than the order described herein. Furthermore, it will be understood that certain steps can be performed simultaneously, other steps can be added, or certain steps described herein can be omitted. In other words, the process descriptions herein are provided for the purpose of illustrating particular embodiments and should not be construed as limiting the scope of the claims in any way.

[0076] Accordingly, it should be understood that the above description is intended to be illustrative, and not limiting. Many embodiments and applications other than the examples provided will become apparent from reading the above description. The scope of the claims should be determined not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technology discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In other words, it should be understood that modifications and variations are possible in this application.

[0077] All terms used in the claims are intended to be given their broadest reasonable interpretation and ordinary meaning as understood by one of ordinary skill in the art described herein, unless expressly stated to the contrary in the specification. In particular, the use of singular articles such as "a," "the," and "said" means that there are one or more of the described element, unless expressly limited to the contrary in the claim.

[0078] The Abstract of the Disclosure is provided to allow the reader to quickly grasp the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Furthermore, in the foregoing Detailed Description, it can be seen that various features are grouped together in various embodiments for clarity of disclosure. This method of disclosure does not reflect an intention that the claimed embodiments include more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive subject matter lies in fewer than all features of a single disclosed embodiment. Accordingly, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as separately claimed subject matter.

[0079] While this disclosure includes reference to exemplary embodiments, this specification is not intended to be construed in a limiting sense. Various modifications of the described embodiments, as well as other embodiments within the scope of the present disclosure that are apparent to those skilled in the art to which the present disclosure pertains, are deemed to be within the principles and scope of the present disclosure, as expressed, for example, in the following claims.

[0080] Some embodiments may be implemented as circuit-based processes, including possible implementation on a single integrated circuit.

[0081] Some embodiments may be embodied in the form of methods and apparatuses for practicing those methods. Some embodiments may also be embodied in the form of program code recorded on a tangible medium, such as a magnetic recording medium, an optical recording medium, a solid-state memory, a floppy disk, a CD-ROM, a hard drive, or any other non-transitory machine-readable storage medium, which, when loaded and executed by a machine, such as a computer, makes the machine an apparatus for practicing various embodiments described herein. Some embodiments may also be embodied in the form of program code stored on a non-transitory machine-readable storage medium, including, for example, being loaded into and / or executed by a machine, which, when loaded and executed by a machine, such as a computer or processor, makes the machine an apparatus for practicing various embodiments described herein. When implemented on a general-purpose processor, the program code segments combine with the processor to provide a unique device that operates analogously to specific logic circuits.

[0082] Unless otherwise specified, each numerical value and range should be construed as approximate, as if the word "about" or "approximately" were before the numerical value or range.

[0083] Use of figure numbers and / or figure reference labels in the claims is intended to identify one or more possible embodiments of the claimed subject matter to facilitate claim interpretation, and such use should not be construed as necessarily limiting the scope of those claims to the embodiments shown in the corresponding figures.

[0084] Although the elements recited in the following method claims, if any, are presented in a particular order with corresponding labeling, these elements are not necessarily intended to be limited to being performed in that particular order, unless the claims imply a particular order for performing some or all of these elements.

[0085] References herein to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of the present disclosure. Appearances of the phrase "in one embodiment" in various places throughout this specification do not necessarily all refer to the same embodiment, nor are separate or additional embodiments necessarily mutually exclusive of other embodiments. The same applies to the term "implementation."

[0086] Unless otherwise specified herein, the use of the ordinal adjectives "first," "second," "third," etc. to refer to one object of a plurality of like objects merely indicates that different instances of such like objects are being referenced and does not imply that the like objects so referenced must be in a corresponding order or sequence, whether in time, space, ranking, or otherwise.

[0087] Unless otherwise defined herein, in addition to its plain meaning, the conjunction "if" may also be interpreted to mean "when," "upon," "in response to determining," or "in response to detecting," and the interpretation may depend on the particular context in which it is used. For example, the phrase "if it is determined" or "if [a stated condition] is detected" may be interpreted to mean "upon determining" or "in response to determining," or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]."

[0088] Also, as used herein, the terms "couple," "coupling," "coupled," "connect," "connected," and "connected" refer to any implementation known in the art or developed in the future in which energy is transferred between two or more elements, and the interposition of one or more additional elements is intended, but not required. Conversely, terms such as "directly coupled," "directly connected," and the like imply the absence of such additional elements.

[0089] The functions of the various elements illustrated in the figures, including functional blocks labeled "processor" and / or "controller," may be provided not only through the use of dedicated hardware, but also through the use of hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, a single shared processor, or multiple individual processors, some of which may be shared. Furthermore, the explicit use of the terms "processor" or "controller" should not be construed to refer solely to hardware capable of executing software, but may implicitly include, without limitation, digital signal processor (DSP) hardware, network processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), read-only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage. Other hardware, conventional and / or custom, may also be included. Similarly, the switches illustrated in the figures are conceptual only. These functions may be performed through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or manually, with the particular technique being selectable by the implementer as more particularly understood from the context.

[0090] As used herein, the term “circuit” may refer to one or more or all of the following: (a) a hardware-only circuit implementation (such as an implementation with only analog and / or digital circuitry); (b) a combination of hardware circuitry and software, such as (where applicable): (i) a combination of analog and / or digital hardware circuitry with software / firmware, (ii) a hardware processor with software (including a digital signal processor), software, and any portion of memory that cooperates to cause a device such as a cell phone or server to perform various functions, and (c) a hardware circuit and / or processor, such as a microprocessor or portion of a microprocessor, that requires software (e.g., firmware) to operate, but that the software may not be present when not necessary for operation. This definition of circuit applies to all uses of the term in this application, including the claims. As a further example, as used herein, the term circuit also covers an implementation solely of a hardware circuit or processor (or processors) or a portion of a hardware circuit or processor with its (or their) accompanying software and / or firmware. The term circuitry also covers, by way of example and where applicable to particular claim elements, baseband or processor integrated circuits for mobile devices, or similar integrated circuits in servers, cellular network devices, or other computing or network devices.

[0091] Those skilled in the art will appreciate that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the present disclosure. Similarly, any flowcharts, flow diagrams, state transition diagrams, pseudocode, etc. will be understood to represent various processes that may be substantially represented in a computer-readable medium and executed by a computer or processor, whether or not such a computer or processor is explicitly shown.

[0092] The "BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS" herein is intended to introduce some exemplary embodiments; additional embodiments are described in the "DETAILED DESCRIPTION" and / or with reference to one or more drawings. The "BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS" is not intended to identify essential elements or features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

Claims

1. 1. A method of image processing comprising training, by a processor, a first neural network (NN) to render a segmentation map corresponding to an arbitrarily selected view of a three-dimensional (3D) scene, said training comprising: calculating, by the processor, color textures and volumetric densities corresponding to selected training views of the 3D scene, the calculation being performed using a 3D representation pre-trained to represent the 3D scene; generating, by the processor, a predicted segmentation map corresponding to the selected training view of the 3D scene, the predicted segmentation map being performed using the first neural network based on the color texture and the volumetric density; adjusting, by the processor, configuration parameters of NN elements of the first neural network based on a loss function configured to receive as a first input a ground truth segmentation map corresponding to the selected training view and further configured to receive as a second input the predicted segmentation map; An image processing method comprising:

2. the first neural network is a multi-layer perceptron; the 3D representation is a second neural network implementing a neural radiance field; The image processing method according to claim 1 .

3. the first neural network is configured to output semantic logits corresponding to the selected training views of the 3D scene in response to the color texture and the volume density.

3. The image processing method according to claim 1.

4. The training includes: calculating a cross-entropy between the predicted segmentation map and the ground truth segmentation map; obtaining a loss function value using the cross-entropy; further comprising:

4. The image processing method according to claim 1.

5. the loss function is configured to receive as a third input a color image of the 3D scene corresponding to the selected training view.

5. The image processing method according to claim 1.

6. The training includes: detecting object edges in the color image; assigning different weights to object edge pixels and object interior pixels; calculating a cross-entropy between the predicted segmentation map and the ground truth segmentation map using the different weights; using the cross-entropy to obtain a loss function value; further comprising: The image processing method according to claim 5 .

7. the loss function includes one or more regularization terms that favor matching of corresponding edges in the predicted segmentation map and the ground truth segmentation map.

7. The image processing method according to claim 1.

8. the generating step is performed in response to position encoding of 3D spatial coordinates corresponding to the selected training view.

8. The image processing method according to claim 1.

9. generating, by the processor, a segmentation map corresponding to an arbitrarily selected view of the 3D scene, the segmentation map being performed using the first neural network after the training; applying, by the processor, a softmax function to the segmentation map corresponding to the arbitrarily selected view of the 3D scene to generate a probability map; converting, by the processor, the probability map into a semantic label map representing the arbitrarily selected view of the 3D scene; The image processing method according to claim 1 , further comprising:

10. A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform operations including the method of any of claims 1 to 9.

11. at least one processor; at least one memory containing program code; An image processing device comprising: the at least one memory and the program code are configured to cause the image processing device to train, by the at least one processor, at least a first neural network (NN) for rendering a segmentation map corresponding to an arbitrarily selected view of a three-dimensional (3D) scene; To train the first neural network, the image processing device: calculating color textures and volumetric densities corresponding to selected training views of the 3D scene, said calculations being performed using a 3D representation pre-trained to represent the 3D scene; generating a predicted segmentation map corresponding to the selected training view of the 3D scene, said generation being performed using the first neural network based on the color texture and the volumetric density; adjusting configuration parameters of NN nodes of the first neural network based on a loss function, the loss function receiving, as a first input, a ground truth segmentation map corresponding to the selected training views and further receiving, as a second input, the predicted segmentation map; The image processing device is configured as follows.

12. the first neural network is a multi-layer perceptron; the 3D representation is a second neural network implementing a neural radiance field; The image processing device according to claim 11 .

13. the first neural network is configured to output semantic logits corresponding to the selected training views of the 3D scene in response to the color texture and the volume density.

13. The image processing device according to claim 11 or 12.

14. To train the first neural network, the image processing device: calculating a cross-entropy between the predicted segmentation map and the ground truth segmentation map; The cross-entropy is used to obtain the loss function value. further configured as follows:

14. The image processing device according to claim 11.

15. the loss function is configured to receive, as a third input, a color image of the 3D scene corresponding to the selected training view.

15. The image processing device according to claim 11.

16. To train the first neural network, the apparatus: Detecting object edges in the color image; assigning different weights to object edge pixels and object interior pixels; calculating a cross-entropy between the predicted segmentation map and the ground truth segmentation map using the different weights; The cross-entropy is used to obtain the loss function value. further configured as follows: The image processing device according to claim 15.

17. the loss function includes one or more regularization terms that favor matching of corresponding edges in the predicted segmentation map and the ground truth segmentation map.

17. The image processing device according to claim 11.

18. The at least one memory and the program code are transmitted by the at least one processor to the device: generating a segmentation map corresponding to an arbitrarily selected view of the 3D scene, said generation being performed using the first neural network after said training; applying a softmax function to the segmentation map corresponding to the arbitrarily selected view of the 3D scene to generate a probability map; converting the probability map into a semantic label map representing the arbitrarily selected view of the 3D scene; further configured as follows:

18. The image processing device according to claim 11.

19. 1. A method of image processing comprising testing, by a processor, a first neural network to render a segmentation map corresponding to an arbitrarily selected view of a three-dimensional (3D) scene, said testing comprising: calculating, by the processor, a color texture and volume density corresponding to a selected view of the 3D scene, the calculation being performed using a 3D representation pre-trained to represent the 3D scene; generating, by the processor, a segmentation map corresponding to the selected view of the 3D scene, the segmentation map being performed using the first neural network based on the color texture and the volumetric density; Including, the first neural network has been trained by the processor using the pre-trained 3D representation; Image processing methods.

20. at least one processor; at least one memory containing program code; An image processing device comprising: the at least one memory and the program code are configured to cause the apparatus to test, by the at least one processor, at least a first neural network (NN) for rendering a segmentation map corresponding to an arbitrarily selected view of a three-dimensional (3D) scene; To test the first neural network, the apparatus: calculating color textures and volumetric densities corresponding to selected training views of the 3D scene, said calculations being performed using a 3D representation pre-trained to represent the 3D scene; generating a predicted segmentation map corresponding to the selected training view of the 3D scene, the generation being performed using the first neural network based on the color texture and the volume density. It is configured as follows: the first neural network has been trained using the pre-trained 3D representation; Image processing device.

Images