Neural segmentation fields for representing 3D scenes

Abstract

According to one example embodiment, a method for training a neural network to render a segmentation map corresponding to an arbitrarily selected view of a 3D scene includes calculating color texture and volume density corresponding to the selected training view of the 3D scene using a 3D representation pre-trained to represent the 3D scene, generating predicted segmentation maps corresponding to the selected training views of the 3D scene using the neural network based on the color texture and volume density, and adjusting configuration parameters of network nodes of the neural network based on a loss function configured to receive as a first input the ground truth segmentation maps corresponding to the selected training views and further receive as a second input the predicted segmentation maps.

Description

[Technical Field]

[0001] 1. References to related applications This application claims priority under U.S. Provisional Application No. 63 / 410,344, filed on 27 September 2022, which is incorporated herein by reference in its entirety.

[0002] 2. Areas of Disclosure Various exemplary embodiments relate to rendering a three-dimensional (3D) scene into a two-dimensional (2D) space. [Background technology]

[0003] 3.Background The process of linking each pixel in an image to a class label is called semantic segmentation. Labels can be, for example, "person," "flower," or "building." Semantic segmentation can be thought of as image classification at the pixel level. Therefore, in semantic segmentation, every pixel in an image is typically associated with a corresponding class label. Semantic segmentation differs from object detection because, unlike object detection, semantic segmentation determines the contours of objects in an image relatively accurately by working at the pixel level.

[0004] Unlike semantic segmentation, which treats multiple objects within a class as a single entity, instance segmentation identifies individual objects within a class. Instance segmentation is sometimes considered an improved version of semantic segmentation. Various computer vision applications use either semantic segmentation or a combination of both. [Overview of the Initiative]

[0005] Outline of a specific embodiment Various embodiments of methods and apparatus for rendering segmentation maps of 3D scenes using machine learning are disclosed herein. According to one exemplary embodiment, a smaller neural network (NN) is trained to render a segmentation map corresponding to an arbitrarily selected view of a 3D scene. Here, the training is performed using a larger neural network that has been pre-trained to represent the 3D scene with color textures and volume density. In other words, the smaller neural network is configured to work on top of the larger, pre-trained neural network, thereby providing the ability to acquire segmentation information faster and with a relatively low additional burden on computational resources. Also disclosed are several embodiments of entropy-based loss functions and regularization terms therefor. These are configured to facilitate the training process for the smaller neural network, for example, by using them in conjunction with or utilizing the auxiliary information available therefrom in the previous training of the larger neural network.

[0006] According to an exemplary embodiment, an image processing method is provided that includes training, by a processor, a first neural network to render a segmentation map corresponding to an arbitrarily selected view of a 3D scene. The training includes: a step of calculating, by the processor, a color texture and a volume density corresponding to a selected training view of the 3D scene, the step being performed using a 3D representation pre-trained to represent the 3D scene; a step of generating, by the processor, a predicted segmentation map corresponding to the selected training view of the 3D scene, the step being performed using the first neural network based on the color texture and the volume density; and a step of adjusting, by the processor, configuration parameters of NN nodes of the first neural network based on a loss function configured to receive, as a first input, a ground truth segmentation map corresponding to the selected training view and further configured to receive, as a second input, the predicted segmentation map.

[0007] According to another exemplary embodiment, a non-transitory computer-readable medium is provided that stores instructions that, when executed by the processor, cause the processor to perform operations including the above or the following image processing method.

[0008] Further exemplary embodiments provide an image processing apparatus comprising at least one processor and at least one memory containing program code. The at least one memory and the program code are configured to cause the at least one processor to train the apparatus to train at least a first neural network to render a segmentation map corresponding to an arbitrarily selected view of a 3D scene. To train the first neural network, the apparatus is configured as follows: it calculates a color texture and volume density corresponding to a selected training view of the 3D scene, the calculation is performed using a 3D representation pre-trained to represent the 3D scene, and generates a predictive segmentation map corresponding to the selected training view of the 3D scene, the generation is performed using the first neural network based on the color texture and volume density, and the configuration parameters of the NN nodes of the first neural network are adjusted based on a loss function, the loss function receiving a ground truth segmentation map corresponding to the selected training view as a first input, and the predictive segmentation map as a second input.

[0009] According to yet another exemplary embodiment, there is provided an image processing method including a step of testing a first neural network by a processor to render a segmentation map corresponding to an arbitrarily selected view of a 3D scene. The step of testing includes a step of calculating, by the processor, a color texture and a volume density corresponding to the selected view of the 3D scene, the step of calculating being performed using a 3D representation pre-trained to represent the 3D scene, and a step of generating, by the processor, a segmentation map corresponding to the selected view of the 3D scene, the step of generating being performed using the first neural network based on the color texture and the volume density. The first neural network is trained by a processor using the pre-trained 3D representation.

[0010] According to yet another exemplary embodiment, there is provided an image processing apparatus including at least one processor and at least one memory including program code, the at least one memory and the program code being configured to cause the at least one processor to test a first neural network to render a segmentation map corresponding to an arbitrarily selected view of a 3D scene. To test the first neural network, the apparatus is configured to, i.e., calculate a color texture and a volume density corresponding to the selected view of the 3D scene, the calculation being performed using a 3D representation pre-trained to represent the 3D scene, generate a segmentation map corresponding to the selected view of the 3D scene, the generation being performed using the first neural network based on the color texture and the volume density, and the first neural network being trained using the pre-trained 3D representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Other aspects, features, and advantages of various embodiments of the disclosure will become more fully apparent through the following detailed description and accompanying drawings.

[0012] [Figure 1] Figures 1A to 1B are block diagrams illustrating exemplary process flows for training and testing neural segmentation fields according to various embodiments.

[0013] [Figure 2] Figure 2 is a block diagram showing a multilayer perceptron (MLP) that can be used to implement a neural segmentation field according to one embodiment.

[0014] [Figure 3] Figure 3 is a flowchart showing the process of training the MLP shown in Figure 2 according to various embodiments.

[0015] [Figure 4] Figure 4 is a flowchart showing the process of testing the MLP of Figure 2 according to various embodiments.

[0016] [Figure 5] Figure 5 is a flowchart showing the calculation of the loss function used in the process shown in Figure 3, according to several examples.

[0017] [Figure 6] Figure 6 is a flowchart illustrating the calculation of an exemplary regularization term for the loss function used in the process shown in Figure 3, according to several embodiments.

[0018] [Figure 7] Figure 7 is a flowchart showing the calculation of the edge regularization term for other illustrative examples of the loss function used in the process of Figure 3, relating to several examples.

[0019] [Figure 8] Figure 8 is a block diagram showing a computing device according to one embodiment. [Modes for carrying out the invention]

[0020] Detailed explanation The disclosure and its embodiments can be implemented in various forms, including computer-implemented hardware, devices or circuits, computer program products, computer systems and networks, user interfaces and application programming interfaces, and hardware-implemented methods, signal processing circuits, memory arrays, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), etc. The foregoing is intended solely to provide concepts of the various embodiments of the disclosure and does not limit the scope of the disclosure in any way.

[0021] The following description includes many details, such as the configuration, timing, and operation of optical devices, in order to provide an understanding of one or more aspects of the present disclosure. It will be readily apparent to those skilled in the art that these specific details are merely illustrative and are not intended to limit the scope of the application.

[0022] Furthermore, while this disclosure primarily focuses on examples of how various circuits are used in digital projection systems, it will be understood that this represents only one implementation example. Moreover, it will be understood that the systems and methods of this disclosure can be used in any device requiring the projection of light, such as cinema, home, and other commercial projection systems, head-up displays, virtual reality displays, and the like.

[0023] A neural network (NN) is a typical nonlinear trainable circuit comprising multiple processing elements (PEs), also called "neurons," "artificial neurons," or "NN nodes." In some implementations, the neural network is an application-specific circuit, where different PEs are implemented as configurable subcircuits connected by physical links that form the corresponding physical network. In some other implementations, the neural network can be emulated by a computer. In this case, one or more electronic processors are programmed to perform signal processing similar to that of the corresponding application-specific circuit.

[0024] Each PE in a neural network typically has connections to one or more other PEs. Multiple connections between PEs (physical or computer-emulated) define the topology of the neural network. In some topologies, a collection of PEs can constitute multiple layers. Different layers may have different types of PEs configured to perform different kinds of transformations on their inputs. A signal can travel from the first PE layer (typically called the input layer) to the last PE layer (typically called the output layer). In some topologies, a neural network may have one or more intermediate PE layers (typically called hidden layers) located between the input and output PE layers. An example PE might scale, sum, and bias an input signal and, using an activation function, generate an output signal that is a static nonlinear function of the biased sum. The resulting PE output can become one of the neural network's outputs or be transmitted to one or more other PEs via the corresponding connections. The respective weights and / or biases applied by each individual PE can be modified (e.g., optimized) during the training (learning) mode of the operation and are typically fixed (i.e., constant) during the testing (working) mode of the operation. Various embodiments disclosed herein may employ or rely on one or more neural networks.

[0025] Neural Radiance Field Neural Radiance Fields (NeRFs) implicitly represent 3D scenes using a neural network that, for example, takes 3D position and line of sight as input and generates corresponding predicted color textures and volume densities as outputs. The corresponding neural network can be trained using a set of 2D images with known camera poses and relevant unique information. After training, the neural network can render color textures and volume densities for any view of a 3D scene by querying the 3D position and line of sight corresponding to various pixels in the view.

[0026] In various examples, NeRF is configured to implicitly represent a scene using a continuous 3D scene density σ and color c=(r,g,b) as a function of a continuous 5D input vector with spatial coordinates x=(x,y,z) and line of sight direction d=(θ,φ). Here, x,y,z are Cartesian coordinates, and θ,φ are angles used in a spherical coordinate system. The continuous function that transforms the 5D input (x,d) to a 4D output (σ,r,g,b) is approximated using a neural network such as a multilayer perceptron (MLP).

[0027] In some NeRF configurations, density σ is a function of x only, while radiance c is a function of both position x and line of sight d. Pixel color in 2D rendering of a scene is calculated via volumetric rendering. Given a set of images of a 3D scene with known camera parameters, the PE parameters of a neural network can be optimized via gradient descent by minimizing the photometric differences between the rendered image and the ground truth image. A brief explanation of examples corresponding to MLP, position coding, and volumetric rendering is provided in the remainder of this section.

[0028] An MLP is a fully connected feedforward neural network having an input layer, one or more hidden layers, and an output layer. The attribute "fully connected" means that there is a weighted connection between each PE in the previous layer and each PE in the adjacent subsequent layer. The entries in the corresponding weight matrix W are: w ab This includes the weights for the connection between the a-th PE of the subsequent layer and the b-th PE of the previous layer. For layers with multiple PEs, the output JPEG0007853519000001.jpg512 is the input value JPEG0007853519000002.jpg511 It can be expressed as a function f of the weighted sum of n, where n o n is the number of PEs in the subsequent layer. i is the number of PEs in the previous layer. The function f is the activation function mentioned above. Mathematically,

number

number

[0029] Because deep neural networks can inherently be biased towards learning lower-frequency functions preferentially, NeRF can be further configured to map the input to a higher-dimensional space using a sequence of trigonometric functions γ, and to pass higher-dimensional inputs to the neural network in order to better fit output data with high-frequency components. In various examples, the sequence γ is defined as follows:

number

[0030] An example of NeRF represents a 3D scene as a volume density and a directional radiance at an arbitrary point in 3D space, and renders the color of any ray passing through the scene using volume rendering by numerical integration. r(t) = o + td r , t n < t < t f be a ray emitted from the camera through an arbitrary pixel and crossing between the near boundary and the far boundary (t n and t f ). Here, o is the origin, and d r is the unit vector in the direction of the ray. And for the selected K random orthogonal points {t n | k = 1, …, K} between t f and t k , the approximated expected color can be calculated using the following equation.

Number

Number

[0031] In addition to color textures, some applications and / or users require and / or expect semantic segmentation of 3D scenes, which is not provided by NeRF as described above. Fortunately, various embodiments disclosed herein address at least this problem in the present level of the art as follows: namely, a neural segmentation field is provided, implemented as an additional smaller neural network trained and configured to implicitly represent the semantic segmentation of the same 3D scene using an initial pre-trained neural network, such as NeRF. According to one exemplary embodiment, training the smaller neural network takes advantage of the fact that the semantics, color textures, and geometry of a 3D scene are typically significantly correlated. Thus, given a 3D scene representation of the initial pre-trained neural network and a set of 2D segmentations with known camera poses and intrinsic parameters, the neural segmentation field can be learned in a relatively fast and efficient manner, and then a 2D segmentation map of the 3D scene corresponding to an arbitrarily selected viewpoint can be rendered using the corresponding smaller neural network. At least some embodiments of the training process implement enhancements that can improve the segmentation quality of at least some 3D scenes, for example, in the form of one or more regularization terms. In some specific examples, some exemplary embodiments of the disclosed neural segmentation field beneficially achieve relatively high segmentation quality (e.g., mIoU > 0.9) with a relatively small model size (e.g., < 0.4 MB) for a relatively large number (e.g., > 25) semantic classes. Various embodiments of the neural segmentation field can be applied, for example, to object editing (such as color manipulation or enhancement) and / or 3D scene encoding and compression.

[0032] In this specification, the acronym "mIoU" stands for mean intersection-over-union. The mIoU metric is typically calculated from a ground truth mask and a predictive mask and has traditionally been used to evaluate the performance of various object detection and semantic segmentation models. Qualitatively, this metric can be understood as measuring the overlap between the ground truth mask and the predictive mask. Therefore, a higher mIoU value indicates better model performance. An mIoU value greater than 0.9 typically indicates excellent performance.

[0033] Neural Segmentation Field Figures 1A-1B are block diagrams illustrating exemplary process flows (100, 101) for training and testing a neural segmentation field (110) according to various embodiments. The training stage (108) of the training process flow (100) generates a neural segmentation field (110), which is an implicit segmentation representation that represents the segmentation of a 3D scene as a neural network, based on a 3D scene representation (102), a set of 2D segmentation maps (104), and corresponding camera parameters (106). In some examples, the 3D scene representation (102) is NeRF. In some other examples, the 3D scene representation (102) is another suitable 3D scene representation that has been pre-trained to represent the 3D scene. For example, such a 3D scene representation could be a non-NN-based solution, e.g., spherical harmonics or another suitable implementation thereof. In various examples, semantic segmentation and instance segmentation can be represented by a neural segmentation field (110). The test stage (114) of the test process flow (101) generates a 2D segmentation map (116) corresponding to a test (e.g., a new or arbitrary) viewpoint, using the 3D scene representation (102), the neural segmentation field (110) generated in the training process flow (100), and camera parameters (112) for the test viewpoint. Further details of specific non-limiting examples of the training stage (108) and the test stage (114) are described later with reference to, for example, Figures 3 and 4.

[0034] Figure 2 is a block diagram showing an MLP(200) that can be used to implement a neural segmentation field according to one embodiment. The MLP(200) has four layers (2101-2104). The first layer (2101) is the input layer. The next two layers (2102, 2103) are hidden layers. The fourth layer (2104) is the output layer. In general, the MLP(200) can have N hidden layers, where N is a positive integer. Therefore, the specific example of the MLP(200) shown in Figure 2 corresponds to N=2. In some specific examples, the number N is in the range of 1 to 4.

[0035] Each layer (2101-2104) has M PE(202), where M is a positive integer. In some specific examples, the number M is in the range of 16-256. Each PE(202) of the second layer (2102) 21 ~202 2M ) is each PE (202) of the input layer (2101). 11 ~202 1M ) are directly connected to receive their respective inputs. Each PE (202) of the third layer (2103) 31 ~202 3M ) is each PE (202) of the second layer (2102) 21 ~202 2M ) are directly connected to receive their respective inputs. Each PE (202) of the output layer (2104) is connected. 41 ~202 4M ) is each PE (202) of the third layer (2103) 31 ~202 3M They are directly connected to receive their respective inputs from ). Similar PE / layer connections are implemented for further embodiments corresponding to other values ​​of N. In some examples, each PE(202) operates using a ReLU activation function (see equation (2)). In other examples, other suitable activation functions may be used.

[0036] In a specific example, the input to MLP(200) includes a position-encoded 3D coordinate (γ(x),γ(y),γ(z)) (see also equation (3)) and a 4D input containing radiance c=(r,g,b) and density σ. The output of MLP(200) is a set of semantic logits. The file is JPEG0007853519000008.jpg511. Here, n s This represents the number of segmentation classes.

[0037] Figure 3 is a flowchart (300) illustrating the process of training an MLP(NN1, 200) according to various embodiments. Such training is performed using camera parameters (302) of a set of training views. Each training view is associated with a corresponding ground truth label map (340) (342). The ground truth label map (340) serves as one of the inputs to the loss function (330). Iterative adjustment (328) of the PE parameters used in the MLP(200) is performed to optimize (e.g., approximately minimize) the loss function (330) across the set of training views. Once the optimization criteria are met, the training process ends, and the corresponding "optimal" PE parameters are fixed and subsequently used in the operation test (work) mode. Further details about the operation test (work) mode are described later with reference to Figure 4.

[0038] For a view selected from the aforementioned set of training views, 5D inputs (304, 306) are obtained for a pre-trained neural network (NN2, 310) using camera parameters (302). In some specific examples, the neural network (310) implements NeRF as described above. The 5D inputs (304, 306) include the corresponding 3D spatial coordinates (x,y,z) (306) and 2D line of sight direction (θ,φ) (304). In response to the 5D inputs (304, 306), the pre-trained neural network (310) generates a 4D output (r,g,b,σ) (314). Position coding (312) is applied to the 3D spatial coordinates (x,y,z) (306) to generate the corresponding sequence γ(p) (318) (see, for example, equation (3)). The 4D output (r,g,b,σ)(314) of the neural network (310), along with the sequence γ(p)(318), provides input to the MLP(200) during training. In response to this input, the MLP(200) generates multiple semantic logits. Generate JPEG0007853519000009.jpg511(320).

[0039] Volume rendering (322) can be implemented, for example, in a manner consistent with equation (4), from the logit s(320) to the corresponding 2D semantic segmentation map of width W and height H. This is used to calculate JPEG0007853519000010.jpg524(324). An example of a mathematical equation for calculating the map(324) is as follows:

number

[0040] Figure 4 is a flowchart (400) showing the process of testing the MLP(NN1, 200) according to various embodiments. As already mentioned above, the term "test" refers to a working mode of operation in which a semantic segmentation map can be generated corresponding to any view of a 3D scene. Before entering the working mode of operation, the MLP(200) is trained as described above with reference to Figure 3. In the test process (400), the PE parameters of the MLP(200) are constant and fixed to the optimal values ​​determined, for example, using the training process (300). The pre-trained neural network (310) remains in the same configuration in the test process (400) as it was in the training process (300).

[0041] For a selected (e.g., arbitrary) new view, the corresponding 5D input (304, 306) is obtained for the neural network (310) using the corresponding camera parameters (402). In response to the 5D input (304, 306), the neural network (310) generates a 4D output (σ,r,g,b) (314). Position coding (312) is applied to the 3D spatial coordinates (x,y,z) (306) to generate the corresponding sequence γ(p) (318). The 4D output (σ,r,g,b) (314) of the neural network (310), along with the sequence γ(p) (318), provides input to the trained MLP (200). In response to this input, the trained MLP (200) generates multiple semantic logits The file JPEG0007853519000012.jpg511(320) is generated. Subsequent processing of the MLP output includes the step of applying the softmax function (426) to the semantic logit (320) to obtain the probability v of a ray / pixel belonging to class c. For example, for a pixel (h,w), the probability of belonging to class c can be calculated as follows:

number

[0042] Example of a loss function In a specific example, the loss function (330) is defined as the cross-entropy between the predictive semantic segmentation map (324) and the ground truth segmentation map (340), for example, as follows:

number

[0043] In another specific example, the loss function (330) is defined as the weighted cross-entropy between the predicted semantic segmentation map (324) and the ground truth segmentation map (340), for example, as follows:

number

number

[0044] Figure 5 is a flowchart (500) showing the calculation of weighted cross-entropy for the loss function (330) in several embodiments. The input to computation (500) includes a predicted semantic segmentation map (324), a ground truth segmentation map (340), and an RGB image (332). Edge detection (504) is performed on the RGB image (332) to detect object edges within it. Next, the weighting function w of equation (11) is calculated (502), giving object edge pixels a higher weight than object interior (body) pixels. In some examples, computation blocks (502, 504) can be implemented using deep learning-based edge detectors. Next, the loss function (330) is calculated according to equations (9) to (10).

[0045] In several other specific examples, the loss function (330) includes one or more weighted regularization terms added to the cross-entropy of equation (8) or the weighted cross-entropy of equation (10). Two exemplary examples of such regularization terms are described below with reference to Figures 6-7. In various other embodiments, other suitable regularization terms can be used in addition to, or instead of, these exemplary regularization terms. Equation (12) provides an example of the loss function (330) employing such regularization terms.

number

[0046] One example of a regularization term is to implement pixel-level edge regularization. The corresponding unary regularization term R(p) at pixel p is defined as follows:

number

number

[0047] Figure 6 is a flowchart (600) showing the calculation of a pixel-level regularization term (610) for a loss function (330) in several embodiments. The input to calculation (600) includes a predicted semantic segmentation map (324) and an RGB image (332). Edge detection (504) is performed on the RGB image (332) to detect object edges within it. An edge map (604) is generated based on the object edges detected in the block (504). A segmentation edge map (602) is generated based on the predicted semantic segmentation map (324). Next, based on the edge maps (602, 604), the pixel-level edge regularization term (610) is calculated according to equation (13).

[0048] Another example of a regularization term implements pairwise edge regularization. For pixels p and q, the corresponding pairwise regularization term R(p,q) is defined as follows:

number

[0049] In a specific example, D rgb This is a weighted graph G, where pixels are nodes and the size of the edges of an RGB image are used as edge weights. rgb This is the length of the shortest path P between the two pixels p and q shown above. The corresponding formula is as follows:

number

[0050] Figure 7 is a flowchart (700) showing the calculation of a pairwise edge regularization term (710) for the loss function (330) in several examples. The input to the calculation (700) includes a predicted semantic segmentation map (324) and an RGB image (332). Edge detection (504) is performed on the RGB image (332) to detect object edges within it. Based on the object edges detected in block (504), an edge map (604) is generated. Based on the edge map (604), a pairwise distance map (702) is generated according to equation (18). Based on the predicted semantic segmentation map (324), D s Using the above definition of (p,q), a pairwise distance map (704) is generated. Next, a pairwise edge regularization term (710) is calculated based on the distance map (702, 704) according to equation (16). In some specific examples, the pairwise edge regularization term (710) may be calculated over a local region around the pixel of interest. The local region may have a size of, for example, 8×8 or 16×16 pixels.

[0051] Hardware examples Figure 8 is a block diagram showing a computing device (800) according to one embodiment. The device (800) can be used, for example, to implement a process flow (100, 101). The device (800) comprises an input / output (I / O) device (810), an image processing engine (IPE, 820), and memory (830). The input / output device (810) may be used to enable the device (800) to receive parameters of at least a 3D representation (102) and output parameters of at least a neural segmentation field (110) and a segmentation map (116). The input / output device (810) may also be used to connect the device (800) to a display.

[0052] Memory (830) may have a buffer for receiving image data corresponding to the 3D scene to be rendered. The image data may be, for example, in the form of one or more image files. Once the image data is received, Memory (830) may provide a portion of the data to IPE (820) for processing. IPE (820) includes a processor (822) and memory (824). Memory (824) may store program code therein. When the program code is executed by the processor (822), it enables IPE (820) to perform image processing. This image processing includes, but is not limited to, image processing that follows some or all of the flowcharts (100, 101, 300, 400, 500, 600, 700) described above. The program code may, in particular, include program code used to emulate various neural networks, for example, NN1 (200) and NN2 (310) described above. Once IPE(820) has generated the various maps described above by executing the corresponding parts of the code, IPE(820) can perform the rendering process and provide the corresponding viewable image for viewing on the display. The viewable image may take the form of an appropriate image file output, for example, via an I / O device (810).

[0053] According to the embodiment described above, for example, with reference to the overview section and / or any combination of any part or all of Figures 1-8, an image processing method is provided which includes training a first neural network by a processor to render a segmentation map corresponding to an arbitrarily selected view of a 3D scene, wherein the training comprises the steps of: the processor calculating a color texture and volume density corresponding to a selected training view of the 3D scene, the calculating step being performed using a second neural network pre-trained to represent the 3D scene; the processor generating a predictive segmentation map corresponding to a selected training view of the 3D scene, the generating step being performed using the first neural network based on the color texture and volume density; and the processor adjusting the configuration parameters of the processing elements of the first neural network based on a loss function configured to receive a ground truth segmentation map corresponding to the selected training view as a first input, and further configured to receive the predictive segmentation map as a second input.

[0054] In some embodiments of the above method, the first neural network is a multilayer perceptron, and the second neural network is another multilayer perceptron implementing a neural radiance field.

[0055] In some embodiment of any of the above methods, the first neural network is configured to output a semantic logit corresponding to the selected training view of the 3D scene in response to the color texture and the volume density.

[0056] In some embodiment of any of the above methods, training further includes the steps of calculating the cross-entropy between the predicted segmentation map and the ground truth segmentation map, and obtaining a loss function value using the cross-entropy.

[0057] In some embodiment of any of the above methods, the loss function is configured to receive a color image of the 3D scene corresponding to the selected training view as a third input.

[0058] In some embodiment of any of the above methods, the training further includes the steps of: detecting object edges in the color image; assigning different weights to object edge pixels and object interior pixels; calculating the cross-entropy between the predicted segmentation map and the ground truth segmentation map using the different weights; and obtaining a loss function value using the cross-entropy.

[0059] In some embodiments of any of the above methods, the loss function includes one or more regularization terms that prioritize the alignment of corresponding edges in the predicted segmentation map and the ground truth segmentation map.

[0060] In some embodiment of the above method, the generating step is performed in response to the position coding of 3D spatial coordinates corresponding to the selected training view.

[0061] In some embodiment of any of the above methods, the image processing method further includes the step of creating a segmentation map corresponding to an arbitrarily selected view of the 3D scene by the processor, wherein the creation step is performed using the first neural network after training.

[0062] In some embodiments of any of the above methods, the image processing method further includes the steps of: generating a probability map by applying a softmax function to the segmentation map corresponding to the arbitrarily selected view of the 3D scene using the processor; and converting the probability map into a semantic label map representing the arbitrarily selected view of the 3D scene using the processor.

[0063] For example, in the Overview section and / or with reference to any one of Figures 1-8 or any combination of parts or all of Figures 1-8, according to other exemplary embodiments described above, a non-temporary computer-readable medium is provided for storing instructions that, when executed by the processor, cause the processor to perform an operation including one of the above methods.

[0064] For example, in the Overview section and / or with reference to any one or all of Figures 1-8, or a combination of some or all of Figures 1-8, an image processing apparatus comprising at least one processor and at least one memory containing program code, wherein the at least one memory and the program code are configured to cause the at least one processor to train the apparatus to train at least a first neural network to render a segmentation map corresponding to an arbitrarily selected view of a 3D scene, and for training the first neural network, the apparatus is configured as follows: i.e., a selected training of the 3D scene An image processing device is provided, which calculates a color texture and volume density corresponding to a ground view, the calculation is performed using a second neural network pre-trained to represent the 3D scene, and generates a predictive segmentation map corresponding to the selected training view of the 3D scene, the generation is performed using the first neural network based on the color texture and volume density, the configuration parameters of the processing elements of the first neural network are adjusted based on a loss function, the loss function takes a ground truth segmentation map corresponding to the selected training view as a first input, and further takes the predictive segmentation map as a second input.

[0065] In some embodiments of the above apparatus, the first neural network is a multilayer perceptron, and the second neural network is another multilayer perceptron implementing a neural radiance field.

[0066] In some embodiment of the above apparatus, the first neural network is configured to output a semantic logit corresponding to the selected training view of the 3D scene in response to the color texture and the volume density.

[0067] In some embodiment of the above apparatus, in order to train the first neural network, the apparatus is further configured to calculate the cross-entropy between the predictive segmentation map and the ground truth segmentation map, and to use the cross-entropy to obtain a loss function value.

[0068] In some embodiment of the above apparatus, the loss function is configured to receive a color image of the 3D scene corresponding to the selected training view as a third input.

[0069] In some embodiment of the above apparatus, to train the first neural network, the apparatus is further configured to detect object edges in the color image, assign different weights to object edge pixels and object interior pixels, use the different weights to calculate the cross-entropy between the predicted segmentation map and the ground truth segmentation map, and use the cross-entropy to obtain a loss function value.

[0070] In some embodiment of the above apparatus, the loss function includes one or more regularization terms that prioritize matching of corresponding edges in the predicted segmentation map and the ground truth segmentation map.

[0071] In any one of the embodiments of the above apparatus, the at least one memory and the program code are further configured to cause the at least one processor to cause the apparatus to create a segmentation map corresponding to an arbitrarily selected view of the 3D scene, the creation of which is performed using the first neural network after the training.

[0072] In any one of the embodiments of the above-described apparatus, the at least one memory and the program code are further configured by the at least one processor to cause the apparatus to apply a softmax function to the segmentation map corresponding to the arbitrarily selected view of the 3D scene to generate a probability map, and to convert the probability map into a semantic label map representing the arbitrarily selected view of the 3D scene.

[0073] For example, with reference to the Overview section and / or to any one or all of Figures 1-8, or any combination of parts or all of Figures 1-8, an image processing method is provided, comprising the steps of: testing a first neural network by a processor to render a segmentation map corresponding to an arbitrarily selected view of a 3D scene, wherein the testing step is the step of the processor calculating a color texture and volume density corresponding to the selected view of the 3D scene, the calculating step being performed using a 3D representation pre-trained to represent the 3D scene; and generating a segmentation map corresponding to the selected view of the 3D scene by the processor, the generating step being performed using the first neural network based on the color texture and volume density, wherein the first neural network is trained by the processor using the pre-trained 3D representation.

[0074] For example, with reference to the Overview section and / or to any one of Figures 1-8 or any combination of parts or all of Figures 1-8, an image processing apparatus is provided comprising at least one processor and at least one memory containing program code, wherein the at least one memory and program code are configured to cause the at least one processor to test a first neural network to render a segmentation map corresponding to an arbitrarily selected view of a 3D scene, and to test the first neural network, the apparatus is configured as follows: i.e., calculate a color texture and volume density corresponding to a selected view of the 3D scene, the calculation is performed using a 3D representation pre-trained to represent the 3D scene, and generate a segmentation map corresponding to the selected view of the 3D scene, the generation is performed using the first neural network based on the color texture and volume density, and the first neural network is trained using the pre-trained 3D representation.

[0075] With regard to the processes, systems, methods, heuristics, etc., described herein, the steps of such processes, etc., have been described as occurring in a certain ordered sequence. However, it goes without saying that such processes may be practiced with the steps performed in an order other than that described herein. Furthermore, it goes without saying that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the descriptions of processes herein are provided for the purpose of illustrating specific embodiments and should not be construed as limiting the scope of the claims.

[0076] Accordingly, it goes without saying that the above description is illustrative and not limiting. Reading the above description will reveal many embodiments and uses beyond the examples provided. The scope of the claims should not be determined by reference to the above description, but rather by reference to the appended claims, in accordance with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technology discussed herein, and that the disclosed systems and methods will be incorporated into such future embodiments. In other words, it goes without saying that this application is subject to modification and alteration.

[0077] All terms used in the claims are intended to be given the broadest reasonable interpretation and the ordinary meaning as understood by those skilled in the art described herein, unless expressly otherwise provided herein. In particular, singular articles such as “a,” “the,” and “said” mean that there is one or more elements described, unless expressly otherwise provided herein.

[0078] The disclosure summary is provided to enable readers to quickly grasp the nature of the technical disclosure. It is submitted with the understanding that it is not intended to be used to interpret or limit the claims or their meaning. Furthermore, as can be seen in the preceding detailed description, various features are grouped together in various embodiments for clarity. This method of disclosure does not reflect the intention that the embodiments described in the claims contain more features than are explicitly described in each claim. Rather, as reflected in the following claims, the inventive subject matter lies in fewer features than all the features of a single disclosed embodiment combined. Therefore, the following claims are incorporated into the detailed description herein, and each claim stands alone as the subject matter claimed individually.

[0079] This disclosure includes references to exemplary embodiments, but this specification is not intended to be constrained. Various modifications of the embodiments described, as well as other embodiments within the scope of this disclosure that are apparent to those skilled in the art to which this disclosure relates, are deemed to be within the principles and scope of this disclosure, as expressed, for example, in the following claims.

[0080] Some embodiments can be implemented as circuit-based processes, including possible implementations on a single integrated circuit.

[0081] Some embodiments can be embodied in the form of methods and apparatus for carrying out those methods. Some embodiments can also be embodied in the form of program code recorded on a tangible medium such as a magnetic recording medium, an optical recording medium, a solid-state memory, a floppy disk, a CD-ROM, a hard drive, or any other non-temporary machine-readable storage medium, and when the program code is loaded into and executed by a machine such as a computer, that machine becomes an apparatus for carrying out the various embodiments described herein. Some embodiments can also be embodied in the form of program code stored on a non-temporary machine-readable storage medium, including being loaded into and / or executed by a machine, and when the program code is loaded into and executed by a machine such as a computer or processor, that machine becomes an apparatus for carrying out the various embodiments described herein. When implemented on a general-purpose processor, the program code segment combines with the processor to provide a unique device that operates similarly to a particular logic circuit.

[0082] Unless otherwise specified, each number and range should be interpreted as an approximation, as if preceded by the words "about" or "approximately."

[0083] The use of figure numbers and / or figure reference labels in the claims is intended to identify one or more possible embodiments of the claimed subject matter in order to facilitate the interpretation of the claims. Such use is not necessarily construed as limiting those claims to the embodiments shown in the corresponding figures.

[0084] Where elements described in the following method claims are listed in a specific order with corresponding designations, these elements are not necessarily intended to be limited to being carried out in that specific order unless the claims imply a specific order for carrying out some or all of these elements.

[0085] Any reference in this specification to “one embodiment” or “a particular embodiment” means that certain features, structures, or characteristics described in relation to that embodiment may be included in at least one embodiment of this disclosure. While the phrase “in one embodiment” appears in various places in this specification, not all of them necessarily refer to the same embodiment, nor are separate or additional embodiments necessarily mutually exclusive with other embodiments. The same applies to the term “implementation.”

[0086] In this specification, unless otherwise specified, the use of sequential adjectives such as “first,” “second,” “third,” etc., to refer to one of several similar objects merely indicates that different examples of such similar objects are being referred to, and does not imply that the similar objects referred to in this manner must be in a corresponding order or sequence, whether temporal, spatial, ranking, or otherwise.

[0087] Unless otherwise defined herein, in addition to its simple meaning, the conjunction "if" can also be interpreted as "when," "upon," "in response to determining," or "in response to detecting," and its interpretation may depend on the corresponding specific context. For example, the phrases "if it is determined" or "if [a stated condition] is detected" can be interpreted as "upon determining" or "in response to determining," or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]."

[0088] Furthermore, in this specification, the terms “couple,” “coupling,” and “connect” refer to any implementation known or to be developed in the art in which energy is transferred between two or more elements, and the intervention of one or more additional elements is intended, though not required. Conversely, terms such as “directly coupled” and “directly connected” mean the absence of such additional elements.

[0089] The functionality of the various elements shown in the diagram, including the functional blocks labeled “Processor” and / or “Controller,” can be provided not only by the use of dedicated hardware but also by the use of software-executable hardware in conjunction with appropriate software. Where provided by a processor, the functionality may be provided by a single dedicated processor, a single shared processor, or multiple individual processors, some of which may be shared. Furthermore, the explicit use of the terms “Processor” or “Controller” should not be interpreted as referring only to, but not limited to, hardware capable of executing software, implicitly including, but not limited to, digital signal processor (DSP) hardware, network processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), read-only memory (ROM), random-access memory (RAM), and non-volatile storage for storing software. Other conventional and / or custom hardware may also be included. Similarly, the switches shown in the diagram are conceptual only. These functions may be executed through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or manually, and specific techniques can be selected by the implementer so that they are understood more concretely from the context.

[0090] As used in this application, the term “circuit” or “circuitry” can mean one or more or all of the following: (a) a hardware-only circuit implementation (such as an implementation of analog and / or digital circuits only); (b) a combination of hardware circuitry and software, for example (where applicable): (i) a combination of analog and / or digital hardware circuitry and software / firmware; (ii) any part of a hardware processor, software, and memory having software (including a digital signal processor) that works together to enable a device such as a mobile phone or a server to perform various functions; and (c) a hardware circuit and / or processor, such as a microprocessor or a part of a microprocessor, that requires software (e.g., firmware) to operate, but that software may not be present when not required for operation. This definition of circuitry applies to all use of the term in this application, including in the claims. As a further example, as used in this application, the term circuitry can also mean simply a hardware circuit or processor (or more processors) or an implementation of a part of a hardware circuit or processor and the software and / or firmware associated with it (or them). Furthermore, the term "circuit" also includes, as an example and applicable to a particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0091] It will be understood by those skilled in the art that any block diagram in this specification represents a conceptual diagram of an exemplary circuit embodying the principles of the present disclosure. Similarly, it will be understood that any flowchart, flow diagram, state transition diagram, pseudocode, etc., represents various processes that are substantially represented in a computer-readable medium and can be executed by a computer or processor, whether such a computer or processor is explicitly indicated or not.

[0092] The “BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS” in this specification are intended to present several exemplary embodiments, and additional embodiments are described with reference to the “DETAILED DESCRIPTION” and / or one or more drawings. The “BRIEF SUMMARY OF SOME SPECIFIC EMBODIMENTS” are not intended to identify essential elements or features of the claimed subject matter, nor are they intended to limit the scope of the claimed subject matter.

Claims

1. An image processing method comprising training a first neural network (NN) by a processor to render a segmentation map corresponding to an arbitrarily selected view of a three-dimensional (3D) scene, wherein the training is: A step of the processor calculating a color texture and volume density corresponding to a selected training view of the 3D scene, the step being performed using a 3D representation that has been pre-trained to represent the 3D scene; A step of generating a predictive segmentation map corresponding to the selected training view of the 3D scene by the processor, the step of using the first neural network based on the color texture, the volume density, and the position-encoded 3D spatial coordinates corresponding to the selected training view, The process involves the processor adjusting the configuration parameters of the NN elements of the first neural network based on a loss function configured to receive a ground truth segmentation map corresponding to the selected training view as a first input, and further configured to receive the predicted segmentation map as a second input. Image processing methods including [specific details omitted].

2. The first neural network described above is a multilayer perceptron, The aforementioned 3D representation is a second neural network that implements a neural radiance field. The image processing method according to claim 1.

3. The first neural network is configured to output a semantic logit corresponding to the selected training view of the 3D scene in response to the color texture and the volume density. The image processing method according to claim 1.

4. The aforementioned training is The steps include: calculating the cross-entropy between the predicted segmentation map and the ground truth segmentation map; The steps include obtaining a loss function value using the cross-entropy, Further including, The image processing method according to claim 1.

5. The loss function is configured to receive the color image of the 3D scene corresponding to the selected training view as a third input. The image processing method according to claim 1.

6. The aforementioned training is The steps include detecting object edges in the aforementioned color image, A step of assigning different weights to object edge pixels and object interior pixels, The steps include: calculating the cross-entropy between the predicted segmentation map and the ground truth segmentation map using the different weights; The steps include obtaining the loss function value using the cross-entropy, Further including, The image processing method according to claim 5.

7. The loss function includes one or more regularization terms that prioritize matching corresponding edges in the predicted segmentation map and the ground truth segmentation map. The image processing method according to claim 1.

8. A step of generating a segmentation map corresponding to an arbitrarily selected view of the 3D scene using the processor, the step of being performed using the first neural network after the training, The processor performs the steps of applying a softmax function to the segment map corresponding to the arbitrarily selected view of the 3D scene to generate a probability map, The processor performs the steps of converting the probability map into a semantic label map representing the arbitrarily selected view of the 3D scene, The image processing method according to claim 1, further comprising:

9. A computer program that, when executed by a processor, includes instructions causing the processor to perform an operation including the method according to any one of claims 1 to 8.

10. At least one processor, At least one memory location containing program code, An image processing apparatus comprising, The at least one memory and the program code are configured to cause the image processing device to train at least one first neural network (NN) for rendering a segmentation map corresponding to an arbitrarily selected view of a three-dimensional (3D) scene by the at least one processor. To train the first neural network, the image processing device, The color texture and volume density corresponding to the selected training view of the 3D scene are calculated, and the calculation is performed using a 3D representation that has been pre-trained to represent the 3D scene. A predictive segmentation map corresponding to the selected training view of the 3D scene is generated, and this generation is performed using the first neural network based on the color texture, the volume density, and the position-encoded 3D spatial coordinates corresponding to the selected training view. The configuration parameters of the NN nodes of the first neural network are adjusted based on a loss function, the loss function receiving a ground truth segmentation map corresponding to the selected training view as a first input, and further receiving the predicted segmentation map as a second input. An image processing device configured in such a way.

11. The first neural network described above is a multilayer perceptron, The aforementioned 3D representation is a second neural network that implements a neural radiance field. The image processing apparatus according to claim 10.

12. The first neural network is configured to output a semantic logit corresponding to the selected training view of the 3D scene in response to the color texture and the volume density. The image processing apparatus according to claim 10 or 11.

13. To train the first neural network, the image processing device, The cross-entropy between the predicted segmentation map and the ground truth segmentation map is calculated. The loss function value is obtained using the cross-entropy described above. It is further structured in the following way: The image processing apparatus according to claim 10 or 11.

14. The loss function is configured to receive a color image of the 3D scene corresponding to the selected training view as a third input. The image processing apparatus according to claim 10 or 11.

15. To train the first neural network, the apparatus, The object edges in the aforementioned color image are detected, By assigning different weights to object edge pixels and object interior pixels, Using the different weights described above, the cross-entropy between the predicted segmentation map and the ground truth segmentation map is calculated. The loss function value is obtained using the cross-entropy described above. It is further structured in the following way: The image processing apparatus according to claim 14.

16. The loss function includes one or more regularization terms that prioritize matching corresponding edges in the predicted segmentation map and the ground truth segmentation map. The image processing apparatus according to claim 10 or 11.

17. The at least one memory and the program code are provided to the device by the at least one processor. A segmentation map is created corresponding to an arbitrarily selected view of the 3D scene, and this creation is performed using the first neural network after the training. The softmax function is applied to the segmentation map corresponding to the arbitrarily selected view of the 3D scene to generate a probability map. The probability map is converted into a semantic label map representing the arbitrarily selected view of the 3D scene. It is further structured in the following way: The image processing apparatus according to claim 10 or 11.

18. An image processing method comprising the step of testing a first neural network to render a segmentation map corresponding to an arbitrarily selected view of a three-dimensional (3D) scene using a processor, wherein the testing step is: The steps include: a step of using the processor to calculate a color texture and volume density corresponding to a selected view of the 3D scene, which is performed using a 3D representation that has been pre-trained to represent the 3D scene; The steps include: generating a segmentation map corresponding to the selected view of the 3D scene using the processor, which is performed using the first neural network based on the color texture, the volume density, and the position-encoded 3D spatial coordinates corresponding to the selected view; Includes, The first neural network uses the pre-trained 3D representation and is trained by the processor. Image processing methods.

19. At least one processor, At least one memory location containing program code, An image processing apparatus comprising, The at least one memory and the program code are configured to cause the at least one processor to test the device with at least one first neural network (NN) for rendering a segmentation map corresponding to an arbitrarily selected view of a three-dimensional (3D) scene. To test the first neural network, the apparatus is used. The color texture and volume density corresponding to the selected training view of the 3D scene are calculated, and the calculation is performed using a 3D representation that has been pre-trained to represent the 3D scene. A predictive segmentation map is generated corresponding to the selected training view of the 3D scene, and the generation is performed using the first neural network based on the color texture, the volume density, and the position-encoded 3D spatial coordinates corresponding to the selected view. It is configured in such a way, The first neural network is trained using the pre-trained 3D representation. Image processing device.

Images