Apparatus, method, storage medium for representing a three-dimensional, 3D, object

Abstract

An apparatus for representing a three-dimensional (3D) object includes: a memory storing instructions; and a processor configured to execute instructions to: transmit a two-dimensional (2D) image to an external device; receive, based on the transmitted 2D image, mapping function parameters obtained from the external device using a first neural network; set a mapping function of a second neural network based on the received mapping function parameters; and obtain a 3D object corresponding to the 2D image using the second neural network with its mapping function set, based on a 3D sample.

Description

Technical Field

[0001] This disclosure relates to representing three-dimensional (3D) objects, and more specifically, to higher-order function networks for learning composable 3D objects and methods for manipulating them. Background Technology

[0002] In pure geometry, the 3D object O is A subset of. However, directly representing this infinite set of points in memory is not straightforward.

[0003] Voxel-based methods use uniform discretization of this set to extend convolutional neural network (CNN)-based methods to the 3D world. However, the inherent sparsity of 3D surfaces makes voxelization inefficient in terms of both memory and computation time.

[0004] Partition-based methods (such as octrees) address the space efficiency drawbacks of voxelization, but they are cumbersome to implement and have low query efficiency.

[0005] Point set representations (discrete and finite subsets of O) have also gained popularity due to the fact that they retain the simplicity of voxel-based methods while eliminating the storage and computational burdens of voxel-based methods. Although point clouds are more flexible than voxels, it remains unclear how to adapt them to tasks that produce resolution predictions of arbitrary or varying resolutions. Summary of the Invention

[0006] Solutions to technical problems

[0007] According to an embodiment, an apparatus for representing a three-dimensional (3D) object includes: a memory storing instructions, and a processor configured to execute instructions to: send a two-dimensional (2D) image to an external device; receive, based on the sent 2D image, mapping function parameters obtained from the external device using a first neural network; set a mapping function of a second neural network based on the received mapping function parameters; and obtain a 3D object corresponding to the 2D image using the second neural network whose mapping function is set, based on a 3D sample.

[0008] 3D samples can be the surface or interior of a 3D canonical domain.

[0009] A 3D standard domain can be a unit sphere or a unit cube.

[0010] A 3D object can be the surface or interior of an object included in a 2D image.

[0011] The first neural network can be a convolutional neural network (CNN) trained to output mapping function parameters based on a 2D image as input to the CNN, and can be trained in combination with a second neural network.

[0012] The second neural network can be a convolutional neural network (CNN) trained to output 3D objects based on 3D samples input to the CNN, and can be trained in combination with the first neural network.

[0013] The device can be a client device that is separate from and located outside of external devices.

[0014] External devices can be server devices that are separate from and located outside the device.

[0015] According to an embodiment, a method for representing a three-dimensional (3D) object is performed by an apparatus and includes: sending a two-dimensional (2D) image to an external device; receiving, based on the sent 2D image, mapping function parameters obtained from the external device using a first neural network; setting a mapping function of a second neural network based on the received mapping function parameters; and obtaining a 3D object corresponding to the 2D image using the second neural network whose mapping function is set, based on a 3D sample.

[0016] 3D samples can be the surface or interior of a 3D standard domain.

[0017] A 3D standard domain can be a unit sphere or a unit cube.

[0018] A 3D object can be the surface or interior of an object included in a 2D image.

[0019] The first neural network can be a convolutional neural network (CNN) trained to output mapping function parameters based on a 2D image as input to the CNN, and can be trained in combination with a second neural network.

[0020] The second neural network can be a convolutional neural network (CNN) trained to output 3D objects based on 3D samples input to the CNN, and can be trained in combination with the first neural network.

[0021] The device can be a client device that is separate from and located outside of external devices.

[0022] External devices can be server devices that are separate from and located outside the device.

[0023] According to an embodiment, a non-transitory computer-readable storage medium stores instructions configured to cause a processor to: send a two-dimensional (2D) image to an external device; receive, based on the sent 2D image, mapping function parameters obtained from the external device using a first neural network; set a mapping function of a second neural network based on the received mapping function parameters; and obtain a 3D object corresponding to the 2D image using the second neural network whose mapping function is set, based on a 3D sample.

[0024] 3D samples can be the surface or interior of a 3D standard domain.

[0025] A 3D standard domain can be a unit sphere or a unit cube.

[0026] A 3D object can be the surface or interior of an object included in a 2D image.

[0027] Additional aspects will be set forth in part in the description which follows, and will become clear in part from the description, or may be learned by practice of the embodiments presented in this disclosure. Attached Figure Description

[0028] The above and other aspects, features, and characteristics of the embodiments of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings, in which:

[0029] Figure 1 This is a block diagram of an apparatus for representing a 3D object according to an embodiment;

[0030] Figure 2A It is to achieve Figure 1 A block diagram of the client device of the apparatus shown;

[0031] Figure 2B It is to achieve Figure 1 A block diagram of the system of the apparatus shown;

[0032] Figure 3 These are, according to an embodiment, two-dimensional (2D) images and diagrams of 3D objects respectively input to a device for representing a 3D object and output from a device for representing a 3D object;

[0033] Figure 4 This is a diagram illustrating a method for obtaining 3D objects by sampling different subsets of a standard 3D domain using a neural network representing a mapping function, according to an embodiment.

[0034] Figure 5 These are diagrams of 3D objects corresponding to 3D samples in the 3D standard domain, according to the embodiments.

[0035] Figure 6This is a flowchart of a method for representing a 3D object according to an embodiment, the method being implemented by a client device;

[0036] Figure 7 This is a flowchart of a method for representing a 3D object according to an embodiment, which is implemented by a system;

[0037] Figure 8A This is a block diagram of an apparatus for representing a first 3D object and a second 3D object according to an embodiment;

[0038] Figure 8B This is a diagram of a method for interpolating between a first 3D object and a second 3D object through function composition according to an embodiment;

[0039] Figure 8C This is a diagram of a method for interpolating between a first 3D object and a second 3D object using a higher-order k-mapping according to an embodiment;

[0040] Figure 8D yes Figure 8A A diagram showing the usage of the device;

[0041] Figure 9 yes Figure 2B The diagram shows the usage of the system, where the robot implementation includes client devices within the system.

[0042] Figure 10 yes Figure 2B The diagram shown illustrates the usage of the system, where augmented reality (AR) glasses are implemented as client devices included in the system; and

[0043] Figure 11 This is a block diagram of an electronic device for representing a 3D object according to an embodiment. Detailed Implementation

[0044] Embodiments of this disclosure provide higher-order function networks for learning composable 3D objects and methods for operating them. That is, 3D objects can be represented using higher-order functions (i.e., the weights and biases of a small neural network). This representation can be used to reconstruct 3D objects by mapping points from a standard space to define the 3D object. The reconstructed 3D object can be more accurate than those produced by prior art reconstruction methods. Furthermore, directly encoding 3D objects into neural networks is highly parameter-efficient compared to object representations that encode objects of interest into latent vector codewords.

[0045] The embodiment also provides a method for object interpolation in a latent function space, wherein new, coherent objects are generated by composing and reconstructing functions for various objects.

[0046] Because this disclosure allows for various variations and numerous examples, embodiments will be shown in the accompanying drawings and described in detail in the written description. However, this is not intended to limit this disclosure to modes of practice, and it will be understood that all changes, equivalents, and substitutions without departing from the spirit and scope of this disclosure are covered herein.

[0047] In the description of the embodiments, detailed descriptions of related technologies are omitted where such descriptions would unnecessarily obscure the focus of this disclosure. Furthermore, the numbers used in the description of this specification (e.g., first, second, etc.) are identifier codes used to distinguish one element from another.

[0048] Furthermore, it will be understood in this specification that when components are “connected” or “coupled” to each other, the components may be directly connected or coupled to each other, but may also be connected or coupled to each other alternately through intermediate components, unless otherwise indicated.

[0049] In this specification, regarding elements represented as "units" or "modules," depending on their subdivided functions, two or more elements may be combined into one element, or one element may be divided into two or more elements. Furthermore, each element described below, in addition to its own primary function, may perform some or all of the functions performed by another element, and some of the primary functions of each element may be entirely performed by another component.

[0050] In addition, in this specification, "image" or "picture" may mean a still image, a moving image comprising multiple consecutive still images (or frames), or a video.

[0051] Furthermore, in this specification, deep neural networks (DNNs) or CNNs are representative examples of artificial neural network models that simulate brain neurons, and are not limited to artificial neural network models that use algorithms.

[0052] Furthermore, in this specification, "parameters" are values ​​used during the operation of each layer of the neural network, and may include, for example, weights used when input values ​​are applied to operational expressions. Here, parameters can be represented in matrix form. Parameters are values ​​set as the result of training and can be updated as needed using individual training data.

[0053] Throughout this disclosure, the expression "at least one of a, b, or c" indicates only a, only b, only c, both a and b, both a and c, both b and c, all a, b, and c, or variations thereof.

[0054] Figure 1This is a block diagram of a device 100 for representing a 3D object according to an embodiment.

[0055] like Figure 1 As shown, the device 100 includes a first neural network 105, a processor 110, and a second neural network 115.

[0056] A first neural network 105 acquires a single 2D image and obtains mapping function parameters based on the acquired 2D image. The 2D image may be an RGB image including an object or a portion of an object. The first neural network 105 may be a pre-trained DNN or CNN for mapping the 2D image to the mapping function parameters. The mapping function parameters are used to set a second neural network 115, which may be a pre-trained DNN or CNN representing a mapping function or higher-order function for mapping 3D samples from a standard 3D domain to 3D objects. The first neural network 105 and the second neural network 115 may be pre-trained simultaneously and in combination with each other.

[0057] 2D images can be obtained from data storage media including magnetic media (such as hard disks, floppy disks, or magnetic tapes), optical recording media (such as CD-ROMs or DVDs), or magneto-optical media (such as floppy disks). 2D images can also be obtained from a camera and via processor 110.

[0058] The processor 110 obtains the mapping function parameters from the first neural network 105 and uses the obtained mapping function parameters to set the mapping function of the second neural network 115.

[0059] Based on the mapping function of the configured second neural network 115, the second neural network 115 acquires or samples 3D samples, and obtains a 3D object corresponding to the 2D image based on the acquired 3D samples. The 3D sample can be the surface or interior of any predetermined standard domain or space (e.g., a unit sphere and a unit cube (e.g., a 512-dimensional unit hypercube)). The 3D object can be a 3D representation (e.g., surface or interior) of an object included in the 2D image, even if the 2D image only includes a portion of the object.

[0060] 3D samples can be obtained from data storage media including magnetic media (such as hard disks, floppy disks, or magnetic tapes), optical recording media (such as CD-ROMs or DVDs), or magneto-optical media (such as floppy disks). 3D samples can also be obtained via processor 110.

[0061] Specifically, for a 2D image I that can be a 64x64 RGB image, the mapping function parameter θ of the mapping network (i.e., the second neural network 115) I This can be represented by the following equation 1:

[0062]

[0063] Among them, encoder (That is, the first neural network 105) uses parameters Training to output a mapping function based on the input 2D image I The mapping function parameter θ I .

[0064] 3D object O is represented as a mapping function Where the mapping function parameter θ I For a 3D object O, it is unique. Mapping function Represented as a multilayer perceptron (MLP), it maps 3D samples from the surface or interior of a standard space X (such as a unit sphere) to a defined reconstructed 3D object. The set of points on or inside the surface, as represented in Equation 2 below:

[0065]

[0066] This construction allows for online reconstructing of 3D objects by continuously sampling 3D samples from standard space X as needed. Sampling at higher or lower resolutions can be performed, and these 3D samples can correspond to points within a unit sphere X = {x: ||x||2≤1}.

[0067] By representing 3D objects as weights and biases of a second neural network 115 and enabling the 3D objects to be modulated by simply changing the connection strength of the second neural network 115, the device 100 reduces the size of the neural network used to represent 3D objects in conventional methods, thereby reducing the processing time for reconstructing 3D objects from 2D images. Furthermore, the device 100 outperforms conventional methods by generating greater similarity between the input 2D image and the output 3D object. The device 100 can also obtain 3D objects of arbitrary resolution by adjusting the density of the input 3D samples.

[0068] In an embodiment, the device 100 may be configured to obtain a partial point cloud instead of a 2D image, and to obtain a 3D object corresponding to the partial point cloud.

[0069] During the training of the first neural network 105 and the second neural network 115, the image I and the corresponding ground truth point cloud model are trained. Perform sampling. It can contain 10,000 points sampled from the surface of a real object O. Obtain the mapping function. And as in Equation 2, an estimated reconstruction of object O is generated. However, this is calculated only for a sample of 1000 points in X (which is uncountably infinite). This sample was drawn from a uniform distribution on the set. The model was computed using differentiable set similarity measures such as Chamfer distance or Earth Mover's Distance. The predicted loss. For example, the asymmetric chamfer distance C(X,Y) is a quasimetric used to quantify the similarity between two point sets X and Y, as shown in Equation 3 below:

[0070]

[0071] encoder (The first neural network 105) is trained to minimize the symmetric objective function.

[0072] Furthermore, the processor 110 can interpolate between input images by composing the roots of the reconstruction mapping function. Specifically, Equation 2 can be extended in the following Equation 4, where object O in image I is represented as the mapping function. k-th power:

[0073]

[0074] Among them, f k Defined as the composition of f with itself raised to the power of (k-1): f k (x)=f(f (k-1) (x)).

[0075] The mapping function for reconstructing object O in image I by its k-th power This can be referred to as the k-mapping of object O.

[0076] This modification to Equation 2 adds an additional constraint to the mapping: the domain and the codomain must be the same. However, evaluating the power of f utilizes the power of weight sharing in the neural network architecture; for an MLP mapping architecture with l layers, evaluating its kth power is equivalent to an MLP with l x k-1 layers having shared weights, which is equivalent to a recurrent neural network (RNN) with weight sharing in time.

[0077] When obtaining two objects O I and O J Given RGB images I and J, and applying the encoder discussed above to these images, we obtain images with parameters respectively. and of and By evaluating 2 k Any one of the possible functions can be used in object O. I and O J Interpolation between them can be expressed by the following equation 5:

[0078]

[0079] Each mapping function The parameter is k-mapping The parameters or k-mapping The parameters.

[0080] k-mapping and Encode semantically meaningful information about the underlying structure. (See below for reference.) Figures 8A-8D This will be described in further detail.

[0081] Figure 2A It is to achieve Figure 1 Block diagram of client device 205A of device 100 shown.

[0082] like Figure 2A As shown, the client device 205 includes Figure 1 The illustrated device 100 comprises a first neural network 105, a processor 110, and a second neural network 115. Therefore, the client device 205 uses the first neural network 105, the processor 110, and the second neural network 115 to obtain a single 2D image and to obtain a 3D object corresponding to the obtained 2D image, as shown in the reference. Figure 1 As described.

[0083] Client device 205A can include any type of electronic device, such as smartphones, laptops, personal computers (PCs), smart TVs, etc.

[0084] Figure 2B It is to achieve Figure 1 A block diagram of the system 200 of the apparatus 100 shown.

[0085] like Figure 2B As shown, system 200 includes client device 205B and server device 210. Client device 205B includes processor 110 and second neural network 115, as shown in reference... Figure 1 As described, and also includes a communication interface 207. Server device 210 includes a first neural network 105, as referenced... Figure 1 As described, it also includes a communication interface 212 and a processor 214.

[0086] Client device 205B can include any type of electronic device, such as smartphones, laptops, personal computers (PCs), smart TVs, etc.

[0087] For reference Figure 1 As described, client device 205B uses processor 110 to acquire a single 2D image. Client device 205B further transmits the acquired 2D image to server device 210 via communication interface 207.

[0088] Server device 210 receives 2D images from client device 205B via communication interface 212. Server device 210 further uses processor 214 to input the received 2D images into first neural network 105.

[0089] The first neural network 105 obtains the mapping function parameters based on the input 2D image, as shown in the reference. Figure 1 As described. Server device 210 further sends the obtained mapping function parameters to client device 205B via communication interface 212.

[0090] Client device 205B receives mapping function parameters from server device 210 via communication interface 207. Client device 205B uses the received mapping function parameters and further uses processor 110 to set the mapping function of the second neural network 115, as shown in reference [reference needed]. Figure 1 As described.

[0091] For reference Figure 1 As described, based on the mapping function of the configured second neural network 115, the second neural network 115 obtains or samples 3D samples, and obtains a 3D object corresponding to a 2D image based on the obtained 3D samples.

[0092] The first neural network 105, used to map 2D images to mapping function parameters, can be larger in size than the second neural network 115, used to map 3D samples to 3D objects. Therefore, by implementing the first neural network 105 on the server device 210 and the second neural network 115 on the client device 205B, the processing efficiency of the client device 205B for mapping 2D images to 3D objects is significantly higher than that of the second neural network 115. Figure 2A The processing efficiency of the client device 205A shown in the figure can be improved.

[0093] exist Figures 1 to 2BIn this embodiment, each of processors 110 and 214 may be implemented by a dedicated processor or by a combination of software and a general-purpose processor (such as an application processor (AP), a central processing unit (CPU), or a graphics processing unit (GPU)). A dedicated processor may be implemented by including memory for implementing embodiments of this disclosure or by including a memory processor for using external memory.

[0094] Furthermore, each of processors 110 and 214 can be configured with multiple processors. In this case, each of processors 110 and 214 can be implemented by a combination of dedicated processors or by a combination of software and general-purpose processors (such as AP, CPU, or GPU).

[0095] exist Figure 2B In this context, each of communication interface 207 and communication interface 212 may include one or more components that enable communication to be performed via a local area network (LAN), a wide area network (WAN), a value-added network (VAN), a mobile radio communication network, a satellite communication network, or a combination thereof.

[0096] exist Figures 1 to 2B In this configuration, each of device 100, client device 205A, client device 205B, and server device 210 may include a CPU, memory, and a computer program including instructions. The computer program is stored in memory. Device 100, client device 205A, client device 205B, and server device 210 may respectively execute references based on the CPU's execution of the computer program. Figures 1 to 2B The described functionality. (Reference) Figures 1 to 2B The described functions are performed by a dedicated hardware chip and / or CPU.

[0097] Figure 3 These are, according to an embodiment, 2D images and 3D object diagrams that are respectively input to a device for representing a 3D object and output from a device for representing a 3D object.

[0098] Figure 3 Part (a) is a 2D image of the object (in this case, the airplane). Figure 1 The device 100 shown can transform a 2D image into a 3D object corresponding to the 2D image, such as... Figure 3As shown in part (b), in this case, the 3D object is a 3D representation of the aircraft included in the 2D image.

[0099] Figure 4 This is a diagram illustrating a method for obtaining 3D objects by sampling different subsets of a standard 3D domain using a neural network representing a mapping function, according to an embodiment.

[0100] Figure 4 Parts (a)-(d) respectively show examples of, for example Figure 1 The second neural network 115 shown in the image samples different highlighted portions of points on the sphere. Figure 4 Parts (a)-(d) also show the different highlighted portions of a 3D object (e.g., an airplane) reconstructed by, for example, a second neural network 115, when different portions of points on the sphere are sampled respectively. Specifically, the mapping function f θ It can be determined by different highlighted parts of the sphere. And their corresponding distinct highlighted portions within the entire reconstructed 3D object {f θ (x i ), x i Visualize using ∈X}.

[0101] Figure 5 These are diagrams of 3D objects corresponding to 3D shape points according to an embodiment.

[0102] Figure 5 Parts (a)-(c) respectively show what can be made by, for example Figure 1 The second neural network 115 shows the reconstructed 3D objects (table, chair, and airplane). The 3D objects are represented using points sampled to obtain the spheres of the 3D objects.

[0103] Figure 6 This is a flowchart of a method 600 for representing a 3D object according to an embodiment, which is implemented by a client device.

[0104] refer to Figure 6 Method 600 can be derived from Figure 2A The client device 205A is used to execute this.

[0105] In operation 605, method 600 includes obtaining mapping function parameters based on a 2D image using a first neural network.

[0106] In operation 610, method 600 includes setting the mapping function of the second neural network based on the obtained mapping function parameters.

[0107] In operation 615, method 600 includes obtaining a 3D object corresponding to a 2D image using a second neural network whose mapping function is set, based on 3D samples.

[0108] Refer to each Figure 1 and Figure 2A The apparatus 100 and client device 205A shown herein describe a detailed description of method 600, therefore a detailed description of method 600 will be omitted herein.

[0109] Figure 7 This is a flowchart of a method 700 for representing a 3D object according to an embodiment, which is implemented by a system.

[0110] refer to Figure 7 This method can be included by […]. Figure 2B The client device 205B in the system shown is used to execute this.

[0111] In operation 705, method 700 includes sending a 2D image to an external device.

[0112] In operation 710, method 700 includes receiving, based on the transmission of a 2D image, mapping function parameters obtained from an external device using a first neural network.

[0113] In operation 715, method 700 includes setting the mapping function of the second neural network based on the received mapping function parameters.

[0114] In operation 720, method 700 includes obtaining a 3D object corresponding to a 2D image using a second neural network whose mapping function is set based on 3D samples.

[0115] refer to Figure 2B The system 200 shown describes a detailed description of method 700, therefore a detailed description of method 700 will be omitted in this document.

[0116] Figure 8A This is a block diagram of a device 100' for representing a first 3D object and a second 3D object according to an embodiment.

[0117] like Figure 8A As shown, the device 100' includes a first neural network 105, a processor 110, and a second neural network 115, as referenced. Figure 1 As described. The device 100' also includes a third neural network 120.

[0118] Apart from Figure 1 In addition, refer to Figure 8A The first neural network 105 obtains the first 2D image (i.e., Figure 1 A second 2D image in addition to the original 2D image. The 2D image can be an RGB image that includes the object or a portion of the object.

[0119] In addition to the first mapping function parameter (i.e., Figure 1 In addition to the mapping function parameters, the first neural network 105 obtains second mapping function parameters based on the acquired second 2D image. The second mapping function parameters are used to set up a third neural network 120, which may be a pre-trained DNN or CNN, representing a mapping function used to map 3D samples from the 3D standard domain to objects other than the first 3D object (i.e., the first 2D image). Figure 1 The mapping function or higher-order function of a second 3D object other than the 3D object itself. The first neural network 105, the second neural network 115, and the third neural network 120 can be pre-trained simultaneously and in combination with each other.

[0120] The second 2D image can be obtained from a data storage medium including magnetic media (such as hard disks, floppy disks, or magnetic tapes), optical recording media (such as CD-ROMs or DVDs), or magneto-optical media (such as floppy disks). The second 2D image can also be obtained from a camera and via processor 110.

[0121] The processor 110 obtains the second mapping function parameters from the first neural network 105 and uses the obtained second mapping function parameters to set the mapping function of the third neural network 120.

[0122] Based on the mapping function of the configured third neural network 120, the third neural network 120 acquires or samples 3D samples, and obtains a second 3D object corresponding to the second 2D image based on the acquired 3D samples. The 3D samples correspond to a reference image. Figure 1 The described 3D sample. The second 3D object can be a 3D representation (e.g., surface or interior) of an object included in the second 2D image, even if the second 2D image only includes a portion of the object.

[0123] Apart from Figure 1 In addition, refer to Figure 8A Based on the mapping function of the configured second neural network 115, the second neural network 115 can obtain or sample the obtained first 3D object and / or the obtained second 3D object via the processor 110, and re-obtain the first 3D object based on the obtained first 3D object and / or the obtained second 3D object. The re-obtained first 3D object can be an interpolated or blended 3D representation (e.g., surface or interior) between the obtained first 3D object and / or the obtained second 3D object. This re-obtaining of the first 3D object can be repeated any number of times under the control of the processor 110.

[0124] Similarly, based on the mapping function of the configured third neural network 120, the third neural network 120 can obtain or sample the obtained first 3D object and / or the obtained second 3D object via the processor 110, and re-obtain the second 3D object based on the obtained first 3D object and / or the obtained second 3D object. The re-obtained second 3D object can be an interpolated or blended 3D representation (e.g., surface or interior) between the obtained first 3D object and / or the obtained second 3D object. This re-obtaining of the second 3D object can be repeated any number of times under the control of the processor 110.

[0125] Figure 8B This is a diagram illustrating a method for interpolating between a first 3D object fA(X) and a second 3D object fB(X) through function composition, according to an embodiment.

[0126] like Figure 8B As shown, the first object OA in the first 2D image is a passenger aircraft, and the second object OB in the second 2D image is a propeller aircraft.

[0127] refer to Figure 8A and Figure 8B The first neural network 105 obtains the mapping function parameters of the first function fA based on the first 2D image of the first object OA, and obtains the mapping function parameters of the second function fB based on the second 2D image of the second object OB.

[0128] Based on the first mapping function fA of the second neural network 115, the second neural network 115 obtains or samples a 3D sample X, and obtains a first 3D object fA(X) corresponding to the first object OA based on the obtained 3D sample X. Based on the second mapping function fB of the third neural network 120, the third neural network 120 obtains or samples a 3D sample X, and obtains a second 3D object fB(X) corresponding to the second object OB based on the obtained 3D sample X.

[0129] Furthermore, the second neural network 115 acquires or samples the acquired first 3D object fA(X), and re-acquires the first 3D object fA(fA(X)) based on the acquired first 3D object fA(X). The re-acquired first 3D object fA(fA(X)) is an interpolated 3D representation of the acquired first 3D object fA(X), and is more similar to the first object OA compared to the acquired first 3D object fA(X).

[0130] The third neural network 120 obtains or samples the obtained first 3D object fA(X), and re-obtains the second 3D object fB(fA(X)) based on the obtained first 3D object fA(X). The re-obtained second 3D object fB(fA(X)) is an interpolated or blended 3D representation between the obtained first 3D object fA(X) and the obtained second 3D object fB(X), and corresponds to a passenger aircraft (first object OA) with a wing wider than that of a propeller plane (second object OB).

[0131] The second neural network 115 obtains or samples the obtained second 3D object fB(X), and re-obtains the first 3D object fA(fB(X)) based on the obtained second 3D object fB(X). The re-obtained first 3D object fA(fB(X)) is an interpolated or blended 3D representation between the obtained first 3D object fA(X) and the obtained second 3D object fB(X), and corresponds to a propeller aircraft (second object OB) with a wing that is straighter than that of a passenger aircraft (first object OA).

[0132] The third neural network 120 obtains or samples the obtained second 3D object fB(X), and re-obtains the second 3D object fB(fB(X)) based on the obtained second 3D object fB(X). The re-obtained second 3D object fB(fB(X)) is an interpolated 3D representation of the obtained second 3D object fB(X), and is more similar to the second object OB than the obtained second 3D object fB(X).

[0133] As described above, by interpolating between the first 3D object fA(X) and the second 3D object fB(X), new 3D objects can be obtained, such as the re-obtained second 3D object fB(fA(X)) and the re-obtained first 3D object fA(fB(X)).

[0134] Figure 8C This is a diagram of a method for interpolating between a first 3D object and a second 3D object using a higher-order k-mapping according to an embodiment.

[0135] like Figure 8C As shown, the first object OA in the first 2D image is a fighter jet, and the second object OB in the second 2D image is a passenger aircraft.

[0136] A new 3D object can be obtained by sampling a 3D sample X using any one or any combination of the first mapping function fA of the first neural network and the second mapping function fB of the second neural network. For example, an intermediate 3D object can be obtained by sampling a 3D sample X using the first mapping function fA twice and then using the second mapping function fB twice. Intermediate 3D object It is an interpolated or blended 3D representation between the first object OA and the second object OB, while the intermediate 3D object... The 3D object on the left is more similar to the first object OA, and the 3D object in the middle... The 3D object on the right is more similar to the second object OB.

[0137] Figure 8D yes Figure 8A A diagram showing the use of the device 100' shown.

[0138] like Figure 8D As shown, the smartphone 800 implementing device 100' can display slider 805. In part (a), based on drag input of dragging slider 805 to the leftmost position, smartphone 800 can display a display including the most similar object to the first object ( Figure 8C 3D objects of OA Figure 8C of Image 810.

[0139] In part (b), based on drag input by dragging slider 805 to the middle position, smartphone 800 may display objects including a first object and a second object. Figure 8C OB) both 3D objects Figure 8C of Image 815.

[0140] In section (c), based on drag input by dragging slider 805 to the far right position, smartphone 800 can display a 3D object including the one most similar to the second object. Figure 8C of Image 820.

[0141] Figure 9 yes Figure 2B The diagram shows the usage of system 200, in which robot 900 implements client device 205B included in system 200.

[0142] like Figure 9As shown in section (a), a robot 900 implementing client device 205B is near multiple objects 905, including a shoe 910, and wants to pick up the shoe 910 with its arm and hand. The robot 900 includes a camera that acquires a 2D image 915 of the multiple objects 905, in which the shoe 910 is partially blocked by at least one of the multiple objects 905.

[0143] To accurately pick up shoe 910, robot 900 may need a 3D model of shoe 910 that is not partially obstructed by any of the multiple objects 905. Using such a 3D model, robot 900 can rotate the 3D model to learn the optimal position for picking up shoe 910. Therefore, robot 900 sends the obtained 2D image to... Figure 2B The server device 210 described herein.

[0144] In part (b), based on the transmitted 2D image, robot 900 receives mapping function parameters from server device 210 and uses the obtained mapping function parameters to set up the neural network included in robot 900. Figure 2B The second neural network 115) has a mapping function. Based on the mapping function of the set neural network, the robot obtains a 3D object 910' corresponding to the shoe 910 in view 920. This 3D object 910' can be used to accurately pick up the shoe 910 among multiple objects 905, as shown in part (b).

[0145] To properly obtain the mapping function parameters and the 3D object 910', the robot 900 may also include a depth sensor contained in an infrared or RGB-D camera, which estimates the true size of the shoe 910 based on the acquired 2D image. The robot 900 can send the estimated size of the shoe 910 along with the acquired 2D image to the server device 210. The server device 210 can adjust the acquired 2D image to a fixed, predetermined scale based on the estimated size of the shoe 910, thereby obtaining the mapping function parameters appropriately and accurately.

[0146] Figure 10 yes Figure 2B The diagram shows the usage of system 200, where AR glasses 1000 implements a client device 205B included in system 200.

[0147] like Figure 10As shown in section (a), a user wearing AR glasses 1000, which implements client device 205B, may be purchasing shoes 1005 in a store, seeing shoes 1005 on a store shelf through AR glasses 1000, and wanting to learn more about shoes 1005. AR glasses 1000 includes a camera that acquires 2D images of the store and shoes 1005. To learn more about shoes 1005, AR glasses 1000 sends the acquired 2D images to... Figure 2B The server device 210 described herein.

[0148] In part (b), based on the transmitted 2D image, the AR glasses 1000 receives mapping function parameters from the server device 210 and uses the obtained mapping function parameters to set up the neural network included in the AR glasses 1000. Figure 2B The second neural network 115) maps to the image. Based on the mapping function of the set neural network, the AR glasses 1000 obtains and displays a 3D object 1005' corresponding to the shoe 1005 in the store's AR environment. The 3D object 1005' can be used to learn more about the shoe 1005. For example, a user wearing the AR glasses 1000 can instruct the AR glasses 1000 to rotate and examine the 3D object 1005' without interacting with the real shoe 1005.

[0149] Figure 11 This is a block diagram of an electronic device 1100 that implements a means for representing 3D objects according to an embodiment.

[0150] refer to Figure 11 Electronic device 1100 includes a memory 1105, a processor 1110, an input interface 1115, a display 1120, and a communication interface 1125. Electronic device 1100 can... Figures 1 to 2B Each of the devices 100, client devices 205A and 205B, and server device 210 shown in the figure is implemented therein.

[0151] Processor 1110 generally controls electronic device 1100. Processor 1110 executes one or more programs stored in memory 1105.

[0152] Memory 1105 stores various data, programs, or applications used to drive and control electronic device 1100. The programs stored in memory 1105 include one or more instructions. The programs (one or more instructions) or applications stored in memory 1105 can be executed by processor 1110.

[0153] Processor 1110 can execute any one or any combination of operations of device 100, client device 205A, and system 200, respectively in Figures 1 to 2B As shown and referenced Figures 1 to 2B , Figure 6 and Figure 7 These operations are described.

[0154] Input interface 1115 can receive user input and / or data (such as 2D images). Input interface 1115 may include, for example, a touch screen, a camera, a microphone, a keyboard, a mouse, or any combination thereof.

[0155] The display 1120 can obtain data from, for example, the processor 1110, and can display the obtained data. The display 1120 may include, for example, a touch screen, a television, a computer monitor, etc.

[0156] The communication interface 1125 sends data to and receives data from other electronic devices, and may include one or more components that enable communication to be performed via a local area network (LAN), a wide area network (WAN), a value-added network (VAN), a mobile radio communication network, a satellite communication network, or a combination thereof.

[0157] A block diagram of electronic device 1100 is provided as an example. Depending on the specifications of the actual implemented electronic device 1100, each component in the block diagram may be integrated, added, or omitted. That is, two or more components may be integrated into one component or a component may be divided into two or more components as needed. Furthermore, the functions performed by the various blocks are provided to illustrate embodiments of this disclosure, and the operation or device of the various blocks does not limit the scope of this disclosure.

[0158] The embodiments described above can be written as computer executable programs or instructions that can be stored in a medium.

[0159] The medium can continuously store computer-executable programs or instructions, or temporarily store computer-executable programs or instructions for execution or download. Furthermore, the medium can be any of various recording or storage media in which one or more hardware components are combined, and the medium is not limited to media directly connected to a computer system, but can be distributed across a network. Examples of media include magnetic media (such as hard disks, floppy disks, and magnetic tapes) configured to store program instructions, optical recording media (such as CD-ROMs and DVDs), magneto-optical media (such as floppy disks), and ROM, RAM, and flash memory. Other examples of media include recording and storage media managed by application stores that distribute applications, or by websites, servers, etc., that supply or distribute various other types of software.

[0160] The models associated with the aforementioned DNN or CNN can be implemented via software modules. When a DNN or CNN model is implemented via a software module (e.g., a program module including instructions), the DNN or CNN model can be stored in a computer-readable recording medium.

[0161] Furthermore, the DNN or CNN model can be a part of the aforementioned device 100 integrated as a hardware chip. For example, the DNN or CNN model can be manufactured as a dedicated hardware chip for AI, or it can be manufactured as a part of an existing general-purpose processor (e.g., a CPU or application processor) or a graphics-specific processor (e.g., a GPU).

[0162] Furthermore, DNN or CNN models can be provided as downloadable software. Computer program products can include products in the form of software programs distributed electronically through manufacturers or electronic marketplaces (e.g., downloadable applications). For electronic distribution, at least a portion of the software program can be stored in a storage medium or can be temporarily generated. In this case, the storage medium can be the storage medium of a manufacturer's or electronic marketplace's server, or a relay server.

[0163] Although embodiments of the present disclosure have been described with reference to the accompanying drawings, those skilled in the art will understand that various changes in form and detail may be made to the embodiments of the present disclosure without departing from the spirit and scope defined by the appended claims.

Claims

1. An apparatus for representing a three-dimensional 3D object, the apparatus comprising: Memory that stores instructions; as well as The processor is configured to execute the instructions to: Send 2D images to external devices; Based on the transmission of the 2D image, the mapping function parameters obtained using the first neural network are received from the external device. Based on the received mapping function parameters, set the mapping function of the second neural network; 3D samples are obtained using a second neural network whose mapping function is set. Based on the 3D samples, a second neural network with its mapping function set is used to obtain a 3D object corresponding to the 2D image; and The 3D object is represented as the weights and biases of the second neural network.

2. The apparatus according to claim 1, wherein, The 3D sample is the surface or interior of a 3D standard domain.

3. The apparatus according to claim 2, wherein, The 3D standard domain is a unit sphere or a unit cube.

4. The apparatus according to claim 1, wherein, The 3D object is the surface or interior of an object included in the 2D image.

5. The apparatus according to claim 1, wherein, The first neural network is a convolutional neural network (CNN) and is trained in combination with a second neural network. The CNN is trained to output mapping function parameters based on a 2D image as input to the CNN.

6. The apparatus according to claim 1, wherein, The second neural network is a convolutional neural network (CNN) trained in conjunction with the first neural network. The CNN is trained to output 3D objects based on 3D samples input into the CNN.

7. The apparatus according to claim 1, wherein, The device is a client device that is separate from and located outside the external device.

8. The apparatus according to claim 1, wherein, The external device is a server device that is separate from the device and located outside the device.

9. A method for representing a three-dimensional 3D object, the method being performed by a device, and the method comprising: Send 2D images to external devices; Based on the transmission of the 2D image, the mapping function parameters obtained using the first neural network are received from the external device. Based on the received mapping function parameters, set the mapping function of the second neural network; 3D samples are obtained using a second neural network whose mapping function is set. Based on the 3D samples, a second neural network with its mapping function set is used to obtain a 3D object corresponding to the 2D image; and The 3D object is represented as the weights and biases of the second neural network.

10. The method according to claim 9, wherein, The 3D sample is the surface or interior of a 3D standard domain.

11. The method according to claim 10, wherein, The 3D standard domain is a unit sphere or a unit cube.

12. The method according to claim 9, wherein, The 3D object is the surface or interior of an object included in a 2D image.

13. The method according to claim 9, wherein, The first neural network is a convolutional neural network (CNN) and is trained in combination with a second neural network. The CNN is trained to output mapping function parameters based on a 2D image as input to the CNN.

14. The method according to claim 9, wherein, The second neural network is a convolutional neural network (CNN) trained in conjunction with the first neural network. The CNN is trained to output 3D objects based on 3D samples input into the CNN.

15. A computer-readable storage medium storing instructions, configured to cause a processor to perform the method according to any one of claims 9 to 14.

Images