Method, computer program and computing device for synthesizing novel view image of 3D image data
The use of a Cauchy distribution in generating images from three-dimensional data enhances image quality and accuracy by optimizing parameter representation, addressing limitations in existing Gaussian-based methods.
Patent Information
- Application Number
- KR1020240075850
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2024-06-11
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2044-06-11
AI Technical Summary
Existing methods for generating images from three-dimensional image data at a new viewpoint, such as Novel View Synthesis (NVS), face challenges in achieving high image quality and accuracy, particularly when using Gaussian distributions for parameter representation.
A method utilizing a Cauchy distribution for initializing and learning parameters through two-dimensional projection and tile raster operations, which includes extracting initial three-dimensional data, initializing a three-dimensional reference distribution, and optimizing parameters based on a correct answer image.
Improves image quality and accuracy by reducing the number of distributions required, allowing for high-quality image rendering with less capacity, as demonstrated by improved PSNR, SSIM, and LPIPS metrics.
Smart Images

Figure 112024062959595-PAT00005_ABST
Abstract
Description
Technology Field
[0001] The present disclosure relates to a method, a computer program, and a computing device for generating an image at a new viewpoint from three-dimensional image data for acquiring an image taken from three-dimensional image data at a new viewpoint.
[0002] delete Background Technology
[0003] Novel View Synthesis (NVS) technology refers to a technique that generates images of a single scene viewed from various perspectives. It is used to generate a scene from a specific viewpoint when that specific view is required but only images captured from other perspectives are available. In other words, the goal is to predict and render a scene from a new viewpoint using a limited number of input images or video frames. NVS technology has evolved by integrating research across various fields, such as computer vision, graphics, and artificial intelligence, and has recently become an important technology required in diverse application areas, including virtual reality (VR), augmented reality (AR), and film and game production.
[0004] Various technologies are being developed to generate images from multiple viewpoints. Multi-View Geometry (MVS) generates a new viewpoint by utilizing the geometric relationships between images taken from multiple angles. This utilizes Structure from Motion (SfM) technology, which uses multiple 2D images captured by a camera to simultaneously estimate the 3D structure of the scene and the camera position.
[0005] There is also a technique that reconstructs 3D structures by estimating depth information of each pixel in an input image using a neural network model. Additionally, techniques such as Neural Radiance Fields (NeRF), which learn the distribution of rays within 3D space based on multiple input images to generate new viewpoints, are also being researched.
[0006] delete Prior art literature
[65535] Jaewoo Jung et al, Relaxing Accurate Initialization Constraint for 3D Gaussian Splatting(2024.03.14) The problem to be solved
[0007] The present disclosure is devised in response to the aforementioned background technology and provides a method, computer program, and computing device for generating an image at a new viewpoint from three-dimensional image data using a Cauchy distribution.
[0008] However, the problems to be solved in this disclosure are not limited to those mentioned above, and other unmentioned problems may be clearly understood based on the description below. means of solving the problem
[0010] According to one embodiment of the present disclosure for realizing the aforementioned problem, a method for generating an image at a new viewpoint from three-dimensional image data is disclosed, which is performed by a computing device comprising at least one processor. The method comprises the steps of: extracting initial three-dimensional data including three-dimensional positions of points constituting the two-dimensional image and camera information that captured the two-dimensional image from training data including a two-dimensional image; initializing a three-dimensional reference distribution based on the initial three-dimensional data; generating a first image from the initialized reference distribution through two-dimensional projection and tile raster operations; and learning parameters constituting the reference distribution based on the first image and a correct answer image included in the training data.
[0011] Alternatively, the above reference distribution may include a three-dimensional Cauchy distribution.
[0012] Alternatively, it may further include a step of generating a second image for a new time point using a learned reference distribution.
[0013] Alternatively, the step of generating the first image may include the step of transforming the parameters of the initialized reference distribution through a two-dimensional projection and the step of generating the distribution generated as a result of the two-dimensional projection as the first image.
[0014] According to one embodiment of the present disclosure for realizing the aforementioned tasks, a computer program stored on a computer-readable storage medium is configured such that, when executed on one or more processors, the computer program performs operations to generate an image at a new viewpoint from three-dimensional image data. The operations include: an operation to extract initial three-dimensional data including the three-dimensional position of a point constituting the two-dimensional image and camera information that captured the two-dimensional image from training data containing a two-dimensional image; an operation to initialize a three-dimensional reference distribution based on the initial three-dimensional data; an operation to generate a first image from the initialized reference distribution through two-dimensional projection and tile raster operations; and an operation to train parameters constituting the reference distribution based on the first image and a correct answer image included in the training data.
[0015] According to one embodiment of the present disclosure for realizing the aforementioned task, a computing device for generating an image at a new viewpoint from three-dimensional image data is disclosed. The device comprises a processor including at least one core and a memory including program codes executable on the processor. The processor extracts initial three-dimensional data including the three-dimensional position of a point constituting the two-dimensional image and camera information that captured the two-dimensional image from training data including a two-dimensional image, initializes a three-dimensional reference distribution based on the initial three-dimensional data, generates a first image from the initialized reference distribution through two-dimensional projection and tile raster operations, and learns parameters constituting the reference distribution based on the first image and a correct answer image included in the training data. Effects of the invention
[0017] According to the present disclosure, compared to the NVS method of generating an image at a new viewpoint from existing 3D image data, the image quality is better and the accuracy of restoration can be improved.
[0018] In addition, the Cauchy distribution function used in the present disclosure has a shape with thicker tails compared to the Gaussian distribution, and as a result, when applying gradient descent, the slope changes, reducing the number of distributions required to represent the same data, so that image rendering of a similar level is possible with less capacity. Brief explanation of the drawing
[0020] FIG. 1 is a block diagram of a computing device according to one embodiment of the present disclosure. FIG. 2 is an exemplary diagram illustrating the operation of a computing device according to one embodiment of the present disclosure to generate an image at a new point in time. FIG. 3 is a flowchart illustrating the operation of a computing device according to one embodiment of the present disclosure to generate an image at a new point in time. FIG. 4 is a graph showing the characteristics of a Cauchy distribution according to one embodiment of the present disclosure. Specific details for implementing the invention
[0021] Embodiments of the present disclosure are described below with reference to the attached drawings so that those skilled in the art (hereinafter, those skilled in the art) can easily implement them. The embodiments presented in the present disclosure are provided to enable those skilled in the art to use or implement the contents of the present disclosure. Accordingly, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure may be embodied in various different forms and is not limited to the embodiments below.
[0022] Throughout the specification of the present disclosure, identical or similar reference numerals refer to identical or similar components. Additionally, to clearly explain the present disclosure, reference numerals in the drawings that are unrelated to the description of the present disclosure may be omitted.
[0023] The term “or” as used in this disclosure is intended to mean an implicit “or” rather than an exclusive “or.” That is, unless otherwise specified in this disclosure or its meaning is not clear from the context, “X uses A or B” should be understood to mean one of the natural implicit substitutions. For example, unless otherwise specified in this disclosure or its meaning is not clear from the context, “X uses A or B” may be interpreted as X using A, X using B, or X using both A and B.
[0024] The term “at least one of A or B” as used in the present disclosure should be interpreted as referring to A, B, and combinations of A and B.
[0025] The term “and / or” as used in this disclosure should be understood to refer to and include all possible combinations of one or more of the enumerated related concepts.
[0026] The terms “comprising” and / or “comprising” as used in this disclosure should be understood to mean the presence of certain features and / or components. However, the terms “comprising” and / or “comprising” should be understood not to exclude the presence or addition of one or more other features, other components and / or combinations thereof.
[0027] Where not otherwise specified in the present disclosure or where it is not clear from the context that the singular form indicates, the singular should generally be interpreted as including “one or more.”
[0028] The term “the N (N is a natural number)” used in this disclosure may be understood as an expression used to distinguish the components of this disclosure from one another according to certain criteria, such as functional perspectives, structural perspectives, or convenience of explanation. For example, components performing different functional roles in this disclosure may be distinguished as a first component or a second component. However, components that are substantially identical within the technical scope of this disclosure but must be distinguished for the convenience of explanation may also be distinguished as a first component or a second component.
[0029] Meanwhile, the terms "module" or "unit" used in this disclosure may be understood as referring to an independent functional unit that processes computing resources, such as a computer-related entity, firmware, software or a part thereof, hardware or a part thereof, or a combination of software and hardware. In this case, "module" or "unit" may be a unit composed of a single element, or a unit expressed as a combination or set of multiple elements. For example, in a narrow sense, "module" or "unit" may refer to a hardware element of a computing device or a set thereof, an application program that performs a specific function of software, a procedure implemented through software execution, or a set of instructions for program execution. Furthermore, in a broad sense, "module" or "unit" may refer to the computing device itself that constitutes the system, or an application executed on the computing device. However, since the above-described concept is merely an example, the concept of "module" or "part" may be defined in various ways within the scope understandable to those skilled in the art based on the contents of this disclosure.
[0030] As used in this disclosure, the term "model" may be understood as a system implemented using mathematical concepts and language to solve a specific problem, a set of software units to solve a specific problem, or an abstract model regarding a processing process to solve a specific problem. For example, a neural network "model" may refer to an overall system implemented as a neural network that possesses problem-solving capabilities through learning. In this case, the neural network may possess problem-solving capabilities by optimizing parameters connecting nodes or neurons through learning. A neural network "model" may include a single neural network or a set of neural networks composed of multiple neural networks.
[0031] The explanation of the foregoing terms is intended to aid in understanding the present disclosure. Accordingly, it should be noted that unless a foregoing term is explicitly stated as a matter limiting the content of the present disclosure, it is not to be used in the sense of limiting the technical concept of the content of the present disclosure.
[0032] FIG. 1 is a block diagram of a computing device according to one embodiment of the present disclosure.
[0033] A computing device (100) according to one embodiment of the present disclosure may be a hardware device or part of a hardware device that performs comprehensive processing and computation of data, or it may be a software-based computing environment connected to a communication network. For example, the computing device (100) may be a server that performs intensive data processing functions and is an entity that shares resources, or it may be a client that shares resources through interaction with the server. Additionally, the computing device (100) may be a cloud system that enables multiple servers and clients to interact to comprehensively process data. Since the above description is merely one example regarding the type of computing device (100), the type of computing device (100) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.
[0034] Referring to FIG. 1, a computing device (100) according to one embodiment of the present disclosure may include a processor (110), a memory (120), and a network unit (130). However, since FIG. 1 is merely an example, the computing device (100) may include other configurations for implementing a computing environment. Additionally, only some of the disclosed configurations may be included in the computing device (100).
[0035] A processor (110) according to one embodiment of the present disclosure may be understood as a constituent unit comprising hardware and / or software for performing computing operations. For example, the processor (110) may read a computer program and perform data processing for machine learning. The processor (110) may process computational processes such as processing input data for machine learning, extracting features for machine learning, and calculating errors based on backpropagation. A processor (110) for performing such data processing may include a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). Since the above-described type of processor (110) is merely an example, the type of processor (110) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.
[0036] The processor (110) can perform a series of operations to generate an image at a new viewpoint from three-dimensional image data.
[0037] The processor (110) can extract initial 3D data including the 3D position of a point constituting a 2D image and camera information that captured the 2D image from training data containing a 2D image. In this process, the processor (110) can use Structure from Motion (SfM). That is, the processor (110) can extract feature points from a 2D image, match feature points between multiple 2D images, and estimate the position of the camera that captured the 2D image based on the matched feature points. Then, based on the estimated camera position, the position of the feature point in 3D space can be calculated. By repeating the above-described methods, the accuracy of the camera position and the 3D position of the point constituting the 2D image can be improved.
[0038] The processor (110) can initialize a 3D reference distribution based on initial 3D data. In this case, the reference distribution may refer to a 3D Cauchy distribution. The processor (110) can generate a first image from the initialized reference distribution through 2D projection and tile raster operations. A tile raster operation is a process of generating an image from a distribution in 3D.
[0039] The processor (110) can perform a two-dimensional projection to reduce the dimensionality and convert a 3D Cauchy distribution into a two-dimensional image. The two-dimensional projection can be determined according to the camera direction. The processor (110) can also convert the parameters of the Cauchy distribution from three dimensions to two dimensions.
[0040] And the processor (110) can perform tile raster operations. Tile raster operations refer to a method of dividing an image into small tiles and performing operations in parallel for each tile. This makes GPU parallel processing easier.
[0041] The processor (110) can learn parameters that constitute a reference distribution based on the first image and the correct image included in the training data. In this process, the processor (110) defines the difference between the first image and the correct image as a loss function and can minimize the loss function by optimizing each parameter through differentiable operations.
[0042] The processor (110) can generate an image at a new viewpoint (novel view) using a learned reference distribution.
[0043] A memory (120) according to one embodiment of the present disclosure may be understood as a configuration unit comprising hardware and / or software for storing and managing data processed by a computing device (100). That is, the memory (120) may store data of any form generated or determined by a processor (110) and data of any form received by a network unit (130). For example, the memory (120) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, RAM (random access memory), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory, a magnetic disk, or an optical disk. Additionally, the memory (120) may include a database system that controls and manages data in a predetermined system. Since the above-described type of memory (120) is merely an example, the type of memory (120) can be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.
[0044] The memory (120) can store information necessary for the processor (110) to perform operations and information generated according to the operations of the processor (110). The memory (120) can store initial three-dimensional data extracted by the processor (110), reference distribution, for example, information on the initial state of the Cauchy distribution, parameters of the Cauchy distribution calculated during the learning process, etc.
[0045] A network unit (130) according to one embodiment of the present disclosure may be understood as a configuration unit that transmits and receives data through any known form of wired or wireless communication system. For example, the network unit (130) may perform data transmission and reception using wired or wireless communication systems such as a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), wireless broadband internet (WiBro), 5th generation mobile communication (5G), ultra-wide-band wireless communication, ZigBee, radio frequency (RF) communication, wireless LAN, wireless fidelity (Wi-Fi), near field communication (NFC), or Bluetooth. Since the communication systems described above are merely examples, wired or wireless communication systems for data transmission and reception of the network unit (130) may be applied in various ways other than those described above.
[0046] FIG. 2 is an exemplary diagram showing the operation of a computing device according to one embodiment of the present disclosure generating an image at a new point in time, FIG. 3 is a flowchart showing the operation of a computing device according to one embodiment of the present disclosure generating an image at a new point in time, and FIG. 4 is a graph showing the characteristics of a Cauchy distribution according to one embodiment of the present disclosure.
[0047] Referring to FIG. 2, the computing device (100) can optimize parameters representing a reference distribution. In this case, the solid line in FIG. 2 may represent the operation flow, and the dotted line may represent the gradient flow.
[0048] Referring to FIG. 3, the computing device (100) can extract initial 3D data including the 3D position of a point constituting a 2D image and camera information that captured the 2D image from training data including a 2D image (S110). In FIG. 2, the initial 3D data may include SfM Points.
[0049] A computing device (100) can initialize a 3D reference distribution based on initial 3D data (S120). For example, the reference distribution may include a 3D Cauchy distribution such as Equation 1. The use of a Cauchy distribution in this disclosure is exemplary, and other distributions having the characteristics of a stable distribution may be used. A stable distribution refers to a distribution in which the shape of the density function of the original distribution does not change even when a linear transformation is applied. The shape of the distribution changes, but the number of parameters representing the distribution does not change. Examples of stable distributions include the Gaussian distribution and the Cauchy distribution.
[0050] Referring to FIG. 4, the present disclosure exemplarily uses a Cauchy distribution rather than a Gaussian distribution because the Cauchy distribution has a shape with thicker tails compared to the Gaussian distribution. As a result, when applying gradient descent, the gradient changes, and the number of distributions required to represent the data is reduced, and there is an advantage that image rendering of a similar level is possible with a small amount of data.
[0051] [Mathematical Formula 1]
[0052]
[0053] In mathematical formula 1, x is the position, μ and Σ are parameters used in the representation of the Cauchy distribution, K is the number of dimensions, which corresponds to the case of three dimensions in this disclosure, so it is represented as 3, and Γ represents the gamma function.
[0054] The computing device (100) can generate a first image from a reference distribution initialized through a two-dimensional projection and a different tile rasterizer (S130).
[0055] Referring again to FIG. 2, the computing device (100) can transform the parameters of a reference distribution initialized through two-dimensional projection. That is, the reference distribution is a distribution existing in three dimensions, and since the image included in the initial three-dimensional data is two-dimensional data, a reduction in dimensionality occurs through the projection process. The computing device (100) performs an operation to generate an image of a new viewpoint according to the direction set by the camera in FIG. 2. That is, projection can be determined according to the direction information of the camera in FIG. 2. Through projection, the parameters of the reference distribution can be transformed from three dimensions to two dimensions, which can be expressed as Equation 2.
[0056] [Mathematical Formula 2]
[0057]
[0058] In mathematical equation 2, Σ represents a 3x3 matrix representing parameters in the 3-dimensional coordinate system before projection, and Σ' represents a 2x2 matrix representing parameters in the 2-dimensional system after projection.
[0059] W represents a linear transformation, which converts the distribution into an image by transforming it from a fixed (world) coordinate system to a coordinate system where the camera is the origin. In this case, the dimension remains 3 dimensions even after undergoing the linear transformation W. J is a transformation that converts the distribution from a coordinate system where the camera is the origin to an image coordinate system. In this case, this transformation may be approximated as a linear transformation for computational convenience. The dimension remains 3 dimensions even after undergoing the linear transformation J. Subsequently, to project the result onto 2 dimensions after passing through J, the 3rd row and 3rd column of the 3-dimensional matrix may be deleted.
[0060] Meanwhile, a raster operation refers to the process of generating an image from a distribution in two dimensions. Since the NVS operation performed in this disclosure generates an image at a specific point in time, a raster operation must be performed. The raster operation can be calculated through differentiable operations of the parameters of the reference distribution. Specifically, the difference between the image generated through the raster operation and the ground truth image included in the training data can be set as the loss function, and each parameter can be optimized in a direction that minimizes the loss function using gradient descent.
[0061] At this time, for parallel data processing of the GPU, the computing device (100) can divide the image into multiple tiles. For example, it can be divided into tiles of size 16x16. For any one tile, the following operations can be performed. The case where there is an overlapping area between the reference distribution and the tile affects the process of generating the image. Therefore, only the distributions that overlap with the tile can be selected to generate a list of distributions. Among the parameters included in the reference distribution, the distributions can be sorted according to depth for image generation using a parameter representing transparency, and after sorting, an image can be generated using a blending algorithm.
[0062] The computing device (100) can learn parameters that constitute a reference distribution based on a first image and a correct image included in the training data (S140). Specifically, in the present disclosure, three-dimensional data is represented as a set of multiple reference distributions (e.g., Cauchy distribution), and the parameters representing each reference distribution are differentiable. The computing device (100) can learn the parameters of the Cauchy distribution through gradient descent in a direction in which the rasterized image of the Cauchy distribution matches the correct image. That is, the method of learning the reference distribution is to optimize by learning the mean and variance based on a two-dimensional image included in the initial three-dimensional data in a direction in which the two-dimensional image matches the rasterized image of the Cauchy distribution.
[0063] Additionally, the distributions used in the learning process are not created or removed after being generated during the initialization process of step S120. At this time, the computing device (100) can control the number of distributions to achieve optimal performance. This is because too few or too many distributions reduce the learning speed. Therefore, the computing device (100) can delete some distributions if too many distributions are generated through the Adaptive Density Control operation of FIG. 2, and increase the number of distributions by dividing the existing distributions into two if there are too few distributions.
[0064] The computing device (100) can generate a second image for a new time point using a reference distribution that has been learned.
[0065] According to the present disclosure, the performance of an operation to generate images at a new viewpoint using a Cauchy distribution as the reference distribution was evaluated. According to Table 1, it can be seen that the Peak Signal-to-noise-ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) metrics are superior compared to the case where a Gaussian distribution is used as the reference distribution.
[0066] PSNR SSIM LPIPS Capacity (Mb) Gaussian 26.2735 0.8715 0.21275 547 Cauchy 26.5735 0.871 0.20975 528.75
[0068] The image generation method of the present disclosure can be utilized in autonomous vehicles using 3D data, medical image analysis, and game industries using virtual reality or augmented reality.
[0069] The various embodiments of the present disclosure described above may be combined with additional embodiments and modified to the extent understandable to those skilled in the art in light of the detailed description above. The embodiments of the present disclosure are illustrative in all respects and should be understood as not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form. Accordingly, all modifications or variations derived from the meaning, scope, and equivalents of the claims of the present disclosure should be interpreted as being included within the scope of the present disclosure.
Claims
Claim 1 A method for generating an image at a new viewpoint from three-dimensional image data, performed by a computing device comprising at least one processor, comprising: a step of extracting initial three-dimensional data including three-dimensional positions of points constituting the two-dimensional image and camera information that captured the two-dimensional image from training data including a two-dimensional image; a step of initializing a three-dimensional reference distribution based on the initial three-dimensional data; a step of generating a first image from the initialized reference distribution through two-dimensional projection and tile raster operations; and a step of learning parameters constituting the reference distribution based on the first image and a ground truth image included in the training data; wherein the reference distribution comprises a three-dimensional Cauchy distribution. Claim 2 delete Claim 3 A method according to claim 1, further comprising the step of generating a second image for a new time point using a reference distribution that has been trained. Claim 4 A method according to claim 1, wherein the step of generating the first image comprises: a step of transforming the parameters of the initialized reference distribution through a two-dimensional projection; and a step of generating the distribution generated as a result of the two-dimensional projection as the first image. Claim 5 A computer program stored on a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs operations to generate an image at a new viewpoint from three-dimensional image data, the operations comprising: an operation to extract initial three-dimensional data including the three-dimensional position of a point constituting the two-dimensional image and camera information that captured the two-dimensional image from training data including a two-dimensional image; an operation to initialize a three-dimensional reference distribution based on the initial three-dimensional data; an operation to generate a first image from the initialized reference distribution through two-dimensional projection and tile raster operations; and an operation to learn parameters constituting the reference distribution based on the first image and a correct answer image included in the training data, wherein the reference distribution comprises a three-dimensional Cauchy distribution. Claim 6 A computing device for generating an image at a new viewpoint from three-dimensional image data, comprising: a processor including at least one core; and a memory including program codes executable on the processor; wherein the processor extracts initial three-dimensional data including the three-dimensional position of a point constituting the two-dimensional image and camera information that captured the two-dimensional image from training data including a two-dimensional image, initializes a three-dimensional reference distribution based on the initial three-dimensional data, generates a first image from the initialized reference distribution through two-dimensional projection and tile raster operations, and learns parameters constituting the reference distribution based on the first image and a ground truth image included in the training data, and wherein the reference distribution includes a three-dimensional Cauchy distribution.
Citation Information
Patent Citations
Method and system for defining a virtual reality resolution distribution
US20170316549A1
