Learning device, server device and program
By constraining the density at background points and virtual viewpoints in the machine learning model, the method addresses background interference issues in photogrammetry, enhancing accuracy and stability in three-dimensional shape estimation.
Patent Information
- Application Number
- JP2023087190
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-26
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2043-05-26
AI Technical Summary
Conventional photogrammetry methods using artificial neural networks require background removal or masking during training due to background interference, which is cumbersome and affects accuracy.
A machine learning model is trained with constraints that fix the density at background points and virtual viewpoints to predetermined constants, eliminating the need for background removal and improving training efficiency.
This approach enhances the accuracy and stability of three-dimensional shape estimation by directly incorporating background information, reducing the complexity of preprocessing and improving model convergence.
Smart Images

Figure 2024170173000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a technique for estimating the three-dimensional shape of an object from a plurality of images of the object. [Background technology]
[0002] Photogrammetry is a technology for estimating the three-dimensional shape of an object from multiple images of the object. For example, Patent Document 1 discloses, as a photogrammetry technology, an apparatus and method for generating a three-dimensional (3D) model of a subject from images taken by multiple cameras. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent Publication No. 2021-71749 Summary of the Invention [Problem to be solved by the invention]
[0004] Rendering is the process of generating an image of an object as it would appear when viewed from a specified viewpoint, given the object's shape, material, color, the position of the light source, light intensity, etc. In contrast, photogrammetry is a process of estimating an object's three-dimensional shape, color, texture, etc. from multiple images of the object. This process is also called inverse rendering, as it is the reverse of rendering. To solve the inverse rendering problem in photogrammetry, artificial neural network (ANN) technology is sometimes applied.
[0005] Photogrammetry technology that applies ANN uses images obtained by photographing an object from multiple viewpoints as training data to train a machine learning model that includes optical assumptions. This machine learning model is composed of, for example, a multilayer perceptron (MLP). By using this trained machine learning model, it is possible to generate a three-dimensional model of an object (also called a virtual object) in a virtual space from new images of the object photographed from multiple viewpoints. Then, by using this generated three-dimensional model, it is possible to generate an image (also called a virtual image) that would be obtained if this virtual object were photographed from any viewpoint (also called a virtual viewpoint) in the virtual space.
[0006] NeRF (Neural Radiance Fields) is a method that uses MLP to represent density and brightness fields in a three-dimensional virtual space, and generates a virtual image of a virtual object captured from an arbitrary virtual viewpoint in this virtual space. NeRF generates the above-mentioned two-dimensional virtual image by volume rendering. For example, NeRF uses a "volumetric representation" in which the three-dimensional virtual space is divided into a grid, with the so-called voxel as the unit. For this reason, inverse rendering based on NeRF is said to be easier to optimize stably than methods based on curved surface representation (described later).
[0007] On the other hand, another method for generating the above-mentioned virtual images is NeuS (Neural Scene Representation). NeuS is one of the methods for generating virtual objects in a virtual space based on a curved surface representation. NeuS is a method for generating virtual objects in the form of a point cloud or mesh by expressing the surface of an object as a zero-level set of a signed distance function (SDF) approximated by MLP, and learning a model of this surface using images taken from multiple viewpoints as training data. Since NeuS expresses the surface of an object as an SDF, which is a continuous function approximated by MLP, it has an advantage in expressing the surface of an object composed of curved surfaces compared to NeRF, which uses a volumetric representation.
[0008] When learning the above machine learning model by deep learning such as NeRF and NeuS, the density at both ends of the line of sight (background point and virtual viewpoint) for viewing a virtual object was conventionally determined by chance. Therefore, in the conventional technology, if the image of the training data contains a background, the background interferes with learning, so it was necessary to remove the background with an appropriate foreground mask or to model it with NeRF. For example, the technology described in Patent Document 1 requires a binary mask image in which the object and other parts are represented in different gradations (for example, black and white) when estimating the three-dimensional shape of the object from multiple images of the object.
[0009] One objective of the present invention is to eliminate the need for a process of removing the background from each image when training a machine learning model that uses images obtained by photographing an object from multiple viewpoints as training data to estimate a three-dimensional model of the object in a virtual space. To do so. [Means for solving the problem]
[0010] The learning device according to claim 1 of the present invention comprises an acquisition means for acquiring teacher data that respectively corresponds a plurality of viewpoints to images obtained by photographing an object from each of the plurality of viewpoints, a generation means for generating a virtual image obtained when a virtual object corresponding to the object is photographed from a virtual viewpoint corresponding to the viewpoint in a virtual space, using a machine learning model including a density for each position on a line of sight that passes from a background point in the background of the virtual object through the virtual object to the virtual viewpoint, and a learning means for comparing, for each virtual viewpoint, the generated virtual image with the image photographed from the viewpoint corresponding to the virtual viewpoint, and training the machine learning model, wherein the machine learning model is characterized in that the densities at the background points and the virtual viewpoints are each constrained to be predetermined constants.
[0011] The learning device according to claim 2 of the present invention is an embodiment of the learning device described in claim 1, wherein the machine learning model includes a surface function that defines the surface of the virtual object by a multilayer perceptron and a density function that indicates the density using the surface function, the learning means trains the machine learning model by updating the surface function by a backpropagation method, and the density function is constrained so that the density at the background points is 1 and the density at the virtual viewpoint is 0.
[0012] The learning device of claim 3 of the present invention is a learning device in the aspect of claim 2, characterized in that in the machine learning model, a transmittance function T indicating the transmittance of light that travels along the line of sight to the virtual viewpoint, and an opacity function α indicating the opacity at any position on the line of sight are expressed by the following equations when a density function Φ indicating the density at any position on the line of sight, and a surface function f defining the surface of the virtual object are used. JPEG2024170173000002.jpg43157
[0013] The server device according to claim 4 of the present invention is a server device having: a captured image acquisition means for acquiring captured images of an object from a plurality of shooting points; an estimation means for estimating the shooting points based on feature points extracted from each of the captured images; and a three-dimensional data generation means for generating three-dimensional data of the object using the estimated shooting points, the captured images corresponding to the shooting points, and a machine learning model according to any one of claims 1 to 3.
[0014] The server device according to claim 5 of the present invention is a server device in the aspect of claim 4, characterized in that it has a receiving means for receiving a request from a terminal, and a supplying means for supplying the stereoscopic data generated by the stereoscopic data generating means in response to the request to the terminal.
[0015] The server device according to claim 6 of the present invention is a server device in the aspect described in claim 4, characterized in that it comprises a receiving means for receiving a designation of a virtual viewpoint from a terminal for photographing the object, a virtual image generating means for generating a virtual image obtained when the object represented by the three-dimensional data generated by the three-dimensional data generating means is photographed from the virtual viewpoint indicated by the designation, and a supplying means for supplying the virtual image generated by the virtual image generating means in response to the designation to the terminal.
[0016] The program according to claim 7 of the present invention causes a computer to function as: an acquisition means for acquiring teacher data that associates a plurality of viewpoints with images obtained by photographing an object from each of the plurality of viewpoints; a generation means for generating a virtual image obtained when a virtual object corresponding to the object is photographed in a virtual space from a virtual viewpoint corresponding to the viewpoint, using a machine learning model including a density for each position on the line of sight from a background point in the background of the virtual object through the virtual object to the virtual viewpoint; and a learning means for comparing, for each virtual viewpoint, the generated virtual image with the image photographed from the viewpoint corresponding to the virtual viewpoint, and training the machine learning model, and constraining the machine learning model so that the densities at the background points and the virtual viewpoints are each predetermined constants.
[0017] The program according to claim 8 of the present invention is a program that causes a computer to function as: an image acquisition means that acquires images of an object taken from a plurality of shooting points; an estimation means that estimates the shooting points based on feature points extracted from each of the captured images; and a three-dimensional data generation means that generates three-dimensional data of the object using the estimated shooting points, the captured images corresponding to the shooting points, and a machine learning model trained by the program described in claim 7. [Brief description of the drawings]
[0018] [Figure 1]1 is a diagram showing an example of the overall configuration of a virtual image generating system 9 according to an embodiment of the present invention. [Diagram 2] FIG. 2 shows an example of the configuration of a learning device 1 and a server device 2. [Diagram 3] FIG. 4 is a diagram showing an example of a viewpoint image DB 120. [Figure 4] 3 is a diagram for explaining images stored in a viewpoint image DB 120. FIG. [Diagram 5] FIG. 2 is a diagram showing an example of the functional configuration of the learning device 1. [Figure 6] A diagram to explain the SDF used in machine learning model 121. [Figure 7] A diagram to explain that a background mask is not required for the training data used in the machine learning model 121. [Figure 8] FIG. 2 is a diagram showing an example of a functional configuration of a server device 2. [Figure 9] FIG. 13 is a diagram showing an example of how a virtual image is generated from three-dimensional data. [Figure 10] FIG. 13 is a diagram showing an example of a new object Jn photographed by the terminal 4. [Figure 11] 13 is a diagram showing three-dimensional data of a virtual object Jnv generated by the server device 2 using the machine learning model 221. [Figure 12] FIG. 13 is a diagram showing an example of a functional configuration of a server device 2 according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0019] <Embodiment> <Overall configuration of the virtual image generation system> FIG. 1 is a diagram showing an example of the overall configuration of a virtual image generation system 9 according to an embodiment of the present invention. The virtual image generation system 9 is a system that generates a virtual image of a virtual object corresponding to an object in a virtual space, using a machine learning model trained with teacher data that associates a plurality of viewpoints with images (photos) of the object taken from each of the viewpoints. The virtual image generation system 9 shown in FIG. 1 includes a learning device 1, a server device 2, a communication line 3, and a terminal 4. The virtual image generation system 9 may include a plurality of terminals 4, and some of these terminals 4 may be connectable to a camera 5.
[0020] The learning device 1 is a device that trains a machine learning model using the above-mentioned teacher data, and is, for example, a computer.
[0021] The server device 2 is a device that uses a copy of the machine learning model that has been trained by the learning device 1 to generate three-dimensional data of a virtual object corresponding to a new object (also called a target object) from a plurality of images of the object that are supplied from the terminal 4. By using this three-dimensional data, the server device 2 or the terminal 4 can generate a virtual image that would be obtained when the virtual object is photographed from an arbitrary virtual viewpoint in a virtual space.
[0022] The terminal 4 is a terminal device owned by a user of the virtual image generation system 9, and is, for example, a smartphone, a personal computer, etc. The terminal 4 may be provided with a photographing function for photographing an object, such as a camera. The terminal 4 may also be connected to a camera 5, such as a digital still camera or a video camera, and may be capable of acquiring images photographed by the camera 5.
[0023] The communication line 3 is a line that connects the learning device 1, the server device 2, and the terminal 4 by wire or wirelessly so that they can communicate with each other. The communication line 3 may be, for example, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, or a combination of these. The communication line 3 may also include a Public Switched Telephone Network (PSTN), an Integrated Services Digital Network (ISDN), etc.
[0024] <Learning device and server device configuration> 2 is a diagram showing an example of the configuration of the learning device 1 and the server device 2. The learning device 1 has a processor 11, a memory 12, and an interface 13. These components are connected to each other so that they can communicate with each other, for example, by a bus.
[0025] The processor 11 reads and executes a computer program (hereinafter simply referred to as a program) stored in the memory 12 to control each part of the learning device 1. The processor 11 is, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).
[0026] The interface 13 is a communication circuit that communicatively connects the learning device 1 to the communication line 3 via wire or wirelessly.
[0027] The memory 12 is a storage means for storing an operating system, various programs, data, and the like, which are loaded into the processor 11. The memory 12 includes a random access memory (RAM) or a read only memory (ROM). The memory 12 may include a solid state drive, a hard disk drive, or the like. The memory 12 also stores a viewpoint image DB 120 and a machine learning model 121.
[0028] The viewpoint image DB 120 is a database that stores a plurality of viewpoints and images obtained by photographing an object from each of the plurality of viewpoints in association with each other. The images stored in the viewpoint image DB 120 are data obtained by actually photographing an object, and are also called GT (Ground Truth) data. The machine learning model 121 is a machine learning model that is trained by using pairs of viewpoints and images stored in the viewpoint image DB 120 as teacher data. This machine learning model 121 is used to infer three-dimensional data of the object photographed in each of the above-mentioned images.
[0029] The server device 2 includes a processor 21, a memory 22, and an interface 23. The processor 21 reads and executes a program stored in the memory 22 to control each unit of the server device 2. The processor 21 is, for example, a CPU or a GPU.
[0030] The interface 23 is a communication circuit that communicatively connects the server device 2 to the communication line 3 via wire or wirelessly.
[0031] The memory 22 is a storage means for storing an operating system, various programs, data, and the like, which are loaded into the processor 21. The memory 22 includes a RAM or a ROM. The memory 22 may include a solid state drive, a hard disk drive, and the like. The memory 22 also stores a machine learning model 221 and a captured image DB 222.
[0032] The machine learning model 221 is a copy of the machine learning model 121 that has been generated and trained in the learning device 1. The server device 2 obtains a copy of the trained machine learning model 121 from the learning device 1 and stores it in the memory 22.
[0033] The captured image DB 222 is a database in which the server device 2 stores a plurality of captured images acquired from the terminal 4. The captured images stored in the captured image DB 222 are pictures of an object existing in real space captured from a plurality of capture points, etc. Unlike the viewpoint image DB 120 described above, the captured images do not need to be associated with the capture points at which the object is captured.
[0034] Both the study device 1 and the server device 2 may have an operation unit and a display unit. The operation unit is configured to receive an operation and send a signal corresponding to the operation content to the processor 11. This operation unit may include, for example, an operation button, a keyboard, a touch panel, a mouse, or other operator for issuing various instructions.
[0035] This display unit is configured to display images under the control of processor 11. This display unit may have a display screen such as a liquid crystal display. In addition, a transparent touch panel of the above-mentioned operation unit may be placed on top of this display screen. Note that study device 1 and server device 2 may be operated from an external device via interface 13 and interface 23, respectively, or may present information to an external device.
[0036] <Configuration of viewpoint image DB> Fig. 3 is a diagram showing an example of the viewpoint image DB 120. The viewpoint image DB 120 shown in Fig. 3 stores the items of ID, viewpoint, and image in association with each other. The item "ID" in the viewpoint image DB 120 is identification information for identifying an image.
[0037] The "viewpoint" item in the viewpoint image DB 120 is information indicating the viewpoint from which the image was captured. This information includes, for example, the coordinates of the viewpoint when the center point of the object to be captured is set as the origin. This information may also include information on the viewing angle at the viewpoint.
[0038] The "image" item in the viewpoint image DB 120 is identified by the above-mentioned ID, and is image data obtained when an object is photographed from the above-mentioned viewpoint.
[0039] Fig. 4 is a diagram for explaining images stored in the viewpoint image DB 120. A user of the virtual image generation system 9 uses, for example, a camera mounted on the terminal 4 to capture an object J (a car in the example of Fig. 4) existing in the real space Sp from various viewpoints P1, P2, .... The positions of the object J and the viewpoints P1, P2, ... in the real space Sp shown in Fig. 4 are expressed, for example, in an xyz right-handed coordinate space.
[0040] At this time, the terminal 4 identifies the positions of the viewpoints P1, P2, etc. The viewpoints may be identified using a distance measuring device such as LiDAR, or may be identified by triangulating markers or the like that are set in advance in the shooting space using a stereo camera. Alternatively, the camera itself that captures the image of the object J may be attached to an arm that drives along a predetermined trajectory, and the terminal 4 may identify the above viewpoints by acquiring drive information of the arm.
[0041] Then, the terminal 4 associates the captured images G1, G2... with the coordinates of the viewpoints P1, P2... at which each of these images was captured, and supplies this data to the learning device 1 via the communication line 3. When the learning device 1 acquires images of the object J captured from multiple viewpoints by the terminal 4 together with coordinate information of each of the viewpoints, the learning device 1 stores these in the viewpoint image DB 120. The processor 11 of the learning device 1 trains the machine learning model 121 using the contents stored in the viewpoint image DB 120 as teacher data. The terminal 4 may acquire the above-mentioned multiple images G1, G2... from an externally connected camera 5 and supply them to the learning device 1.
[0042] <Functional configuration of the learning device> Fig. 5 is a diagram showing an example of the functional configuration of the learning device 1. Processor 11 of learning device 1 reads and executes a program stored in memory 12, thereby functioning as acquisition means 111, generation means 112, and learning means 113 shown in Fig. 5.
[0043] The acquiring means 111 reads out the above-mentioned viewpoint image DB 120 from the memory 12 and supplies it to the generating means 112. This viewpoint image DB 120 is used as teacher data by the generating means 112. Therefore, the acquiring means 111 is an example of an acquiring means for acquiring teacher data in which a plurality of viewpoints are associated with images obtained by photographing an object from each of the plurality of viewpoints.
[0044] The generation means 112 uses the initial state stored in the memory 12 or a machine learning model 121 in the middle of learning to generate a virtual image obtained when a virtual object corresponding to an object is photographed from a virtual viewpoint corresponding to a plurality of viewpoints included in the viewpoint image DB 120.
[0045] The learning means 113 compares the virtual image generated by the generation means 112 with the image stored in the viewpoint image DB 120, and updates the machine learning model 121 so as to reduce the difference. When this difference satisfies a predetermined condition, the machine learning model 121 is considered to have been trained.
[0046] A detailed example of the machine learning model 121 is as follows. The machine learning model 121 is updated by the learning means 113 referring to the viewpoint image DB 120, acquiring an image such as a photograph of an object J existing in the real space Sp and the viewpoint at the time of the image acquisition, and performing deep learning using these as teacher data. As a result of the update by learning, this machine learning model 121 enables the generation of three-dimensional data such as the shape, texture, color, etc. of a virtual object in a virtual space corresponding to the object J. The generated three-dimensional data is used to generate a virtual image obtained when a virtual object is photographed from an arbitrary virtual viewpoint in the virtual space.
[0047] Moreover, this machine learning model 121 employs a physical model that expresses the surface of a virtual object by the above-mentioned SDF. FIG. 6 is a diagram for explaining the SDF employed in the machine learning model 121.
[0048] 6, the surface of a virtual object in the virtual space Sv is itself represented by a surface function f. This surface function f is represented by, for example, a group of weight parameters for each layer in the above-mentioned MLP.
[0049] Furthermore, the physical laws that light follows in the virtual space Sv are modeled using a transmittance function T, an opacity function α, and a density function Φ. The density function Φ is a function that indicates density at an arbitrary position in the virtual space, and is calculated using a surface function f that represents the surface of the virtual object at that position. The density function Φ is, for example, a sigmoid function, but is not limited to this as long as it is a function whose gradient is maximized when the surface function f is 0. For example, the density function Φ may be an erf function (cumulative distribution function of a Gaussian distribution) or a cumulative distribution function of a Laplace distribution.
[0050] Therefore, the machine learning model 121 described here is an example of a machine learning model that includes a surface function that defines the surface of a virtual object using a multi-layer perceptron, and a density function that indicates density using this surface function.
[0051] The line of sight R shown in Fig. 6 represents a ray of light traveling toward a virtual viewpoint P. The line of sight R is expressed by a three-dimensional vector function p(t) that expresses the optical path of the ray of light with a parameter t. The range of p(t) is expressed as a three-dimensional vector, for example, by an XYZ right-handed coordinate system that represents the virtual space Sv.
[0052] 6 is the point where the line of sight R starts. Point p(t0) is called a "background point" because it is a point that exists behind (on the background side of) the virtual object when viewed from the virtual viewpoint P. The background point is determined, for example, by its distance from the viewpoint P.
[0053] 6 is an expression of the virtual viewpoint P in the three-dimensional vector function p(t). Note that the point p(tm) is a point where the line of sight R intersects with a surface where an image is formed (called an imaging surface) on the way to the virtual viewpoint P. The light on this imaging surface forms an image observed at the virtual viewpoint P.
[0054] The generation means 112 applies a machine learning model 121 in an initial state or in the middle of learning to each of a plurality of virtual viewpoints P corresponding to each viewpoint in the viewpoint image DB 120 used as training data, calculates the density, light transmittance, and opacity for each position on the line of sight R where light passes from a background point through a virtual object to the virtual viewpoint P, and thereby generates a virtual image to be formed on the imaging surface.
[0055] Therefore, this generation means 112 is an example of a generation means that generates a virtual image obtained when a virtual object corresponding to an object in real space is photographed in a virtual space from a virtual viewpoint corresponding to a plurality of viewpoints in the real space, using a machine learning model that includes a density for each position on the line of sight from a background point in the background of the virtual object through the virtual object to the virtual viewpoint.
[0056] The learning means 113 calculates an objective function according to the difference between the image stored in the viewpoint image DB 120 and the virtual image generated by the generation means 112. Then, the learning means 113 updates the machine learning model 121 using an error propagation method or the like so that the objective function becomes, for example, a minimum value. In this way, the learning means 113 advances deep learning of the machine learning model 121, and as a result, the trained machine learning model 121 is completed.
[0057] In other words, the learning means 113 is an example of a learning means that, for each virtual viewpoint, compares a generated virtual image with an image captured from a viewpoint corresponding to the virtual viewpoint, and trains a machine learning model.
[0058] Moreover, the learning means 113 is an example of a learning means that updates the surface function by the backpropagation method, thereby learning a machine learning model.
[0059] Here, the machine learning model 121 according to the present invention is constrained so that the above-mentioned background points and the density at the virtual viewpoint are each a predetermined constant. That is, this machine learning model 121 is an example of a machine learning model in which the background points and the density at the virtual viewpoint are each constrained so that they are each a predetermined constant.
[0060] For example, this machine learning model 121 constrains the density function Φ so that the density at the background points is 1 and the density at the virtual viewpoint P is 0. Therefore, the density at the background points at a certain distance from the virtual viewpoint P through the virtual object is always 1. As a result, this machine learning model 121 ignores objects that are farther away than the background points as viewed from the virtual viewpoint P, so that the background that is farther away than the object J in the image used for the training data does not contribute to the calculation. In addition, since the density is always 0 at the virtual viewpoint P, the surface of the virtual object is always defined on the way until the light traveling along the line of sight R reaches the virtual viewpoint P.
[0061] Thus, this density function Φ included in the machine learning model 121 is an example of a density function that is constrained to have a density of 1 at the background points and a density of 0 at the virtual viewpoint.
[0062] 7 is a diagram for explaining that a background mask is not required for training data used in the machine learning model 121. The training data for training the machine learning model 121 includes an image G obtained by capturing an object J existing in a real space Sp, but this image G often contains some object (referred to as a background object B) in addition to the object J, as shown in (a) of FIG.
[0063] Among conventional physical models, for example, NeRF may sometimes represent background object B in voxels, which may result in a decrease in the accuracy of the representation of object J. Furthermore, in conventional NeuS, in order to calculate the surface shape of only object J, it is necessary to exclude background object B that appears in the captured image from the training data. For this reason, conventional NeuS required the preparation of a background mask M for each image G, as shown in Fig. 7(b).
[0064] In the prior art, when the background mask M shown in FIG. 7(b) is applied to the image G shown in FIG. 7(a) to remove the masked area, a clipped image Gm shown in FIG. 7(c) is obtained. Since this clipped image Gm only shows the object J, when this is used as training data, the machine learning model 121 can improve the estimation accuracy of the three-dimensional data of the object J. However, the number of multiple images G used as training data is generally huge, and it is difficult to set an appropriate background mask M for each of them. In addition, the background mask M shown in FIG. 7(b) may be automatically created from the feature amount of the image G itself shown in FIG. 7(a), but in this case, unintended parts are often defined as the foreground or background.
[0065] As described above, the machine learning model 121 according to the present invention is constrained so that the densities of the background points and the virtual viewpoint are each a predetermined constant, and therefore objects that are farther away than the background points as viewed from the virtual viewpoint P are ignored in the learning process. Therefore, this machine learning model 121 does not need to perform mask processing on each of the images G that constitute the training data.
[0066] The physical model described below is an example of a physical model included in the above-mentioned machine learning model 121. The machine learning model 121 includes a physical model using a transmittance function T shown in the following formula (1) and an opacity function α shown in the following formula (2).
[0067]
number
[0068]
number
[0069] The transmittance function T shown in formula (1) is a function for calculating the transmittance of light that travels along the line of sight R and reaches the virtual viewpoint P, and is calculated as the sum of the transparencies of all points along the line of sight R. Here, the transparency is a value obtained by subtracting the opacity function α from 1.
[0070] Moreover, the opacity function α is a function shown in formula (2). This opacity function α is a function that calculates the opacity at an arbitrary position on the line of sight R, and is defined to be the greater of either a value calculated according to the difference between the density Φs at that position and the density Φs at a position that is a discrete distance closer to the virtual viewpoint P than that position, or 0.
[0071] In other words, the learning device 1 that trains the machine learning model 121 that employs the physical model shown in this example is an example of a learning device characterized in that, in the machine learning model, the transmittance function T indicating the transmittance of light that travels along the line of sight to the virtual viewpoint, and the opacity function α indicating the opacity at any position along the line of sight are expressed by the above-mentioned equations (1) and (2) when using the density function Φ indicating the density at any position along the line of sight, and the curved surface function f that defines the surface of the virtual object.
[0072] <Functional configuration of the server device> Fig. 8 is a diagram showing an example of the functional configuration of the server device 2. The processor 21 of the server device 2 reads out and executes a program stored in the memory 22, thereby functioning as a photographed image acquisition means 211, a feature point extraction means 212, an estimation means 213, a three-dimensional data generation means 214, a reception means 215, and a supply means 216 shown in Fig. 8. Note that the communication line 3 connecting the terminal 4 and the server device 2 so that they can communicate with each other is omitted in Fig. 8.
[0073] The photographed image acquiring means 211 acquires photographed images of an object taken from a plurality of photographing points. The photographed image acquiring means 211 stores the acquired photographed images in the photographed image DB 222 of the memory 22. The photographed image acquiring means 211 is an example of a photographed image acquiring means that acquires photographed images of an object taken from a plurality of photographing points.
[0074] The feature point extraction means 212 reads out the multiple captured images acquired by the captured image acquisition means 211 from the captured image DB 222, and extracts feature points from each of them. For example, the feature point extraction means 212 detects the contours of objects captured in the captured images by an edge detection algorithm such as the Sobel method, the Laplacian of Gauss method, or the Canny method, and extracts points showing common features from the detected contours as feature points.
[0075] The estimation means 213 estimates the shooting points of each of the multiple captured images based on the feature points extracted by the feature point extraction means 212. The multiple captured images have different shooting points, and the feature points extracted from each of the multiple captured images tend to be more similar as the shooting points are closer. Therefore, the estimation means 213 estimates the proximity of the shooting points, which are the positions at which the multiple captured images were captured, based on the similarity of the feature points, and thereby estimates the arrangement of each shooting point in space. In other words, the estimation means 213 is an example of an estimation means that estimates the shooting points based on the feature points extracted from each captured image.
[0076] The three-dimensional data generation means 214 receives, as input data, a plurality of captured images read out from the captured image DB 222 and shooting points estimated as positions at which the captured images were captured, and generates three-dimensional data of an object appearing in the plurality of captured images by referring to the machine learning model 221. Note that, as described above, this machine learning model 221 is a copy of the machine learning model 121 that has been trained by the learning device 1. In other words, the three-dimensional data generation means 214 is an example of a three-dimensional data generation means that generates three-dimensional data of an object using the estimated shooting points, the captured images corresponding to the shooting points, and the machine learning model that has been trained by the learning device.
[0077] The accepting unit 215 accepts a request for three-dimensional data of an object captured in a captured image from the terminal 4. In other words, the accepting unit 215 is an example of a accepting unit that accepts a request from a terminal.
[0078] The supplying means 216 supplies the three-dimensional data generated in response to the received request to the terminal 4. This is an example of a supplying means that supplies the three-dimensional data generated by the three-dimensional data generating means in response to a request from the terminal to this terminal.
[0079] Fig. 9 is a diagram showing an example of how a virtual image is generated from three-dimensional data. For example, when the terminal 4 photographs the object J shown in Fig. 4 (in this case, the same as the car used in the training data) as a target object and supplies the photographed image to the server device 2, the server device 2 generates three-dimensional data of the virtual object Jv in the virtual space Sv. The terminal 4 can obtain this three-dimensional data and generate, for example, a virtual image Gv in the case where the virtual object Jv is photographed from an arbitrary virtual viewpoint Pv.
[0080] Furthermore, the terminal 4 may capture an image of an object different from the object J used in the teacher data as the target object. FIG. 10 is a diagram showing an example of a new target object Jn captured by the terminal 4. The target object Jn shown in FIG. 10 is an object existing in the real space Sp, for example, a musical instrument such as a violin. This is an object not included in the viewpoint images constituting the teacher data. The terminal 4 captures the target object Jn at various capture points Pn1, Pn2, ..., generates the captured images Gn1, Gn2, ..., and supplies them to the server device 2.
[0081] 11 is a diagram showing three-dimensional data of a virtual object Jnv generated by the server device 2 using the machine learning model 221. When the server device 2 acquires the captured images Gn1, Gn2, ..., the server device 2 generates three-dimensional data of the virtual object Jnv corresponding to the target Jn in the virtual space Sv by using the captured images Gn1, Gn2, ... as input data and referring to the machine learning model 221. The terminal 4 acquires this three-dimensional data from the server device 2, and based on this, can generate a virtual image Gnv obtained when the virtual object Jnv indicated by this three-dimensional data is captured from an arbitrary (virtual) shooting point Pnv.
[0082] The process of generating a virtual image from the three-dimensional data is performed, for example, according to the following formula (3).
[0083]
number
[0084] Here, Ti is the transmittance of a position designated by i on the line of sight R, αi is the opacity of that position, and ci is the color of that position. Equation (3) indicates that by integrating the product of Ti, αi, and ci on the line of sight R, the color of a pixel of a virtual image formed on the imaging plane of the virtual viewpoint by light traveling along the line of sight R can be obtained. The color of this pixel can be expressed in an RGB color space using the three primary colors of light, red, green, and blue, a YUV color space represented by a luminance signal Y and two color difference signals, a YCbCr color space, or the like.
[0085] In the prior art, the density of a fixed point in a virtual space was not set to a constant and constrained as in the present invention. For example, in the case of NeRF, the density of all coordinates defined in the virtual space is often randomly assigned to an initial value. Since NeRF has high optimization stability, even if the density of any of the coordinates is not specifically constrained, the densities are likely to converge to a desired value if an appropriate number of learning sessions are performed. Therefore, in NeRF, it was not necessary to set the density of any of the coordinates in the virtual space to a constant.
[0086] On the other hand, in the case of NeuS, since an SDF that represents the surface of a virtual object is extracted, images with no background that contains an object equivalent to the virtual object are often used as training data. In addition, NeuS generally has lower optimization stability and slower processing than NeRF. Therefore, when using a machine learning model to obtain a representation of the surface of a virtual object from an image, a masking process that removes information other than the object is performed on the image in advance.
[0087] The learning device 1 of the virtual image generation system 9 according to the present invention trains a machine learning model that estimates a three-dimensional model of an object in a virtual space using images obtained by photographing the object from each of a plurality of viewpoints as training data, and trains the machine learning model in which the background points and the density at the virtual viewpoints are constrained to be predetermined constants. Therefore, this learning device 1 does not need to perform a process of removing the background from each image constituting the training data during the learning process of the machine learning model.
[0088] In particular, the present invention shown in the embodiment trains a machine learning model in which the densities at background points and virtual viewpoints are constrained to be 1 and 0, respectively. The present invention is based on NeuS. In the case of NeuS, the density is always 0 somewhere in the line of sight R during the calculation process, so there is no need to ensure that the density of an empty area in the transmittance function T is 0. Therefore, in the conventional technology based on NeuS, it was not necessary to distinguish the virtual viewpoint from other coordinates and set the density value to 0. The present invention has the effect that the density of an empty area in the transmittance function T is easily converged to 0 because the density value of the virtual viewpoint is set to 0 by distinguishing it from other coordinates.
[0089] In addition, the server device of the present invention can generate three-dimensional data of an object from raw captured images (photos, etc.) taken by a user from multiple viewpoints using a smartphone or the like, using a trained machine learning model.
[0090] An unprocessed captured image also includes a background, but when this is handled by NeuS, light rays (light traveling along the line of sight R) emitted from any pixel constituting the captured image will collide with some object, at which point the transmittance function T should fall to 0. Since the machine learning model trained by the present invention is constrained so that the density at background points is 1, it has the effect of easily obtaining a convergent solution in which the transmittance function T becomes 0 at each position where each of the above-mentioned light rays collide with an object.
[0091] <Modification> The above is a description of the embodiment, but the contents of this embodiment may be modified as follows. In addition, the following modifications may be combined.
[0092] <1> In the above-described embodiment, the virtual image generation system 9 has the learning device 1 and the server device 2, respectively, but one of them may also have the function of the other. For example, the server device 2 may store teacher data corresponding to the above-described viewpoint image DB 120 in the memory 22. In this case, the processor 21 of the server device 2 may train the machine learning model 221 based on the teacher data.
[0093] <2> In the above-described embodiment, the processor 11 and the processor 21 are both a CPU or a GPU, but they may be configured to cooperate with each other, or may have other configurations. For example, these processors may be or include a Field Programmable Gate Array (FPGA). Furthermore, these processors may have an Application Specific Integrated Circuit (ASIC) or other programmable logic device.
[0094] <3> In the above-mentioned embodiment, the terminal 4 requests three-dimensional data of the photographed object, but it may also specify a virtual viewpoint from which to photograph the object and request a virtual image obtained when the object is photographed from the virtual viewpoint. In this case, the server device 2 may generate a virtual image of the object in the virtual space represented by the three-dimensional data generated by the three-dimensional data generating means, photographed from the specified virtual viewpoint, by rendering, and supply the generated image to the terminal 4.
[0095] Fig. 12 is a diagram showing an example of a functional configuration of a server device 2 according to a modified example. The processor 21 of the server device 2 shown in Fig. 12 realizes a virtual image generating means 217 in addition to the functions realized by the processor 21 shown in Fig. 8. This virtual image generating means 217 generates a virtual image using the three-dimensional data generated by the three-dimensional data generating means 214.
[0096] In this case, the receiving means 215 receives designation of a virtual viewpoint for photographing an object in a virtual space from the terminal 4. That is, the receiving means 215 is an example of a receiving means for receiving designation of a virtual viewpoint for photographing an object from a terminal.
[0097] The virtual image generating means 217 generates a virtual image obtained when an object represented by the three-dimensional data is photographed from a specified virtual viewpoint, based on the specification accepted by the accepting means 215. In other words, the virtual image generating means 217 is an example of a virtual image generating means that generates a virtual image obtained when an object represented by the three-dimensional data generated by the three-dimensional data generating means is photographed from a virtual viewpoint indicated by the specification accepted from the terminal.
[0098] Then, the supplying means 216 supplies the generated virtual image to the terminal 4. In other words, the supplying means 216 is an example of a supplying means that supplies a virtual image generated by the virtual image generating means in response to a specification received from the terminal to the terminal. Even in this case, the terminal 4 can obtain a virtual image of the photographed object viewed from a specified virtual viewpoint.
[0099] <4> The program executed by the above-mentioned processor 11 is conceived as a program that causes a computer to function as: an acquisition means for acquiring teacher data that associates multiple viewpoints with images obtained by photographing an object from each of the multiple viewpoints; a generation means for generating a virtual image obtained when a virtual object corresponding to an object is photographed in a virtual space from a virtual viewpoint corresponding to the viewpoint, using a machine learning model including a density for each position on the line of sight from a background point in the background of the virtual object through the virtual object to the virtual viewpoint; and a learning means for comparing, for each virtual viewpoint, the generated virtual image with an image photographed from a viewpoint corresponding to the virtual viewpoint, and training the machine learning model, and constraining the machine learning model so that the densities at the background points and the virtual viewpoints are each predetermined constants.
[0100] In addition, the program executed by the above-mentioned processor 21 is conceived as a program that causes a computer to function as: an image acquisition means for acquiring images of an object photographed from a plurality of shooting points; an estimation means for estimating the shooting points based on feature points extracted from each of the photographed images; and a three-dimensional data generation means for generating three-dimensional data of the object using the estimated shooting points, the photographed images corresponding to the shooting points, and a machine learning model trained in a learning device by the program executed by the processor 11. [Explanation of symbols]
[0101] 1...learning device, 11...processor, 111...acquisition means, 112...generation means, 113...learning means, 12...memory, 120...viewpoint image DB, 121...machine learning model, 13...interface, 2...server device, 21...processor, 211...captured image acquisition means, 212...feature point extraction means, 213...estimation means, 214...stereoscopic data generation means, 215...reception means, 216...supply means, 217...virtual image generation means, 22...memory, 221...machine learning model, 222...captured image DB, 23...interface, 3...communication line, 4...terminal, 5...camera, 9...virtual image generation system.
Claims
1. An acquisition means for acquiring teacher data in which a plurality of viewpoints are associated with images obtained by photographing an object from each of the plurality of viewpoints; a generating means for generating a virtual image obtained by photographing a virtual object corresponding to the object from a virtual viewpoint corresponding to the viewpoint in a virtual space, using a machine learning model including a density for each position on a line of sight from a background point in the background of the virtual object through the virtual object to the virtual viewpoint; A learning means for learning the machine learning model by comparing the generated virtual image with the image captured from the viewpoint corresponding to the virtual viewpoint for each virtual viewpoint; having The machine learning model is constrained so that the background points and the density at the virtual viewpoint are each a predetermined constant. A learning device characterized by:
2. The machine learning model includes a surface function that defines a surface of the virtual object by a multi-layer perceptron, and a density function that indicates the density using the surface function; The learning means updates the surface function by a backpropagation method to train the machine learning model; The density function is constrained such that the density at the background points is 1 and the density at the virtual viewpoint is 0.
2. The learning device according to claim 1 .
3. In the machine learning model, A transmittance function T indicating the transmittance of light traveling along the line of sight to the virtual viewpoint; and The opacity function α indicating the opacity at any position on the line of sight is A density function Φ indicating the density at any position on the line of sight, and A surface function f that defines the surface of the virtual object; When using the formula (1) and the formula (2), 3. The learning device according to claim 2.
4. A photographed image acquisition means for acquiring photographed images of an object from a plurality of photographing points; an estimation means for estimating the photographing point based on feature points extracted from each of the photographed images; a three-dimensional data generating means for generating three-dimensional data of the object using the estimated shooting points, the captured images corresponding to the shooting points, and the machine learning model according to any one of claims 1 to 3; A server device having the above configuration.
5. A receiving means for receiving a request from a terminal; a supplying means for supplying the three-dimensional data generated by the three-dimensional data generating means in response to the request to the terminal; 5. The server device according to claim 4, further comprising:
6. A reception means for receiving, from a terminal, a designation of a virtual viewpoint for photographing the object; a virtual image generating means for generating a virtual image obtained when the object represented by the three-dimensional data generated by the three-dimensional data generating means is photographed from the virtual viewpoint represented by the designation; a supplying means for supplying the virtual image generated by the virtual image generating means in response to the designation to the terminal; 5. The server device according to claim 4, further comprising:
7. Computer, An acquisition means for acquiring teacher data in which a plurality of viewpoints are associated with images obtained by photographing an object from each of the plurality of viewpoints; a generating means for generating a virtual image obtained by photographing a virtual object corresponding to the object from a virtual viewpoint corresponding to the viewpoint in a virtual space, using a machine learning model including a density for each position on a line of sight from a background point in the background of the virtual object through the virtual object to the virtual viewpoint; a learning means for learning the machine learning model by comparing, for each of the virtual viewpoints, the generated virtual image with the image captured from the viewpoint corresponding to the virtual viewpoint; It also functions as a A program comprising: constraining the machine learning model so that the background points and the density at the virtual viewpoint are each a predetermined constant.
8. Computer, A photographed image acquisition means for acquiring photographed images of an object from a plurality of photographing points; an estimation means for estimating the photographing point based on feature points extracted from each of the photographed images; a three-dimensional data generating means for generating three-dimensional data of the object by using the estimated shooting points, the captured images corresponding to the shooting points, and a machine learning model trained by the program according to claim 7; A program characterized by causing the program to function as a
Citation Information
Patent Citations
Method and equipment for determining viewpoint path in three-dimensional scene
CN113628348A
Image rendering method and device, equipment, storage medium and product
CN115861508A
Signal processing device and signal processing method
WO2023281802A1
Three dimensional model generation apparatus and method
JP2021071749A
