Learning device, server device, and program

By constraining the density of background points and virtual viewpoints in the machine learning model, the method addresses the challenge of background interference in photogrammetry, enabling accurate and efficient three-dimensional shape estimation and virtual image generation without manual masking.

WO2024247330A1PCT designated stage expired Publication Date: 2024-12-05BESTAT INC
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2023/046504
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-26
Filing Date
2023-12-25
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Conventional photogrammetry techniques require manual background removal or masking during training, which is cumbersome and affects the accuracy of estimating three-dimensional shapes from images, especially when background points and virtual viewpoints are involved.

Method used

A machine learning model is trained with constraints where the density at background points and virtual viewpoints are predetermined constants, eliminating the need for background masking and improving the accuracy of three-dimensional shape estimation by ignoring background points during the learning process.

Benefits of technology

This approach allows for more accurate and efficient generation of three-dimensional models without the need for background masking, enhancing the stability and precision of virtual image generation by focusing solely on the object's surface.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2023046504_05122024_PF_FP_ABST
    Figure JP2023046504_05122024_PF_FP_ABST
Patent Text Reader

Abstract

A viewpoint image DB 120 associates and stores a plurality of viewpoints and an image obtained by imaging an object from each of the plurality of viewpoints. An acquisition means 111 reads the viewpoint image DB 120 and supplies the same to a generation means 112. The generation means 112 generates a virtual image obtained when a virtual object is imaged from a virtual viewpoint by using an initial state or a machine learning model 121 which is being trained. A learning means 113 updates the machine learning model 121, which is constrained in a manner such that the densities at a background point and a virtual viewpoint each equal a predetermined constant, according to the difference between the image stored in the viewpoint image DB 120 and the virtual image generated by the generation means 112. The present invention renders processing for deleting the background from each image unnecessary, when training a machine learning model for estimating a three-dimensional model of an object in a virtual space, with the image obtained by imaging the object from each of a plurality of viewpoints as training data.
Need to check novelty before this filing date? Find Prior Art

Description

Learning device, server device, and program

[0001] The present invention relates to a technique for estimating the three-dimensional shape of an object from a plurality of images of the object.

[0002] Photogrammetry is a technology for estimating the three-dimensional shape of an object from multiple images of the object. For example, Patent Literature 1 discloses, as a photogrammetry technology, an apparatus and method for generating a three-dimensional (3D) model of a subject from images taken by multiple cameras.

[0003] Japanese Patent Application Laid-Open No. 2021-71749

[0004] Rendering is the process of generating an image of an object that would appear when viewed from a specified viewpoint, given information such as the object's shape, material, and color, as well as the position and light intensity of the light source. In contrast, photogrammetry is a process that estimates the three-dimensional shape, color, texture, etc. of an object from multiple images of the object. This process is also called inverse rendering, as it is the reverse of rendering. Artificial neural network (ANN) technology is sometimes applied to solve the inverse rendering problem in photogrammetry.

[0005] Photogrammetry technology that applies ANN trains a machine learning model that includes optical assumptions using images obtained by photographing an object from multiple viewpoints as training data. This machine learning model is configured, for example, with a multilayer perceptron (MLP). Using this trained machine learning model, it is possible to generate a three-dimensional model of an object (also called a virtual object) in a virtual space from new images of the object photographed from multiple viewpoints. Then, using this generated three-dimensional model, it is possible to generate images (also called virtual images) that would be obtained if the virtual object were photographed from any viewpoint (also called a virtual viewpoint) in the virtual space.

[0006] Neural Radiance Fields (NeRF) is a technique for representing density and brightness fields in a three-dimensional virtual space using MLP and generating a virtual image of a virtual object captured from an arbitrary virtual viewpoint in this virtual space. NeRF generates the above-mentioned two-dimensional virtual image using volume rendering. NeRF uses, for example, a "volumetric representation" in which the three-dimensional virtual space is divided into a grid, so-called voxels. For this reason, inverse rendering based on NeRF is said to be easier to optimize stably than methods based on surface representation (described below).

[0007] Meanwhile, another method for generating the above-mentioned virtual image is NeuS (Neural Scene Representation). NeuS is a method for generating a virtual object in a virtual space based on a curved surface representation. NeuS is a method for generating a virtual object in a form such as a point cloud or a mesh by representing the surface of an object as a zero-level set of a signed distance function (SDF) approximated by MLP and learning a model of this surface using images captured from multiple viewpoints as training data. Because NeuS represents the surface of an object as an SDF, which is a continuous function approximated by MLP, it has an advantage in representing the surface of an object composed of a curved surface compared to NeRF, which uses a volumetric representation.

[0008] When training the above-mentioned machine learning models using deep learning, such as NeRF and NeuS, the density at both ends of the line of sight (background points and virtual viewpoints) of a virtual object has traditionally been determined by chance. Therefore, in conventional techniques, if a background is included in the training data image, the background interferes with learning, so it is necessary to either remove the background using an appropriate foreground mask or model it using NeRF. For example, the technique described in Patent Literature 1 requires a binary mask image in which the object and other parts are represented in different gradations (e.g., black and white) when estimating the three-dimensional shape of an object from multiple images of the object.

[0009] One of the objectives of the present invention is to eliminate the need to remove the background from each image when training a machine learning model that estimates a three-dimensional model of an object in a virtual space using images obtained by photographing the object from multiple viewpoints as training data.

[0010] In one aspect, the present invention provides a learning device comprising: an acquisition means for acquiring training data that associates a plurality of viewpoints with images obtained by photographing an object from each of the plurality of viewpoints; a generation means for generating, in a virtual space, a virtual image obtained when a virtual object corresponding to the object is photographed from a virtual viewpoint corresponding to the viewpoint, using a machine learning model that includes a density for each position on the line of sight that runs from a background point in the background of the virtual object through the virtual object to the virtual viewpoint; and a learning means for comparing, for each virtual viewpoint, the generated virtual image with the image photographed from the viewpoint corresponding to the virtual viewpoint, and training the machine learning model, wherein the machine learning model is constrained so that the densities at the background points and the virtual viewpoints are each predetermined constants.

[0011] In a preferred aspect, the machine learning model includes a surface function that defines the surface of the virtual object using a multi-layer perceptron, and a density function that indicates the density using the surface function, and the learning means trains the machine learning model by updating the surface function using a backpropagation algorithm, and the density function is constrained so that the density at the background point is 1 and the density at the virtual viewpoint is 0.

[0012] In another preferred aspect, in the machine learning model, a transmittance function T indicating the transmittance of light that travels along the line of sight and reaches the virtual viewpoint, and an opacity function α indicating the opacity at any position on the line of sight are expressed by the following equations when a density function Φ indicating the density at any position on the line of sight and a surface function f defining the surface of the virtual object are used:

[0013] In another aspect, the present invention provides a server device having a captured image acquisition means for acquiring captured images of an object from a plurality of shooting points, an estimation means for estimating the shooting points based on feature points extracted from each of the captured images, and a three-dimensional data generation means for generating three-dimensional data of the object using the estimated shooting points, the captured images corresponding to the shooting points, and the machine learning model.

[0014] In a preferred embodiment, the server device further comprises: a receiving means for receiving a request from a terminal; and a supplying means for supplying the terminal with the three-dimensional data generated by the three-dimensional data generating means in response to the request.

[0015] In another preferred aspect, the server device further includes a receiving means for receiving from a terminal a specification of a virtual viewpoint from which to photograph the object, a virtual image generating means for generating a virtual image obtained when the object represented by the three-dimensional data generated by the three-dimensional data generating means is photographed from the virtual viewpoint specified by the specification, and a supplying means for supplying the virtual image generated by the virtual image generating means in accordance with the specification to the terminal.

[0016] In another aspect, the present invention provides a program that causes a computer to execute the following steps: acquiring training data that associates a plurality of viewpoints with images obtained by photographing an object from each of the plurality of viewpoints; generating, in a virtual space, a virtual image obtained when a virtual object corresponding to the object is photographed from a virtual viewpoint corresponding to the viewpoint, using a machine learning model that includes a density for each position on the line of sight that runs from a background point in the background of the virtual object through the virtual object to the virtual viewpoint; and comparing, for each virtual viewpoint, the generated virtual image with the image photographed from the viewpoint corresponding to the virtual viewpoint, to train the machine learning model, wherein the program constrains the machine learning model so that the densities at the background points and the virtual viewpoints are each predetermined constants.

[0017] In another aspect, the present invention provides a program that causes a computer to execute the steps of acquiring photographed images of an object taken from a plurality of shooting points, estimating the shooting points based on feature points extracted from each of the photographed images, and generating three-dimensional data of the object using the estimated shooting points, the photographed images corresponding to the shooting points, and a machine learning model trained by the program.

[0018] 1 is a diagram showing an example of the overall configuration of a virtual image generation system 9 according to an embodiment of the present invention. A diagram showing examples of the configurations of a learning device 1 and a server device 2. A diagram showing an example of a viewpoint image DB 120. A diagram for explaining images stored in the viewpoint image DB 120. A diagram showing an example of the functional configuration of the learning device 1. A diagram for explaining SDF adopted in a machine learning model 121. A diagram for explaining that a background mask is not required for training data used in the machine learning model 121. A diagram showing an example of the functional configuration of a server device 2. A diagram showing an example of how a virtual image is generated from three-dimensional data. A diagram showing an example of a new object Jn photographed by a terminal 4. A diagram showing three-dimensional data of a virtual object Jnv generated by the server device 2 using a machine learning model 221. A diagram showing an example of the functional configuration of a server device 2 according to a modified example.

[0019] <Embodiment> <Overall Configuration of Virtual Image Generation System> Figure 1 is a diagram showing an example of the overall configuration of a virtual image generation system 9 according to an embodiment of the present invention. The virtual image generation system 9 is a system that uses a machine learning model trained with training data that associates multiple viewpoints with images (photographs) of an object taken from each of the viewpoints, and generates a virtual image of a virtual object corresponding to the object in a virtual space. The virtual image generation system 9 shown in Figure 1 includes a learning device 1, a server device 2, a communication line 3, and a terminal 4. The virtual image generation system 9 may include multiple terminals 4, and some of these terminals 4 may be connectable to a camera 5.

[0020] The learning device 1 is a device that trains a machine learning model using the above-mentioned training data, and is, for example, a computer.

[0021] The server device 2 is a device that generates three-dimensional data of a virtual object corresponding to a new object (also referred to as a target object) from multiple images of the object, which are supplied from the terminal 4, by using a copy of the machine learning model that has been trained by the learning device 1. By using this three-dimensional data, the server device 2 or the terminal 4 can generate a virtual image that would be obtained when the virtual object is photographed from an arbitrary virtual viewpoint in a virtual space.

[0022] The terminal 4 is a terminal device carried by a user of the virtual image generation system 9, and may be, for example, a smartphone, a personal computer, etc. The terminal 4 may be provided with a photographing function for photographing an object, such as a camera. The terminal 4 may also be connected to a camera 5, such as a digital still camera or video camera, and may be able to acquire images photographed by the camera 5.

[0023] The communication line 3 is a line that connects the learning device 1, the server device 2, and the terminal 4 via wired or wireless connections so that they can communicate with each other. The communication line 3 may be, for example, a local area network (LAN), a wide area network (WAN), the Internet, or a combination of these. The communication line 3 may also include public switched telephone networks (PSTN), integrated services digital networks (ISDN), etc.

[0024] 2 is a diagram showing an example of the configuration of the learning device 1 and the server device 2. The learning device 1 has a processor 11, a memory 12, and an interface 13. These components are connected to each other so that they can communicate with each other, for example, by a bus.

[0025] The processor 11 controls each part of the learning device 1 by reading and executing a computer program (hereinafter simply referred to as a program) stored in the memory 12. The processor 11 is, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

[0026] The interface 13 is a communication circuit that connects the learning device 1 to the communication line 3 so that communication is possible, either wired or wirelessly.

[0027] The memory 12 is a storage unit that stores an operating system, various programs, data, and the like that are loaded into the processor 11. The memory 12 includes a random access memory (RAM) or a read only memory (ROM). The memory 12 may include a solid state drive, a hard disk drive, or the like. The memory 12 also stores a viewpoint image DB 120 and a machine learning model 121.

[0028] The viewpoint image DB 120 is a database that stores a plurality of viewpoints and images obtained by photographing an object from each of the plurality of viewpoints, in association with each other. The images stored in the viewpoint image DB 120 are data obtained by actually photographing an object, and are also called GT (Ground Truth) data. The machine learning model 121 is a machine learning model that is trained by using pairs of viewpoints and images stored in the viewpoint image DB 120 as training data. This machine learning model 121 is used to infer three-dimensional data of the object photographed in each of the above-mentioned images.

[0029] The server device 2 includes a processor 21, a memory 22, and an interface 23. The processor 21 controls each part of the server device 2 by reading and executing a program stored in the memory 22. The processor 21 is, for example, a CPU or a GPU.

[0030] The interface 23 is a communication circuit that connects the server device 2 to the communication line 3 in a wired or wireless manner so that communication is possible.

[0031] The memory 22 is a storage unit that stores an operating system, various programs, data, etc. that are loaded into the processor 21. The memory 22 includes a RAM or a ROM. The memory 22 may also include a solid state drive, a hard disk drive, etc. The memory 22 also stores a machine learning model 221 and a captured image DB 222.

[0032] The machine learning model 221 is a copy of the machine learning model 121 that has been generated and trained by the learning device 1. The server device 2 obtains a copy of the trained machine learning model 121 from the learning device 1 and stores it in the memory 22.

[0033] The captured image DB 222 is a database in which the server device 2 stores a plurality of captured images acquired from the terminal 4. The captured images stored in the captured image DB 222 are, for example, photographs of objects existing in real space taken from a plurality of shooting points. Unlike the viewpoint image DB 120 described above, the captured images do not need to be associated with the shooting points at which the objects were photographed.

[0034] Both the learning device 1 and the server device 2 may have an operation unit and a display unit. The operation unit is configured to receive operations and send signals corresponding to the operation content to the processor 11. This operation unit may include, for example, operation buttons, a keyboard, a touch panel, a mouse, or other controls for issuing various instructions.

[0035] The display unit is configured to display images under the control of processor 11. The display unit may have a display screen such as a liquid crystal display. The transparent touch panel of the operation unit described above may be placed on top of the display screen. Note that study device 1 and server device 2 may be operated from an external device or may present information to an external device via interface 13 and interface 23, respectively.

[0036] <Configuration of viewpoint image DB> Fig. 3 is a diagram showing an example of the viewpoint image DB 120. The viewpoint image DB 120 shown in Fig. 3 stores the ID, viewpoint, and image items in association with each other. The "ID" item in the viewpoint image DB 120 is identification information that identifies an image.

[0037] The "viewpoint" item in the viewpoint image DB 120 is information indicating the viewpoint from which the image was captured. This information includes, for example, the coordinates of the viewpoint when the center point of the object being photographed is taken as the origin. This information may also include information on the viewing angle at the viewpoint.

[0038] The "image" item in the viewpoint image DB 120 is image data that is identified by the above-mentioned ID and obtained when an object is photographed from the above-mentioned viewpoint.

[0039] Fig. 4 is a diagram for explaining images stored in the viewpoint image DB 120. A user of the virtual image generation system 9 uses, for example, a camera mounted on the terminal 4 to capture images of an object J (a car in the example of Fig. 4 ) existing in a real space Sp from various viewpoints P1, P2, .... The positions of the object J and the viewpoints P1, P2, ... in the real space Sp shown in Fig. 4 are expressed, for example, in an xyz right-handed coordinate space.

[0040] At this time, the terminal 4 identifies the positions of the viewpoints P1, P2, .... The viewpoints may be identified using, for example, a distance measuring device such as LiDAR, or may be identified by triangulating markers or the like that have been installed in advance in the shooting space using a stereo camera. Alternatively, the camera that captures the object J may be attached to an arm that moves along a predetermined trajectory, and the terminal 4 may identify the viewpoints by acquiring drive information for this arm.

[0041] Terminal 4 then associates the captured images G1, G2... with the coordinates of the viewpoints P1, P2... from which each image was captured, and supplies this data to learning device 1 via communication line 3. When terminal 4 acquires images of object J captured from multiple viewpoints along with coordinate information for each viewpoint, learning device 1 stores these in viewpoint image DB 120. Processor 11 of learning device 1 trains machine learning model 121 using the content stored in viewpoint image DB 120 as training data. Terminal 4 may also acquire the above-mentioned multiple images G1, G2... from an externally connected camera 5 and supply them to learning device 1.

[0042] <Functional Configuration of Learning Device> Fig. 5 is a diagram showing an example of the functional configuration of learning device 1. Processor 11 of learning device 1 reads and executes programs stored in memory 12, thereby functioning as acquisition means 111, generation means 112, and learning means 113 shown in Fig. 5.

[0043] The acquisition means 111 reads out the viewpoint image DB 120 from the memory 12 and supplies it to the generation means 112. This viewpoint image DB 120 is used as training data by the generation means 112. Therefore, the acquisition means 111 is an example of an acquisition means that acquires training data that associates a plurality of viewpoints with images obtained by photographing an object from each of the plurality of viewpoints.

[0044] The generation means 112 uses the initial state stored in the memory 12 or the machine learning model 121 in the middle of learning to generate a virtual image obtained when a virtual object corresponding to an object is photographed from a virtual viewpoint corresponding to multiple viewpoints included in the viewpoint image DB 120.

[0045] The learning means 113 compares the virtual image generated by the generation means 112 with the image stored in the viewpoint image DB 120, and updates the machine learning model 121 so as to reduce the difference between them. When this difference satisfies a predetermined condition, the machine learning model 121 is considered to have been trained.

[0046] A detailed example of the machine learning model 121 is as follows. The learning means 113 refers to the viewpoint image DB 120 to acquire images, such as photographs, of an object J present in the real space Sp, and the viewpoint at the time of capturing the images, and performs deep learning using these as training data to update the machine learning model 121. As a result of updating through learning, the machine learning model 121 enables the generation of three-dimensional data, such as the shape, texture, and color of a virtual object in a virtual space corresponding to the object J. The generated three-dimensional data is used to generate a virtual image obtained when a virtual object is captured from an arbitrary virtual viewpoint in the virtual space.

[0047] The machine learning model 121 also employs a physical model that expresses the surface of a virtual object using the SDF described above. FIG. 6 is a diagram illustrating the SDF employed in the machine learning model 121.

[0048] 6, the surface of a virtual object in the virtual space Sv is itself represented by a surface function f. This surface function f is represented by, for example, a group of weight parameters for each layer in the above-mentioned MLP.

[0049] Furthermore, the physical laws that light follows in the virtual space Sv are modeled using a transmittance function T, an opacity function α, and a density function Φ. The density function Φ is a function that indicates the density at any position in the virtual space, and is calculated using a surface function f that represents the surface of the virtual object at that position. For example, a sigmoid function is used as the density function Φ, but it is not limited to this as long as the gradient is maximized when the surface function f is 0. For example, the density function Φ may be an erf function (cumulative distribution function of a Gaussian distribution) or a cumulative distribution function of a Laplace distribution.

[0050] Therefore, the machine learning model 121 described here is an example of a machine learning model that includes a surface function that defines the surface of a virtual object using a multi-layer perceptron and a density function that indicates density using this surface function.

[0051] 6 represents a ray of light traveling toward a virtual viewpoint P. The line of sight R is expressed by a three-dimensional vector function p(t) in which the optical path of the ray of light is expressed with a parameter t. The range of p(t) is expressed as a three-dimensional vector, for example, in an XYZ right-handed coordinate system representing the virtual space Sv.

[0052] 6 is the point where the line of sight R starts. Point p(t0) is called a "background point" because it is a point that exists behind (on the background side of) the virtual object when viewed from the virtual viewpoint P. The background point is determined, for example, by its distance from the viewpoint P.

[0053] 6 is an expression of the virtual viewpoint P in the three-dimensional vector function p(t). Note that the point p(tm) is a point at which the line of sight R intersects with a surface where an image is formed (called an imaging plane) as it moves toward the virtual viewpoint P. The light at this imaging plane forms an image observed at the virtual viewpoint P.

[0054] The generation means 112 applies a machine learning model 121 in its initial state or in the middle of learning to each of a plurality of virtual viewpoints P corresponding to each viewpoint in the viewpoint image DB 120 used as training data, and calculates the density, light transmittance, and opacity for each position on the line of sight R where light passes from a background point through a virtual object to the virtual viewpoint P, thereby generating a virtual image to be formed on the imaging surface.

[0055] Therefore, this generation means 112 is an example of a generation means that generates a virtual image obtained when a virtual object corresponding to an object in real space is photographed in a virtual space from a virtual viewpoint corresponding to a plurality of viewpoints in real space, using a machine learning model that includes a density for each position on the line of sight from a background point in the background of the virtual object, through the virtual object, to the virtual viewpoint.

[0056] The learning means 113 calculates an objective function according to the difference between the image stored in the viewpoint image DB 120 and the virtual image generated by the generation means 112. Then, the learning means 113 updates the machine learning model 121 using an error propagation method or the like so that the objective function becomes, for example, a minimum value. In this way, the learning means 113 advances deep learning of the machine learning model 121, and as a result, the trained machine learning model 121 is completed.

[0057] In other words, this learning means 113 is an example of a learning means that compares, for each virtual viewpoint, the generated virtual image with an image taken from a viewpoint corresponding to this virtual viewpoint, and trains a machine learning model.

[0058] The learning means 113 is an example of a learning means that updates the surface function by backpropagation, thereby learning a machine learning model.

[0059] Here, the machine learning model 121 according to the present invention is constrained so that the densities of the background points and the virtual viewpoint are each predetermined constants. That is, the machine learning model 121 is an example of a machine learning model in which the densities of the background points and the virtual viewpoint are each constrained so that they are each predetermined constants.

[0060] For example, this machine learning model 121 constrains the density function Φ so that the density at background points is 1 and the density at virtual viewpoint P is 0. Therefore, the density at background points that are a certain distance from the virtual viewpoint P across the virtual object is always 1. As a result, this machine learning model 121 ignores objects that are farther away than the background points as viewed from the virtual viewpoint P, and therefore, backgrounds that are farther away than object J in images used for training data do not contribute to the calculation. Furthermore, because the density is always 0 at the virtual viewpoint P, the surface of the virtual object is always defined somewhere along the path of light traveling along the line of sight R until it reaches the virtual viewpoint P.

[0061] Therefore, this density function Φ included in the machine learning model 121 is an example of a density function that is constrained to have a density of 1 at the background points and a density of 0 at the virtual viewpoint.

[0062] 7 is a diagram illustrating that a background mask is not required for the training data used in the machine learning model 121. The training data for training the machine learning model 121 includes an image G obtained by capturing an object J existing in the real space Sp, but this image G often contains some other object (referred to as a background object B) in addition to the object J, as shown in (a) of FIG.

[0063] Among conventional physical models, for example, NeRF may sometimes represent the background object B in voxels, which may result in a decrease in the accuracy of the representation of object J. Furthermore, in the conventional NeuS, in order to calculate the surface shape of only object J, it is necessary to exclude the background object B that appears in the captured image from the training data. Therefore, the conventional NeuS required the preparation of a background mask M as shown in FIG. 7(b) for each image G.

[0064] In the prior art, when the background mask M shown in FIG. 7(b) is applied to the image G shown in FIG. 7(a) and the masked area is removed, a clipped image Gm as shown in FIG. 7(c) is obtained. Because this clipped image Gm captures only the object J, using it as training data improves the accuracy of the machine learning model 121 in estimating the three-dimensional data of the object J. However, the number of multiple images G used as training data is generally enormous, making it difficult to set an appropriate background mask M for each of them. In addition, the background mask M shown in FIG. 7(b) may be automatically created based on the feature values ​​of the image G itself shown in FIG. 7(a), but in this case, unintended portions are often defined as the foreground or background.

[0065] As described above, the machine learning model 121 according to the present invention is constrained so that the densities of background points and the virtual viewpoint are each predetermined constants, and therefore objects that are farther away than the background points as viewed from the virtual viewpoint P are ignored during the learning process. Therefore, this machine learning model 121 does not need to perform mask processing on each of the images G that make up the training data.

[0066] The physical model described below is an example of a physical model included in the above-mentioned machine learning model 121. The machine learning model 121 includes a physical model that uses a transmittance function T shown in the following formula (1) and an opacity function α shown in formula (2).

[0067]

[0068]

[0069] The transmittance function T shown in formula (1) is a function for calculating the transmittance of light that travels along the line of sight R and reaches the virtual viewpoint P, and is calculated as the sum of the transparencies of all points along the line of sight R. Here, the transparency is a value obtained by subtracting the opacity function α from 1.

[0070] The opacity function α is a function expressed by equation (2). This opacity function α is a function for calculating the opacity at an arbitrary position on the line of sight R, and is defined to be the larger of either a value calculated according to the difference between the density Φs at that position and the density Φs at a position closer to the virtual viewpoint P by a discrete distance than that position, or 0.

[0071] In other words, the learning device 1 that trains the machine learning model 121 that employs the physical model shown in this example is an example of a learning device characterized in that, in the machine learning model, the transmittance function T indicating the transmittance of light that travels along the line of sight and reaches the virtual viewpoint, and the opacity function α indicating the opacity at any position along the line of sight are expressed by the above-mentioned equations (1) and (2) when using the density function Φ indicating the density at any position along the line of sight, and the curved surface function f that defines the surface of the virtual object.

[0072] <Functional Configuration of Server Device> Fig. 8 is a diagram showing an example of the functional configuration of the server device 2. The processor 21 of the server device 2 reads and executes programs stored in the memory 22, thereby functioning as a captured image acquisition unit 211, a feature point extraction unit 212, an estimation unit 213, a three-dimensional data generation unit 214, a reception unit 215, and a supply unit 216 shown in Fig. 8. Note that the communication line 3 that communicatively connects the terminal 4 and the server device 2 is omitted from Fig. 8.

[0073] The photographed image acquisition means 211 acquires photographed images of an object taken from a plurality of photographing points, respectively. The photographed image acquisition means 211 stores the acquired photographed images in the photographed image DB 222 of the memory 22. The photographed image acquisition means 211 is an example of a photographed image acquisition means that acquires photographed images of an object taken from a plurality of photographing points, respectively.

[0074] The feature point extraction means 212 reads out the multiple photographed images acquired by the photographed image acquisition means 211 from the photographed image DB 222 and extracts feature points from each of the photographed images. For example, the feature point extraction means 212 detects the contours of objects appearing in the photographed images using an edge detection algorithm such as the Sobel method, the Laplacian of Gauss method, or the Canny method, and extracts points showing common features from the detected contours as feature points.

[0075] The estimation means 213 estimates the shooting points of each of the multiple captured images based on the feature points extracted by the feature point extraction means 212. The multiple captured images each have a different shooting point, and the feature points extracted from each of the multiple captured images tend to be more similar the closer the shooting points are. Therefore, the estimation means 213 estimates the proximity of the shooting points, which are the positions at which the multiple captured images were captured, based on the similarity of the feature points, and thereby estimates the spatial arrangement of each shooting point. In other words, the estimation means 213 is an example of an estimation means that estimates the shooting points based on the feature points extracted from each of the captured images.

[0076] The three-dimensional data generation means 214 receives, as input data, a plurality of captured images read from the captured image DB 222 and shooting points estimated as positions at which the captured images were captured, and generates three-dimensional data of the object appearing in the plurality of captured images by referring to the machine learning model 221. Note that, as described above, this machine learning model 221 is a copy of the machine learning model 121 that has been trained by the learning device 1. In other words, the three-dimensional data generation means 214 is an example of three-dimensional data generation means that generates three-dimensional data of the object using estimated shooting points, captured images corresponding to these shooting points, and a machine learning model trained by the learning device.

[0077] The accepting unit 215 accepts a request for three-dimensional data of an object shown in a captured image from the terminal 4. In other words, the accepting unit 215 is an example of a accepting unit that accepts a request from a terminal.

[0078] The supplying means 216 supplies the three-dimensional data generated in response to the received request to the terminal 4. This is an example of supplying means that supplies the three-dimensional data generated by the three-dimensional data generating means in response to a request from the terminal to this terminal.

[0079] 9 is a diagram showing an example of how a virtual image is generated from three-dimensional data. For example, when the terminal 4 photographs the object J shown in FIG. 4 (in this case, the same as the automobile used in the training data) as a target object and supplies the photographed image to the server device 2, the server device 2 generates three-dimensional data of the virtual object Jv in the virtual space Sv. The terminal 4 can acquire this three-dimensional data and generate a virtual image Gv of the virtual object Jv photographed from an arbitrary virtual viewpoint Pv, for example.

[0080] Furthermore, the terminal 4 may capture an image of an object different from the object J used in the training data as the target object. FIG. 10 is a diagram showing an example of a new target object Jn captured by the terminal 4. The target object Jn shown in FIG. 10 is an object existing in the real space Sp, for example, a musical instrument such as a violin. This is an object that is not included in the viewpoint images that constitute the training data. The terminal 4 captures images of this target object Jn from various shooting points Pn1, Pn2, ..., generates captured images Gn1, Gn2, ..., and supplies them to the server device 2.

[0081] 11 is a diagram showing three-dimensional data of a virtual object Jnv generated by the server device 2 using the machine learning model 221. When the server device 2 acquires the captured images Gn1, Gn2, ..., the server device 2 uses the captured images Gn1, Gn2, ... as input data and generates three-dimensional data of the virtual object Jnv corresponding to the target Jn in the virtual space Sv by referring to the machine learning model 221. The terminal 4 acquires this three-dimensional data from the server device 2 and, based on this, can generate a virtual image Gnv that would be obtained if the virtual object Jnv indicated by the three-dimensional data were captured from an arbitrary (virtual) shooting point Pnv.

[0082] The process of generating a virtual image from the three-dimensional data is performed, for example, according to the following equation (3).

[0083]

[0084] Here, Ti is the transmittance of the position designated by i on the line of sight R, αi is the opacity at that position, and ci is the color at that position. Equation (3) indicates that by integrating the product of Ti, αi, and ci on the line of sight R, the color of a pixel in a virtual image formed on the imaging plane of the virtual viewpoint by light traveling along the line of sight R can be obtained. Note that the color of this pixel can be expressed in an RGB color space using the three primary colors of light, red, green, and blue, a YUV color space represented by a luminance signal Y and two color difference signals, a YCbCr color space, or the like.

[0085] In the prior art, the density of a given point in a virtual space was not constrained by setting a constant, as in the present invention. For example, in the case of NeRF, initial values ​​are often randomly assigned to the densities of all coordinates defined in the virtual space. Furthermore, because NeRF has high optimization stability, even if the densities of any of the coordinates are not specifically constrained, the densities tend to converge to desired values ​​as long as an appropriate number of learning sessions are performed. Therefore, in NeRF, it was not necessary to set the densities of any of the coordinates in the virtual space to a constant.

[0086] On the other hand, in the case of NeuS, an SDF representing the surface of a virtual object is extracted, and images without a background that show an object corresponding to the virtual object are often used as training data. Furthermore, NeuS generally has lower optimization stability and slower processing than NeRF. Therefore, when using a machine learning model to obtain a representation of the surface of a virtual object from an image, a masking process or the like is performed on the image in advance to remove information other than the object.

[0087] The learning device 1 of the virtual image generation system 9 according to the present invention trains a machine learning model that estimates a three-dimensional model of an object in a virtual space using images obtained by photographing the object from multiple viewpoints as training data, and trains the machine learning model in which the densities of background points and virtual viewpoints are constrained to be predetermined constants. Therefore, the learning device 1 does not need to perform processing to remove the background from each image constituting the training data during the training process of the machine learning model.

[0088] In particular, the present invention shown in the embodiment trains a machine learning model in which the densities at background points and the virtual viewpoint are constrained to be 1 and 0, respectively. The present invention is based on NeuS. In the case of NeuS, the density is always 0 somewhere along the line of sight R during the calculation process, so there is no need to ensure that the density of empty areas in the transmittance function T is 0. Therefore, in conventional technologies based on NeuS, it was not necessary to distinguish the virtual viewpoint from other coordinates and set its density value to 0. The present invention achieves the effect of making it easier for the density of empty areas in the transmittance function T to converge to 0, because the virtual viewpoint is distinguished from other coordinates and its density value is set to 0.

[0089] Furthermore, the server device of the present invention can generate three-dimensional data of an object from raw captured images (photographs, etc.) taken by a user using a smartphone or other device, using a trained machine learning model.

[0090] An unprocessed captured image also includes a background, but when this is handled by NeuS, light rays (light traveling along the line of sight R) emitted from any pixel that makes up the captured image will collide with some object, where the transmittance function T should fall to 0. The machine learning model trained by the present invention is constrained to have a density of 1 at background points, and therefore has the effect of easily obtaining a convergent solution where the transmittance function T becomes 0 at each position where each of the above-mentioned light rays collide with an object.

[0091] <Modifications> The above is a description of the embodiment, but the content of this embodiment can be modified as follows: In addition, the following modifications may be combined.

[0092] <1> In the above-described embodiment, the virtual image generation system 9 includes the learning device 1 and the server device 2, but one of these may also perform the functions of the other. For example, the server device 2 may store training data corresponding to the viewpoint image DB 120 described above in the memory 22. In this case, the processor 21 of the server device 2 may train the machine learning model 221 based on the training data.

[0093] <2> In the above-described embodiment, the processor 11 and the processor 21 are both a CPU or a GPU. However, they may be configured to cooperate with each other, or may have other configurations. For example, these processors may be or may include a field programmable gate array (FPGA). Furthermore, these processors may have an application specific integrated circuit (ASIC) or other programmable logic device.

[0094] <3> In the above-described embodiment, terminal 4 requests three-dimensional data of the photographed object, but it may also be possible to specify a virtual viewpoint from which to photograph the object and request a virtual image obtained when the object is photographed from that virtual viewpoint. In this case, server device 2 may generate, by rendering, a virtual image of the object in the virtual space represented by the three-dimensional data generated by the three-dimensional data generation means, photographed from the specified virtual viewpoint, and provide the generated image to terminal 4.

[0095] Fig. 12 is a diagram showing an example of the functional configuration of the server device 2 according to the modified example. The processor 21 of the server device 2 shown in Fig. 12 realizes a virtual image generation means 217 in addition to the functions realized by the processor 21 shown in Fig. 8. This virtual image generation means 217 generates a virtual image using the three-dimensional data generated by the three-dimensional data generation means 214.

[0096] In this case, the accepting unit 215 accepts, from the terminal 4, a designation of a virtual viewpoint from which to photograph an object in the virtual space. That is, the accepting unit 215 is an example of a accepting unit that accepts, from a terminal, a designation of a virtual viewpoint from which to photograph an object.

[0097] The virtual image generation means 217 generates a virtual image obtained when an object represented by the three-dimensional data is photographed from a specified virtual viewpoint, based on the specification received by the reception means 215. In other words, the virtual image generation means 217 is an example of virtual image generation means that generates a virtual image obtained when an object represented by the three-dimensional data generated by the three-dimensional data generation means is photographed from a virtual viewpoint indicated by the specification received from the terminal.

[0098] Then, the supply means 216 supplies the generated virtual image to the terminal 4. In other words, the supply means 216 is an example of a supply means that supplies the virtual image generated by the virtual image generation means in accordance with the specification received from the terminal to the terminal. In this case, too, the terminal 4 can obtain a virtual image of the photographed object as seen from the specified virtual viewpoint.

[0099] <4> In short, in one aspect, the program of the present invention causes a computer including one or more processors to execute the following steps: acquiring training data associating a plurality of viewpoints with images obtained by photographing an object from each of the plurality of viewpoints; generating, in a virtual space, a virtual image obtained when a virtual object corresponding to an object is photographed from a virtual viewpoint corresponding to the viewpoint, using a machine learning model including a density for each position on the line of sight that runs from a background point in the background of the virtual object through the virtual object to the virtual viewpoint; and comparing, for each virtual viewpoint, the generated virtual image with an image photographed from a viewpoint corresponding to the virtual viewpoint, and training the machine learning model; and constraining the machine learning model so that the densities at the background points and the virtual viewpoints are each predetermined constants.

[0100] In another aspect, the program of the present invention may be configured to cause a computer consisting of one or more processors to execute the steps of acquiring photographed images of an object from multiple shooting points, estimating the shooting points based on feature points extracted from each of the photographed images, and generating three-dimensional data of the object using the estimated shooting points, the photographed images corresponding to the shooting points, and a machine learning model trained in a learning device by a program executed by the processor 11.

[0101] 1...learning device, 11...processor, 111...acquisition means, 112...generation means, 113...learning means, 12...memory, 120...viewpoint image DB, 121...machine learning model, 13...interface, 2...server device, 21...processor, 211...captured image acquisition means, 212...feature point extraction means, 213...estimation means, 214...stereoscopic data generation means, 215...reception means, 216...supply means, 217...virtual image generation means, 22...memory, 221...machine learning model, 222...captured image DB, 23...interface, 3...communication line, 4...terminal, 5...camera, 9...virtual image generation system.

Claims

1. A learning device comprising: an acquisition means for acquiring teacher data that associates a plurality of viewpoints with images obtained by photographing an object from each of the plurality of viewpoints; a generation means for generating, in a virtual space, a virtual image obtained when a virtual object corresponding to the object is photographed from a virtual viewpoint corresponding to the viewpoint, using a machine learning model including a density for each position on the line of sight that passes from a background point in the background of the virtual object to the virtual viewpoint, the virtual image being obtained by photographing the virtual object from a virtual viewpoint corresponding to the viewpoint; and a learning means for comparing, for each of the virtual viewpoints, the generated virtual image with the image photographed from the viewpoint corresponding to the virtual viewpoint, and training the machine learning model, wherein the machine learning model is constrained so that the densities at the background points and the virtual viewpoints are each predetermined constants.

2. The learning device described in claim 1, characterized in that the machine learning model includes a surface function that defines the surface of the virtual object using a multi-layer perceptron and a density function that indicates the density using the surface function, the learning means trains the machine learning model by updating the surface function using a backpropagation method, and the density function is constrained so that the density at the background points is 1 and the density at the virtual viewpoint is 0.

3. The learning device described in claim 2, characterized in that in the machine learning model, a transmittance function T indicating the transmittance of light traveling along the line of sight to the virtual viewpoint, and an opacity function α indicating the opacity at an arbitrary position on the line of sight are expressed by equations (1) and (2) when a density function Φ indicating the density at an arbitrary position on the line of sight, and a surface function f defining the surface of the virtual object are used.

4. A server device having: an image acquisition means for acquiring images of an object taken from a plurality of shooting points; an estimation means for estimating the shooting points based on feature points extracted from each of the captured images; and a three-dimensional data generation means for generating three-dimensional data of the object using the estimated shooting points, the captured images corresponding to the shooting points, and a machine learning model described in any one of claims 1 to 3.

5. A server device according to claim 4, further comprising: a receiving means for receiving a request from a terminal; and a supplying means for supplying the three-dimensional data generated by the three-dimensional data generating means in response to the request to the terminal.

6. A server device as described in claim 4, characterized in that it comprises: an accepting means for accepting, from a terminal, a designation of a virtual viewpoint for photographing the object; a virtual image generating means for generating a virtual image obtained when the object represented by the three-dimensional data generated by the three-dimensional data generating means is photographed from the virtual viewpoint designated by the designation; and a supplying means for supplying the virtual image generated by the virtual image generating means in response to the designation to the terminal.

7. A program for causing a computer to execute the following steps: acquiring training data associating a plurality of viewpoints with images obtained by photographing an object from each of the plurality of viewpoints; generating, in a virtual space, a virtual image obtained when a virtual object corresponding to the object is photographed from a virtual viewpoint corresponding to the viewpoint, using a machine learning model including a density for each position on the line of sight from a background point in the background of the virtual object through the virtual object to the virtual viewpoint; and comparing, for each virtual viewpoint, the generated virtual image with the image photographed from the viewpoint corresponding to the virtual viewpoint, to train the machine learning model; and constraining the machine learning model so that the densities at the background points and the virtual viewpoints are each predetermined constants.

8. A program for causing a computer to execute the steps of: acquiring images of an object taken from a plurality of shooting points; estimating the shooting points based on feature points extracted from each of the captured images; and generating three-dimensional data of the object using the estimated shooting points, the captured images corresponding to the shooting points, and a machine learning model trained by the program described in claim 7.

Citation Information

Patent Citations

  • Three dimensional model generation apparatus and method

    JP2021071749A

  • Method for interactively editing geometry of neural radiation field

    CN114429538A