Three-dimensional data generation device and drawing device

The 3D data generation device addresses the loss of optical property information in NeRF technology by using a learning model to integrate color and density data from multiple viewpoints, ensuring accurate representation of object properties like glossiness and transparency.

JP2024169203APending Publication Date: 2024-12-05NTT DOCOMO INC

Patent Information

Application Number
JP2023086469
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-25
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing 3D data generation methods using NeRF technology fail to accurately capture the optical properties of objects, such as glossiness and transparency, as they lose information about the direction of the line of sight when determining the color of each point on the object's surface.

Method used

A three-dimensional data generation device that utilizes a learning model to generate point cloud information and color information based on a luminance field, incorporating the optical properties of objects by integrating color and density data from multiple viewpoints.

Benefits of technology

The solution enables the addition of optical characteristics to the generated 3D data, allowing for the accurate representation of an object's optical properties, such as glossiness and transparency, by correlating color information with the viewing angle.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024169203000001_ABST
    Figure 2024169203000001_ABST
Patent Text Reader

Abstract

To provide a three-dimensional data generation device capable of adding data representing optical properties of an object to three-dimensional data generated form the object.SOLUTION: A point cloud information generation unit 212 generates point cloud information including three-dimensional coordinates of each point belonging to a composite point cloud PGC using a learning model M1. A color information generation unit 213 generates color information representing colors corresponding to multiple viewing points at each point when observing each point belonging to the composite point cloud PGC from multiple viewing points using the learning model M1. An output unit 214 outputs the point cloud information and the color information as three-dimensional data representing a three-dimensional shape and color of an object OB.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to a three-dimensional data generating device and a rendering device. [Background technology]

[0002] Patent Document 1 discloses a technique for extracting a three-dimensional point cloud of a reference object based on a multi-viewpoint image of the reference object captured from a plurality of viewpoints, deriving normals of the extracted three-dimensional point cloud of the reference object, and deriving shading information based on the three-dimensional point cloud and the normals of the three-dimensional point cloud. Measurements using photogrammetry and LiDAR (Light Detection and Ranging) sensors are known as techniques for capturing three-dimensional data of a reference object based on a multi-viewpoint image of the reference object. With the above techniques, optical properties such as glossiness and transparency of an object may not be correctly recognized.

[0003] On the other hand, according to the recent NeRF (Neural Radiance Fields) technology, a learning model is created based on a plurality of images obtained by photographing an object from various angles, and the image of the object seen from the viewpoint is estimated by inputting the viewpoint position and the line of sight direction to the created learning model. This makes it possible to continuously draw two-dimensional images seen from any viewpoint while changing the viewpoint. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2014-120079 A [Non-patent literature]

[0005] [Non-Patent Document 1] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, "NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis", arXiv:2003.08934, 2020 (https: / / arxiv.org / abs / 2003.08934). Summary of the Invention [Problem to be solved by the invention]

[0006] When generating 3D data from a learning model obtained by NeRF technology, the color of each point on the object's surface is obtained as the average of the colors of corresponding points in multiple images taken from various angles. As a result, the information indicating the color of each point on the object's surface loses information about the direction of the line of sight. Therefore, although the optical properties of the object are reflected in the drawing using NeRF technology, there was an issue that information about the optical properties of the object is lost when generating 3D data from the learning model.

[0007] The present invention has been made to solve the above-mentioned problems, and aims to add data representing the optical characteristics of an object to the generated 3D data of the object based on data including optical characteristics acquired using NeRF technology. [Means for solving the problem]

[0008] A three-dimensional data generating device according to a preferred embodiment of the present invention includes a point cloud information generating unit that generates point cloud information including three-dimensional coordinates of each point belonging to a point cloud, which is a collection of multiple points located on the surface of an object, using a learning model that has learned a luminance field, which is a vector field related to color and density in a three-dimensional space including an object; a color information generating unit that uses the learning model to generate color information indicating each color corresponding to a plurality of viewpoints at each point belonging to the point cloud when each of the points is observed from a plurality of viewpoints; and an output unit that outputs the point cloud information and the color information as three-dimensional data indicating the three-dimensional shape and color of the object.

[0009] A virtual object drawing device in a preferred aspect of the present invention includes: a first acquisition unit that acquires point cloud information including three-dimensional coordinates of each point belonging to a point cloud, which is a collection of a plurality of points located on a surface of an object; three-dimensional data including pairs of color information indicating each color corresponding to the plurality of viewpoints at each point and positions of each viewpoint when each of the points is observed from a plurality of viewpoints; a second acquisition unit that acquires viewpoint information indicating the position of a user's viewpoint in a virtual space and coordinate information indicating first coordinates at which a virtual object corresponding to the object is located in the virtual space; and a drawing unit that draws the virtual object at the first coordinates in the virtual space based on the three-dimensional data, the viewpoint information, and the coordinate information. Effect of the Invention

[0010] According to the three-dimensional data generation device of the present invention, data representing the optical characteristics of an object can be added to three-dimensional data generated from the object based on data including optical characteristics acquired using NeRF technology. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram showing the overall configuration of an information processing system including a three-dimensional data generation device and a drawing device according to a first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of the machine learning device in FIG. 1. [Diagram 3]FIG. 1 is a diagram illustrating an example of a method for acquiring training data for a machine learning device. [Figure 4] FIG. 1 illustrates the neural network architecture for learning NeRF brightness fields. [Diagram 5] 2 is a block diagram showing an example of the configuration of the three-dimensional data generating device shown in FIG. 1. [Figure 6] FIG. 13 is a diagram for explaining a composite point group. [Figure 7] 1A to 1C are diagrams illustrating point clouds for each viewpoint when an object is observed from a viewpoint above the object. [Figure 8] 1A to 1C are diagrams illustrating point clouds for each viewpoint when an object is observed from a viewpoint below the object. [Figure 9] 13A and 13B are diagrams illustrating point clouds for each viewpoint when an object is observed from a viewpoint to the right of the object. [Figure 10] 11A and 11B are diagrams showing point clouds for each viewpoint when an object is observed from a viewpoint to the left of the object. [Figure 11] FIG. 2 is a schematic diagram illustrating an example of a composite point cloud of an object. [Figure 12] 1A to 1C are schematic diagrams illustrating an example of a point in a composite point cloud of an object and a point cloud for each viewpoint. [Figure 13] 1A to 1C are schematic diagrams showing one example of a synthetic point cloud of an object and another example of point clouds for each viewpoint. [Figure 14] FIG. 4 is a diagram showing an example of three-dimensional data output by an output unit. [Figure 15] 15 is a diagram showing positions of points in the three-dimensional data of FIG. 14 on an object. FIG. [Figure 16] 2 is a block diagram showing an example of the configuration of the rendering device shown in FIG. 1; [Figure 17] 2 is a block diagram showing an example of the configuration of a server in FIG. 1. [Figure 18] 6 is a flowchart showing the operation of the processing device in FIG. 5. [Figure 19] 20 is a flowchart showing the operation of a subroutine corresponding to step S11 in FIG. 18. [Figure 20] 20 is a flowchart showing the operation of a subroutine corresponding to step S12 in FIG. 18. [Figure 21] 12 is a flowchart showing the operation of the processing device in FIG. [Figure 22] FIG. 11 is a diagram showing an overall configuration of an information processing system according to a second embodiment. [Figure 23] FIG. 11 is a block diagram showing an example of the configuration of a three-dimensional data generating device according to a second embodiment. [Figure 24] 11A to 11C are diagrams for explaining a method of forming a mesh by a mesh forming unit. [Diagram 25] 1A to 1C are schematic diagrams illustrating an example of a point in a composite point cloud of an object and a point cloud for each viewpoint. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] 1. First embodiment Hereinafter, the configurations of a three-dimensional data generation device and a rendering device according to a first embodiment of the present invention will be described with reference to FIGS.

[0013] 1.1. Configuration of the First Embodiment Overall Configuration FIG. 1 is a diagram showing the overall configuration of an information processing system including a three-dimensional data generating device and a rendering device according to the first embodiment.

[0014] The information processing system 1 includes a machine learning device 10, a three-dimensional data generating device 20, a rendering device 30, a server 40 as an external device, and a communication network NET. The rendering device 30 represents one of two rendering devices 30-1 and 30-2.

[0015] In the information processing system 1, the machine learning device 10, the three-dimensional data generation device 20, the drawing device 30-1, the drawing device 30-2, and the server 40 are communicatively connected to each other via a communication network NET. Note that, although two drawing devices 30-1 and 30-2 are illustrated in Fig. 1, this number is merely an example, and the information processing system 1 can include any number of drawing devices 30.

[0016] In the machine learning device 10, a learning model is created using NeRF technology based on data obtained by photographing a real space including a target object from multiple viewpoints, to estimate the color and density of the real space viewed from an arbitrary viewpoint. The three-dimensional data generation device 20 generates three-dimensional data based on the learning model created by the machine learning device 10. The drawing device 30 draws a three-dimensional model of an object in a virtual space using the three-dimensional data of the object generated by the three-dimensional data generation device 20. The server 40 stores various data and provides various data to other devices connected to the communication network NET via the communication network NET.

[0017] That is, the information processing system 1 is a system that provides a user who uses the drawing device 30 with three-dimensional data of an object to be used for drawing in a virtual space.

[0018] 1.1.2. Machine learning device configuration Fig. 2 is a block diagram showing an example configuration of the machine learning device 10 in Fig. 1. The machine learning device 10 includes a processing device 11, a storage device 12, a communication device 13, an imaging device 14, a display 15, and an input device 16. The elements included in the machine learning device 10 are connected to each other by one or more buses for communicating information. Note that the term "apparatus" in this specification may be replaced with other terms such as circuit, device, unit, etc.

[0019] The processing device 11 is a processor that controls the entire machine learning device 10, and is configured, for example, using a single chip or multiple chips. The processing device 11 is configured, for example, using a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, and a register. Note that some or all of the functions of the processing device 11 may be realized by hardware such as a digital signal processor (DSP), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA). The processing device 11 executes various processes in parallel or sequentially.

[0020] The storage device 12 is a recording medium that can be read and written by the processing device 11. The storage device 12 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a Read Only Memory (ROM), an Erasable Programmable Read Only Memory (EPROM), and an Electrically Erasable Programmable Read Only Memory (EEPROM). The volatile memory is, for example, a Random Access Memory (RAM).

[0021] The storage device 12 stores a learning model M1 and a plurality of programs including a control program PR1 to be executed by the processing device 11. The storage device 12 also includes a teacher data storage unit 121.

[0022] The teacher data storage unit 121 stores teacher data. The teacher data is multi-view image data of an object. The multi-view image data is a collection of multiple image data captured by using the imaging device 14 to capture images of an object from various directions around the object. The trained model storage unit 122 stores the trained model as a trained model M1. The storage device 12 also functions as a work area for the processing device 11.

[0023] The learning model M1 stored in the storage device 12 is provided to the three-dimensional data generating device 20 via a communication network NET or a recording medium. The learning model M1 may be provided to an external computer such as a server 40 or a cloud-type computer and stored in a storage device of the external computer.

[0024] The communication device 13 is hardware as a transmitting / receiving device for communicating with other devices. The communication device 13 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 13 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 13 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth (registered trademark), etc.

[0025] The imaging device 14 outputs imaging information obtained by imaging the outside world. The imaging device 14 includes, for example, a lens, an imaging element, an amplifier, and an AD converter. Light collected through the lens is converted into an imaging signal, which is an analog signal, by the imaging element. The amplifier amplifies the imaging signal and outputs it to the AD converter. The AD converter converts the amplified imaging signal, which is an analog signal, into imaging information, which is a digital signal. The converted imaging information is output to the processing device 11.

[0026] The imaging device 14 captures images of an object from various directions by moving the imaging device 14 around the object or by rotating the object, and generates a large amount of imaging data. That is, the imaging device 14 generates multi-view image data. The imaging device 14 may be a single-lens reflex camera or a camera provided in a smartphone.

[0027] The display 15 is a device that displays images and text information. The display 15 displays various images under the control of the processing device 11. For example, various display panels such as a liquid crystal panel and an organic EL (Electro Luminescence) panel are suitably used as the display 15.

[0028] The input device 16 receives an operation from a user U1. The user U1 is a provider of a three-dimensional learning model of an object. For example, the input device 16 includes a keyboard, a touch pad, a touch panel, and a pointing device such as a mouse. Here, when the input device 16 includes a touch panel, it may also serve as the display 15.

[0029] The processing device 11 functions as an acquisition unit 111 and a generation unit 112, for example, by reading out a control program PR1 from the storage device 12 and executing it.

[0030] The acquiring unit 111 acquires a plurality of pieces of teacher data, each of which is configured by associating input data with output data, from the imaging device 14. The plurality of pieces of teacher data acquired by the acquiring unit 111 are stored in the teacher data storage unit 121.

[0031] The generation unit 112 generates a trained learning model M1 by having the learning model M1 learn a plurality of pieces of teacher data. More specifically, the generation unit 112 performs machine learning using the teacher data stored in the teacher data storage unit 121. That is, the generation unit 112 inputs a plurality of pieces of teacher data to the learning model M1, thereby having the learning model M1 learn the correlation between the input data and the output data that constitute the teacher data. In this way, the generation unit 112 generates a trained learning model M1.

[0032] FIG. 3 is a diagram showing an example of a method for acquiring teacher data for the machine learning device 10. In FIG. 3, a mouse is shown as an example of an object OB. In FIG. 3, the real space RS is shown as a rectangular parallelepiped. The object OB is included in the real space RS. The imaging device 14 captures the real space RS including the object OB. The image captured by the imaging device 14 can be considered as an image in which the object OB is projected onto a virtual screen SC arranged between the imaging device 14 and the object OB. A line VL1 extending from the imaging device 14 represents the line of sight of the imaging device 14. The line VL1 starts from an intersection RP1 between the line VL1 and the object OB and passes through the object OB.

[0033] An intersection SP1 between the virtual screen SC and the line VL1 corresponds to a point RP1 on the surface of the object OB. Also, a point RP0 is an intersection between the entrance surface of the line VL1 in the real space RS and the line VL1, and a point RP1 is an intersection between the exit surface of the line VL1 in the real space RS and the line VL1.

[0034] NeRF is a technique for training a function that represents a three-dimensional object through differentiable three-dimensional CG (Computer Graphics) rendering. The function that represents a three-dimensional object is a neural network, which corresponds to the learning model M1 in this embodiment. The data set required for training the neural network is only images of the object OB captured from each viewpoint using the imaging device 14 and pose information at the viewpoints. The pose information is information related to the position, orientation, and focal length of the imaging device 14.

[0035] NeRF is a research project published in March 2020 by researchers at the University of California, Berkeley (Non-Patent Document 1). In line with the gist of Non-Patent Document 1, radiance fields are vector fields that associate color and density with each coordinate in space. In other words, radiance fields are vector fields related to color and density in a three-dimensional space including an object OB.

[0036] Here, density is the opacity at a point indicated by each coordinate, i.e., an index indicating the presence of an object OB at the point of the coordinate. Therefore, according to the NeRF technique, a luminance field is obtained which indicates that the density at the coordinate of a position where an object OB exists is higher than the density at the coordinate of a position where an object OB does not exist.

[0037] The basic concept of NeRF is to approximate the luminance field by a neural network. To approximate the luminance field, for example, a method of training using pairs of multiple coordinates and luminance at the coordinates that are created in advance based on multiple captured images is considered. However, this method requires a lot of time to create a dataset. Therefore, in NeRF, an image is rendered using the luminance field represented by a neural network, and the luminance field is trained by comparing the rendering result with the actually captured image.

[0038] According to the NeRF technology, when the coordinates (x, y, z) of a point and the line of sight angle (θ, φ) are given to a neural network that has already learned the luminance field, that is, a learning model M1, the color (R, G, B) of the point on the line of sight and the density σ are output. As can be seen from this, the learning model M1 by NeRF does not have information on the surface of the object OB.

[0039] Image rendering is the integration of color information at each point on the line VL1 weighted by the density σ at each point. For example, in the example shown in FIG. 3, color information in the section IS0 from point RP0 to point RP2 is integrated. Since no object OB exists in the section from point RP0 to just before point RP1, the density σ in this section is substantially zero. Therefore, the color information in the section from point RP0 to just before point RP1 is not substantially reflected in the rendering result. Furthermore, the density σ at point RP1 is relatively high, and the color at point RP1 is dominant in the rendering result.

[0040] We define the neural network that represents the luminance field as F Θ Then, F Θis given by the following equation (1).

[0041] F Θ :(x,y,z,θ,φ)→(R,G,B,σ) ···(1)

[0042] Below is the vector (x,y,z) that indicates the location information. T is represented as "x", and the vector (θ,φ) indicating angle information is represented as "d".

[0043] 4 is a diagram showing the architecture of a neural network for learning a luminance field. The neural network of this embodiment is composed of a fully-connected multi-layer perceptron.

[0044] As shown in FIG. 4, three input vectors IV1, IV2, and IV3 are provided to the intermediate layers HL1 to HL10 of the neural network, and two output vectors OV1 and OV2 are output. The numbers in each block of the intermediate layers HL1 to HL10 indicate the dimension of the vector. Each arrow indicates a transformation between layers. In FIG. 4, solid arrows, dashed arrows, and dashed arrows are shown according to differences in activation functions. The solid arrows indicate nonlinear transformation using a rectified linear function (ReLU: Rectified Linear Unit), and the dashed arrows indicate nonlinear transformation using a sigmoid function. The dashed arrows indicate linear transformation without using an activation function.

[0045] Additionally, "+" represents the concatenation of vectors. As shown in Figure 4, the position information x is input to two places, the first stage of the network and the vector in the fifth stage. Additionally, the angle information d is input after the density σ is output in the vector in the ninth stage. In this way, the density σ is not affected by the angle information d. This means that while it changes depending on the color (R, G, B) and the angle of gaze, the density σ does not change depending on the angle of gaze.

[0046] 1.1.3.Configuration of 3D data generation device Fig. 5 is a block diagram showing an example of the configuration of the three-dimensional data generation device 20 of Fig. 1. As shown in Fig. 5, the three-dimensional data generation device 20 includes a processing device 21, a storage device 22, a communication device 23, a display 24, and an input device 25. The elements of the three-dimensional data generation device 20 are connected to each other by one or more buses for communicating information.

[0047] The processing device 21 is a processor that controls the entire 3D data generation device 20, and is configured, for example, using one or more chips. The processing device 21 is configured, for example, using a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, and a register. Note that some or all of the functions of the processing device 21 may be realized by hardware such as a DSP, an ASIC, a PLD, or an FPGA. The processing device 21 executes various processes in parallel or sequentially.

[0048] The storage device 22 is a recording medium that can be read and written by the processing device 21. The storage device 22 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM.

[0049] The storage device 22 stores a plurality of programs including a control program PR2 to be executed by the processing device 21. The storage device 22 also functions as a work area for the processing device 21.

[0050] The communication device 23 is hardware as a transmitting / receiving device for communicating with other devices. The communication device 23 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 23 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 23 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, USB, etc. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth, etc.

[0051] The display 24 is a device that displays images and text information. The display 24 displays various images under the control of the processing device 21. For example, various display panels such as a liquid crystal panel and an organic EL panel are suitably used as the display 24.

[0052] The input device 25 receives an operation from a user U2. The user U2 is an operator who generates three-dimensional data of an object OB. The user U2 may be the same as the user U1. For example, the input device 25 includes a keyboard, a touch pad, a touch panel, and a pointing device such as a mouse. Here, when the input device 25 includes a touch panel, it may also function as the display 24.

[0053] The processing device 21 functions as an acquisition unit 211, a point cloud information generation unit 212, a color information generation unit 213, an output unit 214, and a display control unit 215 by reading and executing a control program PR2 from the storage device 22.

[0054] The acquisition unit 211 acquires the learning model M1 from the machine learning device 10. The learning model M1 is a model that has already learned the luminance field of the real space RS that includes the object OB.

[0055] The point cloud information generating unit 212 generates point cloud information using a learning model M1 that has already learned the luminance field. The point cloud information is information including the three-dimensional coordinates of each point belonging to the composite point cloud. The composite point cloud is a collection of multiple points located on the surface of the object OB when the object OB is observed from multiple viewpoints. The composite point cloud is a point cloud obtained by combining multiple points observed from multiple viewpoints.

[0056] 6 is a diagram for explaining the composite point cloud. As described above, when the line VL1 is directed toward the object OB, the density σ in the section from point RP0 to just before point RP1 is relatively low. That is, the transparency in that section is relatively high. However, at point RP1 where the line VL1 penetrates the object OB, that is, at a point on the surface of the object OB, the transparency drops sharply, that is, the density σ increases sharply. Therefore, the point cloud information generating unit 212 can estimate the coordinate position of the surface of the object OB from the degree of change in density σ estimated by NeRF.

[0057] The point cloud information generating unit 212 divides the real space RS including the object OB at a predetermined interval. For example, when the point cloud information generating unit 212 divides the real space RS into a plurality of minute cubes, the point cloud information generating unit 212 calculates the color (R, G, B) and density σ at each vertex of the plurality of minute cubes using the learning model M1.

[0058] At this time, in order to calculate the density σ of the entire surface of the object OB, the point cloud information generating unit 212 calculates the colors (R, G, B) and density σ at the vertices of each cube when the vertices are viewed from various viewpoints. The various viewpoints are those realized by rotating the viewpoints VP1 and VP2 around the object OB as shown in FIG. 6. This allows the point cloud information generating unit 212 to observe from any viewpoint within 360 degrees around the object OB. Preferably, the point cloud information generating unit 212 calculates the density σ when the viewpoints VP1 and VP2 are each changed by 10 degrees.

[0059] Since each vertex is observed from multiple viewpoints, multiple colors (R, G, B) and densities σ are calculated for each vertex. The point cloud information generation unit 212 synthesizes the calculation results of the colors (R, G, B) and densities σ that overlap at the same vertex by averaging or the like. The point cloud information generation unit 212 calculates the density σ by averaging the calculated density σ. th A set of points exceeding this number is extracted as a composite point cloud PGC. In FIG. 6, each point of the extracted composite point cloud PGC is indicated by a black circle. The color (R, G, B) at each point of the extracted composite point cloud PGC is the average value of the colors obtained by overlapping at each vertex. Information about this average value is called average color information.

[0060] The average color information can be said to be information indicating the color obtained by averaging the colors at each point belonging to the composite point group PGC when observed from a certain number of viewpoints or more. In other words, the color information at each point of the composite point group PGC obtained by the point group information generating unit 212 does not include information that depends on the angle of the "line of sight." The composite point group PGC is an example of a "first point group."

[0061] The color information generating unit 213 generates color information using the learning model M1. The color information is information indicating a plurality of colors at each point when each point belonging to the composite point group PGC is observed from a plurality of viewpoints.

[0062] More specifically, the color information generating unit 213 observes each point belonging to the composite point group PGC from multiple viewpoints. That is, the color information generating unit 213 uses the learning model M1 to extract multiple points located on the surface of the object OB when the object OB is observed from a certain viewpoint. As with the method by the point group information generating unit 212, the points located on the surface of the object OB are extracted when the density σ is equal to or smaller than the threshold σ th A set of points located on the surface of the object OB when the object OB is observed from each of a plurality of viewpoints is hereinafter referred to as a point cloud for each viewpoint.

[0063] As described above, when extracting the composite point group PGC, it is preferable to change the viewpoint by 10 degrees at a time, and it is also preferable to change the viewpoint by 10 degrees at a time when extracting the point group for each viewpoint. However, to simplify the explanation, the following explanation will be given by exemplifying only four representative viewpoints, namely, top, bottom, left, and right.

[0064] Note that changing the viewpoint by 10 degrees is merely an example, and the angle at which the viewpoint is changed is not particularly limited. For example, the angle at which the viewpoint is changed may be changed according to the physical size of the object OB. If the angle at which the viewpoint is changed is constant, the larger the physical size of the object OB, the greater the number of points in the composite point group PGC and the point group for each viewpoint. Therefore, the larger the physical size of the object OB, the larger the angle at which the viewpoint is changed, thereby suppressing an increase in the number of viewpoint points. In addition, when the color and brightness of the object OB change significantly depending on the angle of the viewpoint, the smaller the angle at which the viewpoint is changed, the more the color change that is closer to reality is reproduced.

[0065] Fig. 7 is a diagram showing a point cloud for each viewpoint when the object OB is observed from a viewpoint above the object OB. Fig. 8 is a diagram showing a point cloud for each viewpoint when the object OB is observed from a viewpoint below the object OB. Fig. 9 is a diagram showing a point cloud for each viewpoint when the object OB is observed from a viewpoint to the right of the object OB. Fig. 10 is a diagram showing a point cloud for each viewpoint when the object OB is observed from a viewpoint to the left of the object OB.

[0066] As shown in Fig. 7, when an object OB is observed from a viewpoint VPU above the object OB, the color information generating unit 213 extracts each point on the upper surface of the object OB as a point cloud PGV1 for each viewpoint, but is unable to extract points on the lower surface of the object OB. As shown in Fig. 8, when an object OB is observed from a viewpoint VPB below the object OB, the color information generating unit 213 extracts each point on the lower surface of the object OB as a point cloud PGV2 for each viewpoint, but is unable to extract points on the upper surface of the object OB.

[0067] As shown in Fig. 9, when an object OB is observed from a viewpoint VPR to the right of the object OB, the color information generating unit 213 extracts each point on the right surface of the object OB as a point cloud PGV3 for each viewpoint, but is unable to extract points on the left surface of the object OB or points on the bottom surface of the object OB. As shown in Fig. 10, when an object OB is observed from a viewpoint VPL to the left of the object OB, the color information generating unit 213 extracts each point on the left surface of the object OB as a point cloud PGV4 for each viewpoint, but is unable to extract points on the right surface of the object OB or points on the bottom surface of the object OB. The point clouds PGV1 to VPG4 for each viewpoint are examples of the "second point cloud".

[0068] In addition, the coordinate positions of each point of the composite point group PGC estimated by the point group information generation unit 212 and the coordinate positions of each point of the point group for each viewpoint estimated by the color information generation unit 213 do not necessarily match due to estimation errors of the density σ using the learning model M1, etc.

[0069] The operation of the color information generating unit 213 will be described in more detail below with reference to Figs. 11 to 13. Fig. 11 is a schematic diagram showing an example of a composite point group PGC of an object OB. Fig. 12 is a schematic diagram showing one point of the composite point group PGC of an object OB and one example of a point group PGV for each viewpoint. Fig. 13 is a schematic diagram showing one point of the composite point group PGC of an object OB and another example of a point group PGV for each viewpoint. Note that the point group PGV for each viewpoint is an example of a "second point group".

[0070] 11, focusing on one target point Ps1 of the composite point group PGC, the space within a radius L0 (mm) centered on the target point Ps1 is defined as S1. In this manner, the color information generating unit 213 defines a space S1 within a radius L0 centered on each point of the composite point group PGC. Note that the radius L0 is an example of a "predetermined distance."

[0071] Next, the color information generating unit 213 defines a space S2 for each point belonging to the point group PGV for each viewpoint. The space S2 is a space within a radius L0 (mm) centered on the target point Ps1, similar to the space S1. The color information generating unit 213 searches for points included in the space S2 among the points belonging to the point group PGV for each viewpoint. As shown in FIG. 12, the points included in the space S2 are Pr1, Pr2, and Pr3.

[0072] The color information generating unit 213 generates color information for the target point Ps1 by averaging the colors of the points Pr1, Pr2, and Pr3 that exist in the space S2 among the points belonging to the point group PGV for each viewpoint. For example, the color information C1 for the point Pr1 is [R1, G1, B1], the color information C2 for the point Pr2 is [R2, G2, B2], the color information C3 for the point Pr3 is [R3, G3, B3], and the color information C4 for the target point Ps1 is [R4, G5, B6]. s1 [R s1 ,G s1 ,B s1 ], the color information generating unit 213 generates color information C s1 =[R s1 ,G s1 ,B s1 ] is calculated using the following formula (2).

[0073] C s1 =(C1+C2+C3) / 3 (2)

[0074] If the number of points in the space S2 is n, the color information at any point Prn is expressed as C n =[R n ,G n ,B n ], then color information Cs1 at the target point Ps1 is calculated by the following equation (3).

[0075] C s1 =(C1+C2+···+C n ) / n ···(3)

[0076] 13, as a result of the search, there are cases where none of the points belonging to the point group PGV for each viewpoint exists in the space S2. In such a case, the color information generating unit 213 sets the color of the target point Ps1 to the color corresponding to the target point Ps1 in the point group information, rather than the color of each point belonging to the point group PGV for each viewpoint.

[0077] That is, when there is no point within a radius L0 of each point belonging to the composite point group PGC among the points belonging to the point group PGV for each viewpoint, the color information generating unit 213 generates average color information as color information.

[0078] The output unit 214 outputs the point cloud information and color information as three-dimensional data indicating the three-dimensional shape and color of the object OB. The function of the output unit 214 will be described below with reference to Fig. 14 and Fig. 15. Fig. 14 is a diagram showing an example of three-dimensional data output by the output unit 214. Fig. 15 is a diagram showing the positions of points P1, P2, and P3 on the object OB in the three-dimensional data of Fig. 14.

[0079] The three-dimensional data output by the output unit 214 is represented in a list format as shown in Fig. 14, in which the ID of each point of the composite point group, the coordinates of each point, a viewpoint, and a color corresponding to the viewpoint are associated with each other. As shown in Fig. 15, point P1 is located on the top surface of the object OB and is located in the center in the left-right direction. As can be understood from the explanation using Figs. 7 to 10, point P1 is observed from a viewpoint VPR from the right of the object OB, a viewpoint VPL from the left of the object OB, and a viewpoint VPU from above the object OB, but is not observed from a viewpoint VPB from below the object OB.

[0080] In FIG. 14, the first line of the list contains the color (R 11 ,G 11 ,B 11 ) and the coordinates (x1, y1, z1) of point P1. The second line of the list shows the color (R 12 ,G 12 ,B12 ) and the coordinates (x1, y1, z1) of point P1. The third line of the list shows the color (R 13 ,G 13 ,B 13 ) and the coordinates (x1, y1, z1) of point P1 are shown. Since point P1 cannot be observed from viewpoint VPB from below object OB, the fourth line of the list shows the average color (R1, G1, B1) corresponding to the coordinates (x1, y1, z1) of point P1 based on the average color information, and the coordinates (x1, y1, z1) of point P1. In this way, the color of point P1 differs depending on the direction of the viewpoint. If there is a viewpoint from which point P1 is in a blind spot, the average color of point P1 is complemented based on the average color information as the color when point P1 is observed from the viewpoint in the blind spot. In addition, since the coordinates of point P1 are coordinates included in the point cloud information related to the composite point cloud PGC, the same coordinates are adopted for any viewpoint.

[0081] Point P2 is located on the inclined plane to the right of object OB. Therefore, point P2 can be observed from the viewpoint VPR to the right of object OB and from the viewpoint VPU above object OB, but cannot be observed from the viewpoint VPL to the left of object OB or from the viewpoint VPB below object OB.

[0082] In FIG. 14, the fifth line of the list indicates the color (R 21 ,G 21 ,B 21 ) and the coordinates (x2, y2, z2) of point P2. Since point P2 cannot be observed from the viewpoint VPL from the left of the object OB, the sixth line of the list shows the average color (R2, G2, B2) corresponding to the coordinates (x2, y2, z2) of point P2 based on the average color information, and the coordinates (x2, y2, z2) of point P2. The seventh line of the list shows the color (R2, G2, B2) of point P2 when observed from the viewpoint VPU from above the object OB. 22 ,G 22 ,B 22) and the coordinates (x2, y2, z2) of point P2 are shown. Since point P2 cannot be observed from viewpoint VPB from below object OB, the 8th line of the list shows the average color (R2, G2, B2) corresponding to the coordinates (x2, y2, z2) of point P2 based on the average color information, and the coordinates (x2, y2, z2) of point P2. In this way, the color of point P2 differs depending on the direction of the viewpoint. If there is a viewpoint from which point P2 is in a blind spot, the average color of point P2 is complemented based on the average color information as the color when point P2 is observed from the viewpoint in the blind spot. In addition, since the coordinates of point P2 are coordinates included in the point cloud information regarding the composite point cloud, the same coordinates are adopted for any viewpoint.

[0083] Point P3 is located on the inclined plane on the left side of object OB. Therefore, point P3 can be observed from viewpoint VPL from the left of object OB and viewpoint VPU from below object OB, but cannot be observed from viewpoint VPR from the right of object OB or viewpoint VPB from above object OB.

[0084] In Fig. 14, the ninth line of the list shows the average color (R3, G3, B3) corresponding to the coordinates (x3, y3, z3) of point P3 based on the average color information, and the coordinates (x3, y3, z3) of point P3. The tenth line of the list shows the color (R3, G3, B3) of point P3 when observed from a viewpoint VPL to the left of the object OB. 31 ,G 31 ,B 31 ) and the coordinates (x3, y3, z3) of point P3. The 11th line of the list shows the average color (R3, G3, B3) corresponding to the coordinates (x3, y3, z3) of point P3 based on the average color information, and the coordinates (x3, y3, z3) of point P3. The 12th line of the list shows the color (R3, G3, B3) of point P3 when observed from viewpoint VPB below object OB. 32 ,G 32 ,B 32) and the coordinates (x3, y3, z3) of point P3 are shown. Thus, the color of point P3 varies depending on the direction of the viewpoint. If there is a viewpoint from which point P3 is in a blind spot, the average color of point P3 is complemented based on the average color information as the color when point P3 is observed from the viewpoint in the blind spot. In addition, since the coordinates of point P3 are coordinates included in the point cloud information regarding the composite point cloud, the same coordinates are used for any viewpoint.

[0085] In this way, the list lists the colors of the points included in the composite point group when observed from each viewpoint. Note that in this disclosure, four viewpoints, top, bottom, left, and right, are used for explanation, but this is merely an example, and the number of viewpoints is not limited to four.

[0086] The display control unit 215 controls the display on the display 24. For example, the display control unit 215 can transmit the three-dimensional data of the object OB output from the output unit 214 to the display 24 and display the three-dimensional data of the object OB on the display 24 in a list format or the like.

[0087] 1.1.4. Drawing device configuration Fig. 16 is a block diagram showing an example of the configuration of the drawing device 30 in Fig. 1. As shown in Fig. 16, the drawing device 30 includes a processing device 31, a storage device 32, a communication device 33, a display 34, and an input device 35. The elements included in the drawing device 30 are connected to each other by one or more buses for communicating information.

[0088] The processing device 31 is a processor that controls the entire rendering device 30, and is configured, for example, using one or more chips. The processing device 31 is configured, for example, using a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, a register, etc. Note that some or all of the functions of the processing device 31 may be realized by hardware such as a DSP, an ASIC, a PLD, an FPGA, etc. The processing device 31 executes various processes in parallel or sequentially.

[0089] The storage device 32 is a recording medium that can be read and written by the processing device 31. The storage device 32 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM.

[0090] The storage device 32 stores a plurality of programs including a control program PR3 to be executed by the processing device 31. The storage device 32 also functions as a work area for the processing device 31.

[0091] The communication device 33 is hardware as a transmitting / receiving device for communicating with other devices. The communication device 33 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 33 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 33 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, USB, etc. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth, etc.

[0092] The display 34 is a device that displays images and text information. The display 34 displays various images under the control of the processing device 31. For example, various display panels such as a liquid crystal panel and an organic EL panel are suitably used as the display 34.

[0093] The input device 35 receives operations from the user U3. For example, the input device 35 includes a keyboard, a touch pad, a touch panel, and a pointing device such as a mouse. Here, when the input device 35 includes a touch panel, it may also function as the display 34.

[0094] The processing device 31 functions as a first acquisition unit 311, a second acquisition unit 312, a third acquisition unit 313, a drawing unit 314, and a display control unit 315 by reading and executing a control program PR3 from the storage device 32.

[0095] The first acquisition unit 311 acquires three-dimensional data including point cloud information and pairs of color information and each viewpoint position. That is, the first acquisition unit 311 acquires the three-dimensional data shown in FIG.

[0096] The second acquisition unit 312 acquires viewpoint information and coordinate information. The viewpoint information is information indicating the position of the viewpoint of the user U1 in the virtual space VS. The coordinate information is information indicating first coordinates. The first coordinates are coordinates at which a virtual object VO corresponding to the object OB is located in the virtual space VS.

[0097] The third acquisition unit 313 acquires operation information from the input device 35. Moreover, the third acquisition unit 313 acquires image information from the server 40 by using the communication device 33. Moreover, the third acquisition unit 313 stores the acquired image information in the storage device 32. The third acquisition unit 313 acquires the image information stored in the storage device 32.

[0098] The drawing unit 314 draws a virtual object at a first coordinate in the virtual space based on the three-dimensional data, the viewpoint information, and the coordinate information.

[0099] This allows the user U2 to render in the virtual space a three-dimensional model expressing the optical characteristics of the object OB. The optical characteristics are, for example, the glossiness and transparency of the object OB. The glossiness is a characteristic determined by the degree to which the object OB reflects light, and the transparency is a characteristic determined by the degree to which the object OB transmits light.

[0100] The display control unit 315 controls the display on the display 34. For example, the display control unit 315 can transmit the drawing result by the drawing unit 314 to the display 34 and display the drawing result on the display 34. More specifically, the display control unit 315 generates data of an image to be displayed on the display 34 based on an output signal from the drawing unit 314. The generated data is, for example, a composite image of an image of a virtual space and an image of a virtual object. The display control unit 315 transmits the generated data to the display 34.

[0101] 1.1.5. Server configuration Fig. 17 is a block diagram showing an example of the configuration of the server 40 in Fig. 1. As shown in Fig. 17, the server 40 includes a processing device 41, a storage device 42, a communication device 43, a display 44, and an input device 45. The elements included in the server 40 are connected to each other by one or more buses for communicating information.

[0102] The processing device 41 is a processor that controls the entire server 40, and is configured, for example, using one or more chips. The processing device 41 is configured, for example, using a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, a register, etc. Note that some or all of the functions of the processing device 41 may be realized by hardware such as a DSP, an ASIC, a PLD, an FPGA, etc. The processing device 41 executes various processes in parallel or sequentially.

[0103] The storage device 42 is a recording medium that can be read and written by the processing device 41. The storage device 42 includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM.

[0104] The storage device 42 stores a plurality of programs including a control program PR4 to be executed by the processing device 41. The storage device 42 also functions as a work area for the processing device 41. The storage device 42 may also store image data captured by the imaging device 14 and a learning model M1.

[0105] The communication device 43 is hardware as a transmitting / receiving device for communicating with other devices. The communication device 43 is also called, for example, a network device, a network controller, a network card, a communication module, etc. The communication device 43 may include a connector for wired connection and an interface circuit corresponding to the connector. The communication device 43 may also include a wireless communication interface. Examples of the connector and interface circuit for wired connection include products that comply with wired LAN, IEEE1394, and USB. Examples of the wireless communication interface include products that comply with wireless LAN, Bluetooth, etc.

[0106] The display 44 is a device that displays images and text information. The display 44 displays various images under the control of the processing device 41. For example, various display panels such as a liquid crystal panel and an organic EL panel are suitably used as the display 44.

[0107] The input device 45 is a device that accepts operations by an administrator of the information processing system 1. For example, the input device 45 includes a keyboard, a touch pad, a touch panel, and a pointing device such as a mouse. Here, when the input device 45 includes a touch panel, it may also function as the display 44. In particular, the administrator of the information processing system 1 can modify the control program PR4 by using the input device 45.

[0108] The processing device 41 functions as an acquisition unit 411 and an output unit 412, for example, by reading out a control program PR4 from the storage device 42 and executing it.

[0109] The acquisition unit 411 acquires various pieces of information from the three-dimensional data generation device 20 by using the communication device 43. In addition, the acquisition unit 411 acquires a trained learning model M1 from the machine learning device 10 by using the communication device 43.

[0110] The output unit 412 uses the communication device 43 to output the trained learning model M1 to the drawing device 30, thereby providing the learning model M1 to the user U1.

[0111] 1.2. Operation of the First Embodiment 1.2.1.3D data generation flow Fig. 18 is a flowchart showing the operation of the processing device 21 in Fig. 5. Hereinafter, the operation of the processing device 21 will be described with reference to Fig. 18. The routine in Fig. 18 is started, for example, when the processing device 21 is started, and is executed every time a certain period of time has elapsed.

[0112] In step S11, the processing device 21 functions as the point cloud information generating unit 212 to extract a composite point cloud PGC using the learning model M1, and generates point cloud information based on the extracted composite point cloud PGC.

[0113] In step S12, the processing device 21 functions as a color information generation unit 213 to extract a point cloud PGV for each viewpoint using the learning model M1, and generates color information based on the extracted point cloud PGV for each viewpoint.

[0114] In step S13, the processing device 21 functions as the output unit 214 to output the point cloud information and color information as three-dimensional data, and then ends this routine.

[0115] FIG. 19 is a flowchart showing the operation of a subroutine corresponding to step S11 in FIG.

[0116] In step S21, the processing device 21 functions as the point cloud information generating unit 212 to extract the coordinates (x, y, z) of each vertex of a plurality of microcubes obtained by dividing the real space RS into the microcubes.

[0117] In step S22, the processing device 21 functions as a point cloud information generation unit 212 to calculate the color (R, G, B) and density σ at the coordinates (x, y, z) of each vertex of multiple microcubes when viewed from various viewpoints.

[0118] In step S23, the processing device 21 functions as the point cloud information generating unit 212 to combine the calculation results of the colors (R, G, B) and density σ that overlap at the same vertex by averaging or the like.

[0119] In step S24, the processing device 21 functions as the point cloud information generating unit 212 to determine whether the combined density σ is equal to or smaller than the threshold σ th A set of points exceeding this limit is extracted as the synthetic point cloud PGC.

[0120] In step S25, the processing device 21 functions as the point cloud information generating unit 212 to generate the coordinates and averaged color of each point of the extracted composite point cloud PGC as point cloud information.

[0121] FIG. 20 is a flowchart showing the operation of a subroutine corresponding to step S12 in FIG.

[0122] In step S31, the processing device 21 functions as the color information generating unit 213 to extract a point group PGV for each viewpoint by using the learning model M1.

[0123] In step S32, the processing device 21 functions as the color information generating unit 213 to select one point belonging to the composite point group PGC as the target point Ps1.

[0124] In step S33, the processing device 21 functions as the color information generating unit 213 to define a space S1 having a radius L0 and centered on the target point Ps1.

[0125] In step S34, the processing device 21 functions as the color information generating unit 213 to search for points that exist within the space S2 among the points of the point group VCG for each viewpoint. The space S2 is a space related to the point group VCG for each viewpoint, and has the same center and radius as the space S1.

[0126] In step S35, the processing device 21 functions as the color information generating unit 213 to determine whether or not each point of the point group VCG for each viewpoint exists within the space S2.

[0127] In step S35, when the processing device 21 determines that a point of the point cloud for each viewpoint exists within the space S2, that is, when the determination result in step S35 is positive, the processing device 21 executes the process of step S36. In this case, in step S36, the processing device 21 functions as the color information generating unit 213 to set the color of the target point Ps1 to the average value of the colors of each point of the point cloud VCG for each viewpoint that exists within the space S2.

[0128] On the other hand, in step S35, if the processing device 21 determines that the point of the point cloud VCG for each viewpoint does not exist in the space S2, that is, if the determination result in step S35 is negative, the processing device 21 executes the process of step S37. In this case, in step S37, the processing device 21 functions as the color information generating unit 213 to set the color of the target point Ps1 to the color corresponding to the target point Ps1 in the point cloud information.

[0129] Next, in step S38, the processing device 21 functions as the color information generating section 213 to determine whether or not the search has been completed for all the target points.

[0130] When it is determined that the search has been completed for all the target points, that is, when the determination result in step S38 is positive, the processing device 21 ends this routine for the time being. That is, the generation of color information by the color information generating unit 213 is thereby completed.

[0131] On the other hand, if it is determined that the search has not been completed for all target points, that is, if the determination result in step S38 is negative, the processing device 21 returns to the process of step S32, selects another target point, and executes the processes of steps S32 to S37.

[0132] 1.2.2.Drawing flow Fig. 21 is a flowchart showing the operation of the processing device 31 in Fig. 11. Hereinafter, the operation of the processing device 31 will be described with reference to Fig. 21. The routine in Fig. 21 is started, for example, when the processing device 31 is started, and is executed every time a certain period of time has elapsed.

[0133] In step S41, the processing device 31 functions as the first acquisition unit 311 to acquire three-dimensional data.

[0134] Next, in step S42, the processing device 31 acquires viewpoint information and coordinate information by functioning as the second acquisition unit 312. The coordinate information is information indicating the first coordinates.

[0135] Next, in step S43, the processing device 31 functions as the drawing unit 314 to draw the virtual object VO at the first coordinates, and then ends this routine.

[0136] 1.3. Advantages of the First Embodiment According to the above description, the three-dimensional data generating device 20 according to the first embodiment includes a point cloud information generating unit 212, a color information generating unit 213, and an output unit 214. The point cloud information generating unit 212 generates point cloud information using a learning model M1 that has learned a luminance field. The point cloud information is information including three-dimensional coordinates of each point belonging to the composite point cloud PGC. The composite point cloud PGC is a collection of multiple points located on the surface of the object OB. The color information generating unit 213 generates color information using the learning model M1. The color information is information indicating each color corresponding to multiple viewpoints at each point when each point belonging to the composite point cloud PGC is observed from multiple viewpoints. The output unit 214 outputs the point cloud information and the color information as three-dimensional data indicating the three-dimensional shape and color of the object OB.

[0137] According to this, the learning model M1, which has already learned the luminance field, learns the relationship between coordinates, colors, and density when the space including the object OB is viewed from an arbitrary viewpoint. Furthermore, the learning model M1 learns the relationship between coordinates, line of sight direction, and colors when the space including the object OB is viewed from an arbitrary viewpoint. Since the color information generating unit 213 generates the color of the surface of the object OB for each viewpoint as color information, it is possible to incorporate as information the situation in which the color of the surface of the object OB looks different depending on the viewing angle. It is considered that there is a correlation between the fact that the color of the surface of the object OB looks different depending on the viewing angle and the optical properties of the object OB.

[0138] Therefore, the three-dimensional shape of the object OB and a color reflecting the optical characteristics of the object OB are displayed in the three-dimensional data output from the output unit 214. In this manner, according to this aspect of the present embodiment, data expressing the optical characteristics of the object OB can be added to the three-dimensional data generated from the object OB based on data including the optical characteristics acquired using the NeRF technology.

[0139] Furthermore, the color information generating unit 213 generates color information based on colors observed at points that are within a radius L0 of the points in the composite point group PGC among the points in the point group PGV for each viewpoint. The point group PGV for each viewpoint is a collection of multiple points located on the surface of the object OB when the object OB is observed from each of multiple viewpoints.

[0140] The coordinate positions of each point belonging to the composite point group PGC and the coordinate positions of each point belonging to the point group PGV for each viewpoint do not necessarily match due to an estimation error of the density σ using the learning model M1. However, it is considered that the color of each point "belonging to the point group PGV for each viewpoint" existing within a radius L0 from each point belonging to the composite point group PGC has a strong correlation with the color of each point belonging to the composite point group PGC. Furthermore, the color of each point "belonging to the point group PGV for each viewpoint" changes depending on the line of sight angle (θ, φ). Therefore, according to this, it is possible to give a color according to the line of sight angle (θ, φ) to each point belonging to the composite point group PGC. As a result, data expressing the optical characteristics of the object OB can be added to the three-dimensional data generated from the object OB.

[0141] Furthermore, the color information generating unit 213 generates, as color information for each point belonging to the composite point group PGC, a color obtained by averaging the colors of each point within a radius L0 out of each point belonging to the point group PGV for each viewpoint.

[0142] This makes it possible to easily assign a color according to the line of sight angle (θ, φ) to each point belonging to the composite point group PGC.

[0143] Furthermore, when there is no point that is within a radius L0 of each point in the point group PGV for each viewpoint and is distant from each point in the composite point group PGC, the color information generating unit 213 generates average color information as color information. The average color information is information that indicates the color obtained by averaging the colors at each point when observed from a certain number of viewpoints or more. The point group information includes the average color information at each point in the composite point group PGC.

[0144] According to this, even if there is no point belonging to the point group PGV for each viewpoint whose distance to each point belonging to the composite point group PGC is within a radius L0, the color of each point belonging to the composite point group PGC can be easily complemented.

[0145] The drawing device 30 also includes a first acquisition unit 311, a second acquisition unit 312, and a drawing unit 314. The first acquisition unit 311 acquires three-dimensional data including point cloud information and a set of color information and each viewpoint position. The point cloud information is information including three-dimensional coordinates of each point belonging to the composite point cloud PGC. The composite point cloud PGC is a collection of multiple points located on the surface of the object OB. The color information is information indicating each color corresponding to multiple viewpoints at each point when each point belonging to the first point cloud is observed from multiple viewpoints.

[0146] The second acquisition unit 312 acquires viewpoint information and coordinate information. The viewpoint information is information indicating the position of the viewpoint of the user in the virtual space VS. The coordinate information is information indicating first coordinates. The first coordinates are coordinates at which a virtual object corresponding to the object OB is located in the virtual space. The drawing unit 314 draws the virtual object at the first coordinates in the virtual space based on the three-dimensional data, the viewpoint information, and the coordinate information.

[0147] This allows the user U2 to render in the virtual space a three-dimensional model that reflects the optical characteristics of the object OB.

[0148] 2. Second embodiment Hereinafter, the configuration of a three-dimensional data generating device according to the second embodiment of the present invention will be described with reference to Figs. 22 to 24. In the following description, for the sake of simplicity, the same reference numerals are used for the same components as in the first embodiment, and the description of their functions may be omitted. In addition, in the following description, for the sake of simplicity, differences between the second embodiment and the first embodiment will be mainly described.

[0149] 2.1. Configuration of the second embodiment 22 is a diagram showing an overall configuration of an information processing system 1A according to the second embodiment. The information processing system 1A is different from the information processing system 1 according to the first embodiment in that the information processing system 1A includes a three-dimensional data generation device 20A instead of the three-dimensional data generation device 20.

[0150] 23 is a block diagram showing an example of the configuration of a three-dimensional data generation device 20A according to the second embodiment. The three-dimensional data generation device 20A differs from the three-dimensional data generation device 20 according to the first embodiment in that it includes a processing device 21A instead of the processing device 21 and a storage device 22A instead of the storage device 22.

[0151] The processing device 21A is a processor that controls the entire three-dimensional data generation device 20A, and is configured, for example, using one or more chips. The processing device 21A is configured, for example, using a central processing unit (CPU) including an interface with peripheral devices, an arithmetic unit, and a register. Note that some or all of the functions of the processing device 21A may be realized by hardware such as a DSP, an ASIC, a PLD, or an FPGA. The processing device 21A executes various processes in parallel or sequentially.

[0152] The storage device 22A is a recording medium that can be read and written by the processing device 21A. The storage device 22A includes, for example, a nonvolatile memory and a volatile memory. The nonvolatile memory is, for example, a ROM, an EPROM, and an EEPROM. The volatile memory is, for example, a RAM.

[0153] The storage device 22A stores a plurality of programs including a control program PR2A to be executed by the processing device 21A. The storage device 22A also functions as a work area for the processing device 21A.

[0154] The communication device 23, the display 24, and the input device 25 are similar to the communication device 23, the display 24, and the input device 25 of the first embodiment, and therefore a description thereof will be omitted.

[0155] The processing device 21A reads out and executes a control program PR2A from the storage device 22A, thereby functioning as an acquisition unit 211, a point cloud information generation unit 212, a color information generation unit 213, a display control unit 215, a mesh formation unit 216, a mesh color determination unit 217, an output unit 214A, and a display control unit 215. Note that the point cloud information generation unit 212 and the color information generation unit 213 are functionally identical to the point cloud information generation unit 212 and the color information generation unit 213 as functions of the processing device 21 according to the first embodiment, and therefore a description thereof will be omitted.

[0156] The acquisition unit 211 acquires the learning model M1 from the machine learning device 10. The learning model M1 is a model that has already learned the luminance field of the real space RS that includes the object OB.

[0157] The mesh forming unit 216 forms a plurality of meshes corresponding to the composite point cloud based on the point cloud information. Hereinafter, a method of forming meshes will be described with reference to FIG.

[0158] FIG. 24 is a diagram for explaining a method of forming a mesh by the mesh forming unit 216. As shown in FIG. 24, the mesh forming unit 216 extracts three adjacent points V1, V2, and V3 from among the points included in the composite point group PGC. A triangle MS1 having the three extracted points V1, V2, and V3 as vertices is formed as one mesh. In this way, the mesh forming unit 216 extracts three adjacent points from the entire composite point group, and forms a mesh for each of them. Each mesh may also be called a polygon.

[0159] The mesh color determination unit 217 determines the color of each of the multiple meshes based on the color information. More specifically, when a certain viewpoint is input by the user U1, the mesh color determination unit 217 obtains a color corresponding to the viewpoint for each of the three points that are the vertices of each mesh from the color information. The mesh color determination unit 217 determines the average value of the colors of the three points obtained for each mesh as the color of each mesh.

[0160] The output unit 214A outputs information relating to the plurality of meshes and the colors of each of the plurality of meshes as three-dimensional data indicating the three-dimensional shape and color of the object OB.

[0161] The display control unit 215A controls the display on the display 24. For example, the display control unit 215A can transmit mesh-like three-dimensional data of the object OB output from the output unit 214A to the display 24 and display the three-dimensional data of the object OB on the display 24 in a list format or the like.

[0162] 2.2. Advantages of the Second Embodiment As described above, the processing device 21A of the 3D data generation device 20A according to the second embodiment further includes a mesh forming unit 216 and a mesh color determining unit 217 in addition to the components of the processing device 21 of the 3D data generation device 20 according to the first embodiment. The mesh forming unit 216 forms a plurality of meshes corresponding to the composite point group PGC based on the point group information. The mesh color determining unit 217 determines the color of each of the plurality of meshes based on the color information.

[0163] This makes it easier to create a three-dimensional model that reflects the optical characteristics of the object OB.

[0164] 3. Variations The present disclosure is not limited to the above-described embodiments. Specific modified embodiments are exemplified below. Two or more embodiments selected from the following examples may be combined.

[0165] 3.1. Variation 1 In the first embodiment, when none of the points belonging to the point group PGV for each viewpoint exists in the space S2 centered on the target point Ps1, the color information generating unit 213 sets the color of the target point Ps1 to the color corresponding to the target point Ps1 in the point group information. However, this is not the only case, and the color information generating unit 213 may set the color of the target point Ps1 to the color corresponding to the target point Ps1 in the point group information when the number of points existing in the space S2 among the points belonging to the point group PGV for each viewpoint is equal to or less than a set threshold. For example, the set threshold is preferably 1 or 2.

[0166] 3.2. Variation 2 In the first embodiment, the color information generating unit 213 generates the color obtained by averaging the colors of the points Pr1, Pr2, and Pr3 existing in the space S2 among the points belonging to the point group PGV for each viewpoint as the color information of the target point Ps1. However, the color information generating unit 213 may average the colors after weighting the three points Pr1, Pr2, and Pr3 existing in the space S2 according to the distance between the target point Ps1 and each of them.

[0167] A more specific description will be given below with reference to FIG. 25. FIG. 25 is a schematic diagram showing one point of the composite point group PGC of the object OB and an example of the point group PGV for each viewpoint. As shown in FIG. 25, the distances between the target point Ps1 and each of three points Pr1, Pr2, and Pr3 existing in the space S2 are different from each other. If the distance between the point Pr1 and the target point Ps1 is d1, the distance between the point Pr2 and the target point Ps1 is d2, and the distance between the point Pr3 and the target point Ps1 is d3, in the example of FIG. 25, the distance d2 is the shortest and the distance d1 is the longest. For these three points, the color information generating unit 213 calculates color information Cs1=[R s1 ,G s1 ,B s1 ] is calculated as follows:

[0168] If the weights of three points Pr1, Pr2, and Pr3 in the space S2 according to the distance from the target point Ps1 are W1, W2, and W3, respectively, the weights W1, W2, and W3 are given by the following equation (4).

[0169] W1=L0-d1,W2=L0-d2,W3=L0-d3...(4)

[0170] That is, the weight W n1 , W n2 , and W n3 The farther away from the perimeter of the space S, the larger the point becomes. If the color information of the three points Pr1, Pr2, and Pr3 is C1 = [R1, G1, B1], C2 = [R2, G2, B2], and C3 = [R3, G3, B3], respectively, the color information C at the target point is s1 is calculated using the following formula (5).

[0171] C s1 =(W1×C1+W2×C2+W3×C3) / (W1 + W2 + W3) (5)

[0172] If the number of points in the space S2 is n, the distance between any point Prn among the n points in the space S2 and the target point Ps1 is d n Then, the weight W for any point Prn n is expressed by the following equation (6).

[0173] W n =L0-d n (0≦d n ≦L0) (6)

[0174] That is, the weight W n The further away from the periphery of the space S the larger the point becomes. The color information at any point Prn is n =[R n ,G n ,B n ], then the color information C s1 =[R s1 ,G s1 ,B s1 ] is calculated by the following formula (7). n is prevented from diverging to infinity.

[0175] C s1=(W1×C1+W2×C2+···+W n ×C n ) / (W1+W2+···+W n ) ···(7)

[0176] In this manner, the color information generating unit 213 according to this modification weights the colors of each point within the radius L0 among the points belonging to the point group PGV for each viewpoint according to distance, and generates the average color of the weighted points as color information for each point belonging to the composite point group PGC. That is, in this modification, the points in the space S2 that are closer to the target point Ps1 are set to have a higher color mixing ratio. This makes it possible to reproduce the optical characteristics of the object OB more accurately.

[0177] 3.3. Variation 3 In the first embodiment, when none of the points belonging to the point group PGV for each viewpoint exists within the space S2 centered on the target point Ps1, i.e., within a radius L0, the color information generating unit 213 sets the color of the target point Ps1 to the color corresponding to the target point Ps1 in the point group information. However, when none of the points belonging to the point group PGV for each viewpoint exists within the space S2 centered on the target point Ps1, i.e., within a radius L0, the color information generating unit 213 may change the radius of the space S2 to L1, which is longer than L0, and search for points included in the space S2 again.

[0178] 3.4. Variation 4 NeRF requires pose information, i.e., camera parameters, in advance, but there is a technology called NeRF-- that does not require camera parameters, and the learning model M1 may be trained by NeRF-- instead of NeRF. NeRF-- is a method for optimizing the internal and external parameters of the camera simultaneously with NeRF training.

[0179] In addition, there is a technique called Bundle-Adjusting Neural Radiance Fields (BARF) that assumes that the internal parameters of a camera are known and estimates and corrects the external parameters, and the learning model M1 may be trained using BARF instead of NeRF.

[0180] 3.5. Variation 5 In addition, in this embodiment, the machine learning device 10 and the three-dimensional data generation device 20 are configured as separate entities, but the machine learning device 10 and the three-dimensional data generation device 20 may be combined as appropriate, or may be configured as an integrated entity.

[0181] 3.6. Variation 6 In addition, in the second embodiment, the color of the mesh is calculated as the average color of each of the three vertices V1, V2, and V3 of the triangle MS1 that constitutes the mesh, but this is not limited to this, and the color of the mesh may be calculated as the median color of each of the three vertices V1, V2, and V3.

[0182] 3.7. Variation 7 In the second embodiment, the processing device 21A functions as the mesh forming unit 216 to form a triangular mesh, but the shape of each mesh is not limited to a triangle and may be a quadrangle. Furthermore, each mesh does not necessarily have to be a flat surface and may be a curved surface such as a free-form surface. A free-form surface is a method of constructing a curved surface using a NURBS (Non-Uniform Rational B-Spline) curve, a spline curve, a Bezier curve, or the like.

[0183] 4.Other (1) In the above-described embodiment, the storage device 12, the storage device 22, the storage device 22A, the storage device 32, and the storage device 42 are exemplified by ROM and RAM, but may be a flexible disk, a magneto-optical disk (e.g., a compact disk, a digital versatile disk, a Blu-ray (registered trademark) disk), a smart card, a flash memory device (e.g., a card, a stick, a key drive), a CD-ROM (Compact Disc-ROM), a register, a removable disk, a hard disk, a floppy (registered trademark) disk, a magnetic strip, a database, a server, or any other suitable storage medium. The program may also be transmitted from a network via a telecommunication line. The program may also be transmitted from a communication network NET via a telecommunication line.

[0184] (2) In the above-described embodiments, the described information, signals, etc. may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any combination thereof.

[0185] (3) In the above-described embodiment, the input / output information, etc. may be stored in a specific location (e.g., memory) or may be managed using a management table. The input / output information, etc. may be overwritten, updated, or added. The output information, etc. may be deleted. The input information, etc. may be transmitted to another device.

[0186] (4) In the above-described embodiments, the determination may be made based on a value represented using one bit (0 or 1), a Boolean value (true or false), or a numerical comparison (e.g., comparison with a predetermined value).

[0187] (5) The order of the process steps, sequences, flow charts, etc. illustrated in the above-described embodiments may be changed without causing any inconsistency. For example, the methods described in this disclosure present elements of various steps using an example order, and are not limited to the particular order presented.

[0188] (6) Each function illustrated in Fig. 1 to Fig. 25 is realized by any combination of at least one of hardware and software. Furthermore, the method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device that is physically or logically combined, or may be realized by using two or more devices that are physically or logically separated and directly or indirectly connected (for example, by wire, wirelessly, etc.). The functional block may be realized by combining the one device or the multiple devices with software.

[0189] (7) The programs exemplified in the above-described embodiments should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, functions, etc., regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names.

[0190] Additionally, software, instructions, information, etc. may be transmitted or received over a transmission medium. For example, if the software is transmitted from a website, server, or other remote source using wired and / or wireless technologies (such as coaxial cable, fiber optic cable, twisted pair, Digital Subscriber Line (DSL)), and / or wireless technologies (such as infrared, microwave, etc.), then these wired and / or wireless technologies are included within the definition of transmission media.

[0191] (8) In each of the above embodiments, the terms “system” and “network” are used interchangeably.

[0192] (9) The information, parameters, etc. described in this disclosure may be expressed using absolute values, relative values ​​from a predetermined value, or corresponding other information.

[0193] (10) In the above-described embodiment, the machine learning device 10, the three-dimensional data generating device 20, the three-dimensional data generating device 20A, the drawing device 30, and the server 40 may be a mobile station (MS). A mobile station may be referred to by a person skilled in the art as a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a remote device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a handset, a user agent, a mobile client, a client, or some other appropriate term. In the present disclosure, the terms "mobile station", "user terminal", "user equipment (UE)", "terminal", etc. may be used interchangeably.

[0194] (11) In the above-mentioned embodiments, the terms "connected" and "coupled" or any variation thereof refer to any direct or indirect connection or coupling between two or more elements, and may include the presence of one or more intermediate elements between two elements that are "connected" or "coupled" to each other. The coupling or connection between elements may be a physical coupling or connection, a logical coupling or connection, or a combination thereof. For example, "connected" may be read with "access". As used in this disclosure, two elements may be considered to be "connected" or "coupled" to each other using at least one of one or more wires, cables, and printed electrical connections, as well as electromagnetic energy having wavelengths in the radio frequency range, microwave range, and light (both visible and invisible) range, as some non-limiting and non-exhaustive examples.

[0195] (12) In the above embodiments, the phrase "based on" does not mean "based only on," unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

[0196] (13) The terms "determining" and "determining" as used in this disclosure may encompass a wide variety of actions. "Determining" and "determining" may include, for example, judging, calculating, computing, processing, deriving, investigating, looking up, search, inquiry (e.g., searching in a table, database, or other data structure), ascertaining, and the like. Also, "determining" and "determining" may include receiving (e.g., receiving information), transmitting (e.g., sending information), input, output, accessing (e.g., accessing data in a memory), and the like. In addition, "judgment" and "decision" can include considering resolving, selecting, choosing, establishing, comparing, etc., to be a "judgment" or "decision." In other words, "judgment" and "decision" can include considering some action to be a "judgment" or "decision." In addition, "judgment" can be interpreted as "assuming," "expecting," "considering," etc.

[0197] (14) In the above embodiments, when "include," "including," and variations thereof are used, these terms are intended to be inclusive, similar to the term "comprising." Furthermore, the term "or" used in this disclosure is not intended to be an exclusive or.

[0198] (15) In this disclosure, where articles have been added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are in the plural.

[0199] (16) In this disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean "A and B are each different from C." Terms such as "separate" and "combined" may also be interpreted in the same way as "different."

[0200] (17) Each aspect / embodiment described in this disclosure may be used alone, in combination, or switched according to execution. In addition, notification of specific information (e.g., notification that "X is the case") is not limited to explicit notification, but may be implicit (e.g., not notifying the specific information).

[0201] Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented as modified and altered forms without departing from the spirit and scope of the present disclosure as defined by the claims. Therefore, the description of the present disclosure is for illustrative purposes only and does not have any limiting meaning on the present disclosure. [Explanation of symbols]

[0202] Reference Signs List 1, 1A... Information processing system, 10... Machine learning device, 20... 3D data generation device, 30... Rendering device, 212... Point cloud information generation unit, 213... Color information generation unit, 214, 214A... Output unit, 216... Mesh formation unit, 217... Mesh color determination unit, 311... First acquisition unit, 312... Second acquisition unit, 314... Rendering unit, L0... Radius, M1... Learning model, OB... Object, PGC... Composite point cloud, PGV... Point cloud for each viewpoint, σ... Density

Claims

1. a point cloud information generator that generates point cloud information including three-dimensional coordinates of each point belonging to a first point cloud that is a set of a plurality of points located on a surface of an object, using a learning model that has learned a luminance field that is a vector field related to color and density in a three-dimensional space including the object; a color information generating unit that generates color information indicating each color corresponding to a plurality of viewpoints at each of the points belonging to the first point group when the points are observed from a plurality of viewpoints using the learning model; an output unit that outputs the point cloud information and the color information as three-dimensional data indicating a three-dimensional shape and color of the object; A three-dimensional data generating device comprising:

2. the color information generating unit generates the color information based on colors observed at points belonging to a second point group, which is a set of a plurality of points located on the surface of the object when the object is observed from each of the plurality of viewpoints, the points being within a predetermined distance from each of the points belonging to the first point group.

2. The three-dimensional data generating device according to claim 1.

3. the color information generating unit generates, as the color information for each point belonging to the first point group, a color obtained by averaging colors of each point within the predetermined distance among each point belonging to the second point group; 3. The three-dimensional data generating device according to claim 2.

4. the color information generating unit weights the colors of each point within the predetermined distance among the points belonging to the second point group according to the distance, and generates a color obtained by averaging the colors of the weighted points as the color information for each point belonging to the first point group.

3. The three-dimensional data generating device according to claim 2.

5. The point cloud information is generated by inputting a certain number of viewpoints or more into the learning model, the point cloud information includes average color information at each point belonging to the first point cloud, the average color information is information indicating a color obtained by averaging the colors at the respective points when observed from the certain number or more of viewpoints, the color information generating unit generates the average color information as the color information when there is no point that is within a predetermined distance from each point belonging to the first point group among points belonging to a second point group, which is a set of a plurality of points located on the surface of the object when the object is observed from each of the plurality of viewpoints; 2. The three-dimensional data generating device according to claim 1.

6. a mesh forming unit that forms a plurality of meshes corresponding to the first point cloud based on the point cloud information; a mesh color determination unit that determines a color of each of the plurality of meshes based on the color information; The three-dimensional data generating device according to claim 1 , further comprising:

7. a first acquisition unit that acquires point cloud information including three-dimensional coordinates of each point belonging to a first point cloud, which is a collection of a plurality of points located on a surface of an object, and three-dimensional data including a pair of color information indicating each color corresponding to the plurality of viewpoints at each of the points and the positions of the respective viewpoints when the points are observed from a plurality of viewpoints; a second acquisition unit that acquires viewpoint information indicating a position of a viewpoint of a user in a virtual space and coordinate information indicating a first coordinate at which a virtual object corresponding to the object is located in the virtual space; a drawing unit that draws the virtual object at the first coordinates in the virtual space based on the three-dimensional data, the viewpoint information, and the coordinate information; A drawing device comprising:

Citation Information

Patent Citations

  • Shading information derivation device, shading information derivation method, and program

    JP2014120079A

Cited By

  • Position and orientation detection device and position and orientation detection method

    WO2026197100A1