Three-dimensional model generation method, information processing apparatus, and program

Abstract

This three-dimensional model generation method to be carried out by an information processing device comprises: a first acquisition step (S101) for acquiring a first three-dimensional model, which has first position information indicating a plurality of first three-dimensional positions on an object under measurement, from a measurement instrument that generates the first three-dimensional model by emitting electromagnetic waves and acquiring reflected waves resulting from the reflection of the electromagnetic waves by the object under measurement; a second acquisition step (S101) for acquiring a multi-viewpoint image obtained by photographing the object under measurement from multiple viewpoints using one or more cameras; and a generation step (S104) for generating a second three-dimensional model by using the multi-viewpoint image to increase the definition of the first three-dimensional model.

Description

Technical Field

[0001] The present disclosure relates to a three-dimensional model generation method, an information processing apparatus, and a program.

Background Art

[0002] Patent Document 1 discloses a three-dimensional shape measurement apparatus that acquires a three-dimensional shape using a three-dimensional laser scanner.

Prior Art Document

Patent Document

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] For the generation process of a three-dimensional model, improvements related to the generation process of the three-dimensional model are desired.

[0005] The present disclosure provides a three-dimensional model generation method and the like that realize improvements related to the generation process of a three-dimensional model.

Means for Solving the Problems

[0006] A three-dimensional model generation method according to an aspect of the present disclosure is a three-dimensional model generation method executed by an information processing apparatus. The method includes: a first acquisition step of acquiring a first three-dimensional model generated by a measuring instrument that emits electromagnetic waves and acquires a reflected wave obtained by reflecting the electromagnetic waves from a measurement target, the first three-dimensional model having first position information indicating a plurality of first three-dimensional positions on the measurement target; a second acquisition step of acquiring multi-viewpoint images of the measurement target captured by one or more cameras from a plurality of viewpoints different from each other; and a generation step of generating a second three-dimensional model of the measurement target based on the multi-viewpoint images and the first three-dimensional model. In the generation step, the second three-dimensional model is generated by refining color information included in the first three-dimensional model using the multi-viewpoint images, and the second three-dimensional model is the first three-dimensional model where the pornographic information is assigned in units finer than the area where the pornographic information is assigned .

[0007] An information processing apparatus according to an aspect of the present disclosure includes: a first acquisition unit that acquires a first three-dimensional model generated by a measuring instrument that emits electromagnetic waves and acquires a reflected wave obtained by reflecting the electromagnetic waves from a measurement target, the first three-dimensional model having first position information indicating a plurality of first three-dimensional positions on the measurement target; a second acquisition unit that acquires multi-viewpoint images of the measurement target captured by one or more cameras from a plurality of viewpoints different from each other; and a generation unit that generates a second three-dimensional model of the measurement target based on the multi-viewpoint images and the first three-dimensional model. The generation unit generates the second three-dimensional model by refining color information included in the first three-dimensional model using the multi-viewpoint images, and the second three-dimensional model is the first three-dimensional model where the pornographic information is assigned in units finer than the area where the pornographic information is assigned .

[0008] Note that the present disclosure may be implemented as a program that causes a computer to execute the steps included in the three-dimensional model generation method. Further, the present disclosure may be implemented as a non-transitory recording medium such as a CD-ROM that can be read by a computer on which the program is recorded. Further, the present disclosure may be implemented as information, data, or a signal indicating the program. And those programs, information, data, and signals may be distributed via a communication network such as the Internet.

Effect of the Invention

[0009] According to the present disclosure, it is possible to provide a three-dimensional model generation method and the like that realize improvements related to the generation process of a three-dimensional model.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

MODE FOR CARRYING OUT THE INVENTION

[0011] (Background Leading to the Present Disclosure) In conventional techniques such as Patent Document 1, three-dimensional measurement of measurement objects such as cultural heritages and infrastructure is performed by using a three-dimensional laser rangefinder of a TOF (Time Of Flight) method or a phase difference method. The three-dimensional model generated by the three-dimensional laser rangefinder includes three-dimensional point cloud data in which the three-dimensional shape of the measurement object is represented by a set of three-dimensional points. Further, when a camera is built in the three-dimensional laser rangefinder, the three-dimensional model further includes color information indicating the color of the measurement object at each three-dimensional point based on the image taken by the camera.

[0012] When the object to be measured requires highly reproducible three-dimensional digitization such as cultural heritage, a stationary three-dimensional laser measuring instrument (e.g., 3D scanner) capable of acquiring a high-density three-dimensional point cloud is used. Also, when wide-area three-dimensional digitization such as road infrastructure is required, a mobile three-dimensional laser measuring instrument (e.g., LiDAR) capable of quickly acquiring a three-dimensional point cloud is used.

[0013] Such three-dimensional laser measuring instruments require a lot of time to acquire a high-precision and high-density three-dimensional point cloud. Also, the images taken by the cameras built into the three-dimensional laser measuring instruments do not have sufficient resolution compared to the high-precision and high-density three-dimensional points, so the accuracy of the color information corresponding to each three-dimensional point is not sufficient.

[0014] Also, since the three-dimensional laser measuring instrument irradiates laser light radially, the density of the three-dimensional point cloud obtained as the measurement result of the object to be measured decreases as the distance from the object to be measured increases. Therefore, in order to acquire a high-density three-dimensional point cloud, measurement from a position close to the object to be measured is required. For this reason, in order to acquire a high-density three-dimensional point cloud, the larger the size of the object to be measured, the more measurements are required from many positions around the object to be measured. However, since the three-dimensional laser measuring instrument is heavy and inconvenient to carry, it takes more time to install the three-dimensional laser measuring instrument at many positions of the object to be measured. Also, even when using multiple three-dimensional laser measuring instruments, means for moving multiple three-dimensional laser measuring instruments are required for each of the multiple units. Thus, it is difficult to acquire a high-precision and high-density three-dimensional point cloud using a three-dimensional laser measuring instrument.

[0015] Also, as another prior art for generating a three-dimensional model, it is known to use multi-viewpoint images obtained by imaging a subject with a camera from a plurality of viewpoints. The images captured by the camera used in this prior art are generally higher in resolution than the images captured by the camera incorporated in the three-dimensional laser range finder. That is, the images captured by the camera used in this prior art are high-definition (high-precision) with respect to the color information of the subject. Further, in this prior art, multi-viewpoint images obtained by continuously imaging the subject while moving the camera may be used. Such continuous imaging has a shorter processing time compared to repeating the movement and measurement of the three-dimensional laser range finder. Also, the camera used in this prior art has higher portability than the three-dimensional laser range finder. For example, it is possible to provide a camera on a moving body or a flying object that moves on the ground. Therefore, since the camera used in this prior art has high portability, it is possible to image the subject from all directions. Also, this prior art of generating a three-dimensional model based on multi-viewpoint images can generate a three-dimensional model with high accuracy even for a subject on which laser light is difficult to reflect.

[0016] Therefore, in the present disclosure, a three-dimensional model generation method and the like that realize an improvement in the generation process of a three-dimensional model are provided.

[0017] A three-dimensional model generation method according to an aspect of the present disclosure is a three-dimensional model generation method executed by an information processing apparatus, including: a first acquisition step of acquiring a first three-dimensional model generated by a measuring instrument that emits electromagnetic waves and acquires a reflected wave obtained by reflecting the electromagnetic waves from a measurement target, the first three-dimensional model having first position information indicating a plurality of first three-dimensional positions on the measurement target; a second acquisition step of acquiring multi-viewpoint images captured by one or more cameras from a plurality of viewpoints different from the measurement target; and a generation step of generating a second three-dimensional model of the measurement target based on the multi-viewpoint images and the first three-dimensional model.

[0018] For example, when a camera is built into the measuring instrument and the first three-dimensional model includes color information indicating the color of the measurement target, if one or more cameras with higher resolution than the built-in camera are used, the color information included in the first three-dimensional model can be refined to a high definition using the color information of the multi-viewpoint images to generate a second three-dimensional model. Also, for example, if only the part of the measurement target that requires high accuracy with respect to the position information is three-dimensionally modeled using the measuring instrument and the other parts of the measurement target are three-dimensionally modeled using multi-viewpoint images to generate the second three-dimensional model, the processing time can be shortened compared to three-dimensionally modeling the entire measurement target using the measuring instrument. Also, for example, even if there is a part on the measurement target where electromagnetic waves are difficult to reflect, the second three-dimensional model can be generated by interpolation at the three-dimensional positions calculated from the multi-viewpoint images. Also, for example, a third three-dimensional model is generated based on the multi-viewpoint images, and the second position information indicating a plurality of positions of the measurement target included in the third three-dimensional model is refined to a high definition using the first position information, whereby the second three-dimensional model of the measurement target can also be generated.

[0019] By combining the advantages of the measuring instrument and the advantages of the camera in this way, the three-dimensional model generation method according to one aspect of the present disclosure can achieve improvements in the generation process of the three-dimensional model.

[0020] Note that the refinement of the color information in the present disclosure indicates high-precision using high-resolution images, but does not mean using high-resolution images with an expanded shooting area (angle of view). For example, the refinement of the color information in the present disclosure means that when the three-dimensional model is divided into a plurality of regions and color information is assigned to each region, the color information is assigned in finer units by increasing the number of those regions.

[0021] Also, the refinement of the position information in the present disclosure means increasing the density of the positions that can be represented by the three-dimensional model. Specifically, it means increasing the number of positions on the measurement target indicated by the three-dimensional model. Furthermore, the refinement of the position information in the present disclosure also means increasing the accuracy of the positions on the measurement target indicated by the three-dimensional model.

[0022] In the present disclosure, the refinement of the three-dimensional model indicates at least one of the refinement of color information and the refinement of position information.

[0023] Further, in the generation step, a third three-dimensional model is generated using the multi-viewpoint image, and the three-dimensional coordinate axes of the first three-dimensional model and the three-dimensional coordinate axes of the third three-dimensional model are aligned to identify a first positional relationship between the first three-dimensional model and the multi-viewpoint image. Then, the second three-dimensional model may be generated by refining the first three-dimensional model using the identified first positional relationship and the multi-viewpoint image.

[0024] Therefore, by using the multi-viewpoint image in which the first positional relationship with the first three-dimensional model is identified, the first three-dimensional model can be refined more effectively because the first three-dimensional model can be refined.

[0025] Further, in the generation step, for each of the plurality of first three-dimensional positions, the second three-dimensional model may be generated by adding second color information based on the pixels of the multi-viewpoint image corresponding to the first three-dimensional position as the color information corresponding to the first three-dimensional position using the first positional relationship and the multi-viewpoint image.

[0026] Therefore, high-precision color information can be added to the first three-dimensional model by using the multi-viewpoint image in which the positional relationship with the first three-dimensional model is identified.

[0027] Further, the first three-dimensional model further has first color information indicating the color of the measurement object at each of the plurality of first three-dimensional positions generated using the image of the measurement object being photographed, and each of the plurality of images included in the multi-viewpoint image is an image with a higher resolution than the image photographed by the measuring instrument. In the generation step, for each of the plurality of first three-dimensional positions, the second three-dimensional model is generated by changing the first color information corresponding to the first three-dimensional position to second color information based on the pixels of the multi-viewpoint image corresponding to the first three-dimensional position using the first positional relationship and the multi-viewpoint image.

[0028] Therefore, by using the multi-viewpoint image in which the first positional relationship with the first three-dimensional model is specified, the first color information of the first three-dimensional model can be changed to second color information with higher accuracy than the first color information.

[0029] Further, in the generation step, the second three-dimensional position on the measurement target between two or more positions included in the plurality of first three-dimensional positions may be interpolated by using the first positional relationship and the multi-viewpoint image, thereby generating the second three-dimensional model including the plurality of first three-dimensional positions and the interpolated second three-dimensional position.

[0030] Thereby, the three-dimensional position between two or more first three-dimensional positions of the first three-dimensional model can be interpolated by using the multi-viewpoint image in which the first positional relationship with the first three-dimensional model is specified. Therefore, a second three-dimensional model with a higher density than the first three-dimensional model can be generated.

[0031] Further, in the generation step, the missing part of the first position information may be detected, and the second three-dimensional model including the plurality of first three-dimensional positions and the interpolated third three-dimensional position may be generated by interpolating the third three-dimensional position on the measurement target in the detected missing part by using the first positional relationship and the multi-viewpoint image.

[0032] Thereby, even when a missing part occurs in the first three-dimensional model due to occlusion or the like during measurement by the measuring instrument, the three-dimensional position of the missing part of the first three-dimensional model can be interpolated by using the multi-viewpoint image in which the first positional relationship with the first three-dimensional model is specified.

[0033] Further, in the generation step, the third three-dimensional model is generated by using the multi-viewpoint image, and the second three-dimensional model is generated by refining the third three-dimensional model by using the first three-dimensional model.

[0034] Further, the method further includes a third acquisition step of acquiring a specific image in which a second positional relationship with the first three-dimensional model is specified, and the second positional relationship, wherein, in the generation step, a third three-dimensional model is generated using the multi-viewpoint image and the specific image, and the first positional relationship between the first three-dimensional model and the multi-viewpoint image is specified using the third three-dimensional model and the second positional relationship, and the second three-dimensional model is generated by refining the first three-dimensional model using the specified first positional relationship and the multi-viewpoint image.

[0035] According to this, by generating a third three-dimensional model using, together with the multi-viewpoint image, a specific image in which the second positional relationship with the measuring instrument has already been specified, the first positional relationship between the first three-dimensional model and the multi-viewpoint image can be easily specified.

[0036] Further, an information processing apparatus according to an aspect of the present disclosure includes a first acquisition unit that acquires a first three-dimensional model generated by a measuring instrument that emits electromagnetic waves and acquires a reflected wave obtained by reflecting the electromagnetic waves from a measurement target, the first three-dimensional model having first position information indicating a plurality of first three-dimensional positions on the measurement target, a second acquisition unit that acquires a multi-viewpoint image taken by one or more cameras from a plurality of viewpoints different from the measurement target, and a generation unit that generates a second three-dimensional model of the measurement target based on the multi-viewpoint image and the first three-dimensional model.

[0037] By combining the advantages of the measuring instrument and the advantages of the camera, the information processing apparatus according to an aspect of the present disclosure can achieve an improvement in the generation process of the three-dimensional model.

[0038] Note that the present disclosure may be implemented as a program that causes a computer to execute the steps included in the above three-dimensional model generation method. Further, the present disclosure may be implemented as a non-transitory recording medium such as a CD-ROM that can be read by a computer recording the program. Further, the present disclosure may be implemented as information, data, or signals indicating the program. And those programs, information, data, and signals may be distributed via a communication network such as the Internet.

[0039] Hereinafter, each embodiment of the three-dimensional model generation method and the like according to the present disclosure will be described in detail with reference to the drawings. Note that each embodiment described below shows a specific example of the present disclosure. Therefore, the numerical values, shapes, materials, components, arrangements and connection forms of the components, steps, order of steps, etc. shown in each of the following embodiments are merely examples and are not intended to limit the present disclosure.

[0040] Further, each figure is a schematic diagram and is not necessarily drawn precisely. Also, in each figure, the same reference numerals are given to substantially the same configurations, and redundant descriptions may be omitted or simplified.

[0041] (Embodiment) [Overview] First, with reference to FIG. 1, the overview of the three-dimensional model generation method according to the embodiment will be described.

[0042] FIG. 1 is a diagram for explaining the overview of the three-dimensional model generation method according to the embodiment.

[0043] In the three-dimensional model generation method, as shown in FIG. 1, a three-dimensional model of the measurement target 500 is generated from a plurality of images taken at different viewpoints using the measuring instrument 100 and a plurality of cameras 101. The measurement target 500 may be, for example, a stationary object such as a building or an infrastructure structure. Further, the measurement target 500 may include a moving object in addition to stationary objects. The moving object is, for example, a person or a vehicle moving within the space when the space where the measurement target 500 exists is a space on a road. Further, the moving object is, for example, a sports competitor, a sports equipment held by the competitor, a spectator, etc. when the space where the measurement target 500 exists is a sports competition venue. Note that the measurement target 500 may include not only a specific object but also a landscape or the like. FIG. 1 illustrates the case where the measurement target 500 is a building.

[0044] FIG. 2 is a block diagram showing a characteristic configuration of the three-dimensional model generation apparatus according to the embodiment. FIG. 3 is a diagram for explaining the configuration of the measuring instrument.

[0045] As shown in FIG. 2, the three-dimensional model generation system 400 includes a measuring instrument 100, a plurality of cameras 101, and a three-dimensional model generation apparatus 200.

[0046] (Measuring instrument) The measuring device 100 generates a first three-dimensional model by emitting electromagnetic waves and acquiring reflected waves obtained by reflecting the emitted electromagnetic waves from the measurement target 500. Specifically, the measuring device 100 measures the time it takes for the emitted electromagnetic waves to be reflected by the measurement target 500 and return to the measuring device 100 after being emitted, and calculates the distance between the measuring device 100 and a point on the surface of the measurement target 500 using the measured time and the wavelength of the electromagnetic waves. The measuring device 100 emits electromagnetic waves in a plurality of predetermined radial directions from a reference point of the measuring device 100. For example, the measuring device 100 emits electromagnetic waves at a first angular interval around the horizontal direction and emits electromagnetic waves at a second angular interval around the vertical direction. Therefore, the measuring device 100 can calculate the three-dimensional coordinates of a plurality of points on the measurement target 500 by detecting the distances to the measurement target 500 in a plurality of directions around the measuring device 100. Thus, the measuring device 100 can calculate first position information indicating a plurality of first three-dimensional positions on the measurement target 500 around the measuring device 100, and can generate a first three-dimensional model having the first position information. The first position information may be a first three-dimensional point cloud including a plurality of first three-dimensional points indicating a plurality of first three-dimensional positions.

[0047] In the present embodiment, as shown in FIG. 3, the measuring device 100 is a three-dimensional laser measuring device having a laser irradiation unit 111 that irradiates laser light as electromagnetic waves and a laser light receiving unit 112 that receives reflected light obtained by reflecting the irradiated laser light from the measurement target 500. The measuring device 100 scans the measurement target 500 with laser light by rotating or swinging a unit including the laser irradiation unit 111 and the laser light receiving unit 112 around two different axes, or by installing a movable mirror (MEMS (Micro Electro Mechanical Systems) mirror) that swings around two axes on the optical path of the laser to be irradiated or received. Thereby, the measuring device 100 can generate a highly accurate and high-density first three-dimensional model of the measurement target 500. Here, the generated first three-dimensional model is, for example, a three-dimensional model in the world coordinate system.

[0048] The measuring device 100 is an example of a three-dimensional laser measuring device that measures the distance to the measurement target 500 by irradiating a laser beam. However, the present invention is not limited to this, and it may be a millimeter-wave radar measuring device that measures the distance to the measurement target 500 by emitting millimeter waves.

[0049] In addition, the measuring device 100 may generate a first three-dimensional model having first color information. The first color information is color information generated using an image captured by the measuring device 100, and is color information indicating the color of each of a plurality of first three-dimensional points included in the first three-dimensional point cloud.

[0050] Specifically, the measuring device 100 may incorporate a camera that captures the measurement target 500 around the measuring device 100. The camera incorporated in the measuring device 100 captures an area including the irradiation range of the laser beam irradiated by the measuring device 100. In addition, the imaging range captured by the camera is associated in advance with the irradiation range. Specifically, a plurality of directions in which the laser beam is irradiated by the measuring device 100 and each pixel in the image captured by the camera are associated in advance, and the measuring device 100 sets the pixel value of the image associated with the direction of the first three-dimensional point as the first color information indicating the color of each of the plurality of first three-dimensional points included in the first three-dimensional point cloud.

[0051] As described above, the first three-dimensional model of the measurement target 500 generated by the measuring device 100 is represented by, for example, a set of first three-dimensional points indicating the first three-dimensional positions of a plurality of measurement points on the measurement target 500 (the surface of the measurement target 500). The set of three-dimensional points is called a three-dimensional point cloud. The first three-dimensional position indicated by each three-dimensional point in the three-dimensional point cloud is represented by, for example, three-dimensional coordinates of three-value information consisting of the X component, Y component, and X component of a three-dimensional coordinate space composed of the XYZ axes. Note that the first three-dimensional model may include not only the three-dimensional coordinates but also the first color information indicating the color of each point, or the shape information representing the surface shape of each point and its periphery. The first color information may be represented, for example, in the RGB color space, or may be represented in another color space such as HSV, HLS, or YUV.

[0052] The measuring instrument 100 may be directly connected to the three-dimensional model generation device 200 by wired communication or wireless communication so that the generated first three-dimensional model can be output to the three-dimensional model generation device 200, or may be indirectly connected to the three-dimensional model generation device 200 via a hub (not shown) such as a communication device or a server.

[0053] Further, the measuring instrument 100 may generate a first three-dimensional model of the measurement target 500 around the measuring instrument 100 at each of a plurality of measurement positions. In this case, the measuring instrument 100 may output the plurality of generated first three-dimensional models to the three-dimensional model generation device 200, or may generate one first three-dimensional model by integrating the plurality of first three-dimensional models in the world coordinate system and output the integrated one first three-dimensional model to the three-dimensional model generation device 200.

[0054] Also, although the position of the measurement point 501 on the measurement target 500 is shown by the three-dimensional point cloud in the first three-dimensional model, it is not limited to this, and it may be shown by a depth image having the distance information from the measuring instrument 100 to the measurement point 501 as a pixel value. The pixel value of each pixel of the depth image may include color information indicating the color of the measurement target 500 in addition to the distance information.

[0055] (Plural cameras) The plurality of cameras 101 are a plurality of imaging devices that photograph the measurement target 500. Each of the plurality of cameras 101 photographs the measurement target 500 and outputs the plurality of photographed frames to the three-dimensional model generation device 200. Also, the plurality of cameras 101 photograph the same measurement target 500 from different viewpoints. A frame is, in other words, an image. The image photographed by each camera 101 is an image with a higher resolution than the image photographed by the measuring instrument 100. Note that each camera 101 does not have to be a camera with a higher resolution than the camera built into the measuring instrument 100, and any camera that can photograph with more pixels than the camera of the measuring instrument 100 with respect to the size of the measurement target 500 is sufficient. In the image photographed by each camera 101, the number of pixels per unit area when the measurement target 500 is two-dimensionally projected is larger than that of the image photographed by the camera of the measuring instrument 100. Therefore, the accuracy of the color information at a specific point of the measurement target 500 obtained from the image photographed by each camera 101 is higher than the accuracy of the color information at a specific point of the measurement target 500 obtained from the image photographed by the camera of the measuring instrument 100.

[0056] Note that although the three-dimensional model generation system 400 is described as including a plurality of cameras 101, it is not limited to this, and it may include only one camera 101. For example, in the three-dimensional model generation system 400, the measurement target 500 existing in the real space may be photographed so that a multi-viewpoint image including a plurality of frames with different viewpoints is generated by one camera 101 while moving one camera 101. Each of the plurality of frames is a frame photographed (generated) by a camera 101 in which at least one of the position and orientation of the camera 101 is different from each other.

[0057] Also, each camera 101 may be a camera that generates a two-dimensional image, or a camera equipped with a three-dimensional measurement sensor that generates a three-dimensional model. In the present embodiment, the plurality of cameras 101 are cameras that each generate a two-dimensional image.

[0058] The plurality of cameras 101 may be directly connected to the three-dimensional model generation device 200 by wired communication or wireless communication so that each can output the frame it has captured to the three-dimensional model generation device 200, or may be indirectly connected to the three-dimensional model generation device 200 via a hub (not shown) such as a communication device or a server.

[0059] Note that the frames captured by the plurality of cameras 101 may be output to the three-dimensional model generation device 200 in real time. Also, the frames may be output to the three-dimensional model generation device 200 from an external storage device such as a memory or a cloud server after being recorded once in the external storage device.

[0060] In addition, the plurality of cameras 101 may each be a fixed camera such as a surveillance camera, a mobile camera such as a video camera, a smartphone, or a wearable camera, or a moving camera such as a drone with a photographing function. Each of the plurality of cameras 101 may be anything as long as it does not have a configuration for measuring by emitting electromagnetic waves and receiving reflected waves.

[0061] Moreover, each camera 101 may be a camera that captures an image with a higher resolution than the camera built in the measuring instrument 100. The number of pixels of the image captured by each camera 101 may be more than the number of three-dimensional point clouds that the measuring instrument 100 can measure at one time.

[0062] (Three-dimensional model generation device) The three-dimensional model generation device 200 acquires a first three-dimensional model from the measuring instrument 100. Further, the three-dimensional model generation device 200 acquires a plurality of frames from each of the plurality of cameras 101, thereby acquiring multi-viewpoint images of the measurement target 500 taken from a plurality of different viewpoints. Then, the three-dimensional model generation device 200 generates a second three-dimensional model by refining the first three-dimensional model using the multi-viewpoint images. Note that each of the plurality of viewpoints may be the same as any measurement position (the position of the measuring instrument 100 at the time of measurement) by the measuring instrument 100, or may be different. In other words, the plurality of viewpoints at the time of shooting by the plurality of cameras 101 may be the same as any of the viewpoints at the time of shooting by the built-in camera of the measuring instrument 100, or may be different.

[0063] The three-dimensional model generation device 200 includes at least a computer system including, for example, a control program, a processing circuit such as a processor or a logic circuit that executes the control program, and a recording device such as an internal memory that stores the control program or an accessible external memory. The three-dimensional model generation device 200 is an information processing device. The functions of each processing unit of the three-dimensional model generation device 200 may be realized by software or by hardware.

[0064] Further, the three-dimensional model generation device 200 may store camera parameters in advance. Also, the plurality of cameras 101 may be communicably connected to the three-dimensional model generation device 200 wirelessly or by wire.

[0065] Also, the plurality of frames captured by the camera 101 may be directly output to the three-dimensional model generation device 200. In this case, the camera 101 may be directly connected to the three-dimensional model generation device 200, for example, by wired communication or wireless communication, or may be indirectly connected to the three-dimensional model generation device 200 via a hub (not shown) such as a communication device or a server.

[0066] The details of the configuration of the three-dimensional model generation device 200 will be described with reference to FIG. 2.

[0067] The three-dimensional model generation device 200 includes a receiving unit 201, a storage unit 202, an acquisition unit 203, a generation unit 204, and an output unit 205.

[0068] The receiving unit 201 receives the first three-dimensional model from the measuring instrument 100. The receiving unit 201 receives a plurality of frames (i.e., multi-viewpoint images) from the plurality of cameras 101. The receiving unit 201 outputs the received first three-dimensional model and the plurality of frames to the storage unit 202. The receiving unit 201 may divide the position information of the first three-dimensional model or cut out a part thereof, and output a three-dimensional model including the divided or cut-out part of the position information to the storage unit 202 for storage in the storage unit 202. Note that the receiving unit 201 may receive the first three-dimensional model from the measuring instrument 100 via another information processing device. Similarly, the receiving unit 201 may receive a plurality of frames from the plurality of cameras 101 via another information processing device.

[0069] The receiving unit 201 is, for example, a communication interface for communicating with the measuring instrument 100 and the plurality of cameras 101. When the three-dimensional model generation device 200 wirelessly communicates with the measuring instrument 100 and the plurality of cameras 101, the receiving unit 201 includes, for example, an antenna and a wireless communication circuit. Alternatively, when the three-dimensional model generation device 200 communicates with the measuring instrument 100 and the plurality of cameras 101 by wire, the receiving unit 201 includes, for example, a connector connected to a communication line and a wired communication circuit. The receiving unit 201 is an example of the first acquisition unit and the second acquisition unit. Thus, the first acquisition unit and the second acquisition unit may be realized by one processing unit or may be realized by two independent processing units respectively.

[0070] The storage unit 202 stores the first 3D model and the plurality of frames received by the reception unit 201. The storage unit 202 may store the processing results of the processing units included in the 3D model generation device 200. The storage unit 202 may store, for example, the control programs executed by the respective processing units included in the 3D model generation device 200. The storage unit 202 is realized by, for example, an HDD (Hard Disk Drive), a flash memory, or the like.

[0071] The acquisition unit 203 acquires the first 3D model and the plurality of frames stored in the storage unit 202 from the storage unit 202 and outputs them to the generation unit 204.

[0072] Note that the 3D model generation device 200 may not include the storage unit 202 and the acquisition unit 203. In this case, the reception unit 201 may output the first 3D model received from the measuring instrument 100 and the plurality of frames received from the plurality of cameras 101 to the generation unit 204.

[0073] The generation unit 204 generates a second 3D model with higher precision and higher density than the first 3D model by refining at least one of the position information and the color information of the first 3D model using the multi-viewpoint images. Specific processing of the generation unit 204 will be described later.

[0074] The output unit 205 transmits the second 3D model generated by the generation unit 204 to an external device. The output unit 205 includes, for example, a display device such as a display (not shown) and an antenna, a communication circuit, a connector, etc. for communicably connecting to the display device by wire or wirelessly. The output unit 205 outputs the integrated 3D model to the display device to display the 3D model on the display device.

[0075] [Operation of 3D Model Generation Device] Next, the operation of the 3D model generation device 200 will be described with reference to FIG. 4. FIG. 4 is a flowchart showing an example of the operation of the 3D model generation device.

[0076] First, in the three-dimensional model generation device 200, the receiving unit 201 receives the first three-dimensional model from the measuring instrument 100 and receives a plurality of frames (i.e., multi-viewpoint images) from the plurality of cameras 101 (S101). Step S101 is an example of the first acquisition step and the second acquisition step. Note that the receiving unit 201 does not necessarily receive the first three-dimensional model and the multi-viewpoint images at the same timing, and may receive them at different timings. That is, the first acquisition step and the second acquisition step may be performed at the same timing or at different timings.

[0077] Next, the storage unit 202 stores the first three-dimensional model and the multi-viewpoint images received by the receiving unit 201 (S102).

[0078] Next, the acquisition unit 203 acquires the first three-dimensional model and the multi-viewpoint images stored in the storage unit 202, and outputs the acquired first three-dimensional model and multi-viewpoint images to the generation unit 204 (S103).

[0079] The generation unit 204 generates a second three-dimensional model with higher accuracy and higher density than the first three-dimensional model by refining at least one of the position information and the color information of the first three-dimensional model using the multi-viewpoint images acquired by the acquisition unit 203 (S104). Step S104 is an example of the generation step.

[0080] Then, the output unit 205 outputs the second three-dimensional model generated by the generation unit 204 (S105).

[0081] Next, the process of the generation unit 204 of the three-dimensional model generation device 200 (S104) will be described with reference to FIG. 5. FIG. 5 is a flowchart showing an example of the detailed process of the generation step.

[0082] The generation unit 204 identifies a first positional relationship, which is the positional relationship between the first three-dimensional model and the multi-viewpoint images (S111). That is, the generation unit 204 identifies, as the first positional relationship, the position and orientation of the camera 101 when each image included in the multi-viewpoint images was captured, in the three-dimensional coordinate axes of the first three-dimensional model. The position of the camera 101 when it captured the image is the viewpoint in the captured image, and the orientation of the camera 101 when it captured the image is the direction of the optical axis of the camera 101, that is, the shooting direction. The position and orientation of the camera 101 are the external parameters of the camera 101. Details of the alignment process will be described later.

[0083] Next, the generation unit 204 refines at least one of the first position information and the first color information of the first three-dimensional model using the identified first positional relationship and the multi-viewpoint images (S112). Specifically, the generation unit 204 may refine the color information of the first three-dimensional model by changing the first color information of the first three-dimensional model to second color information with higher accuracy than the first color information. Also, the generation unit 204 may interpolate a second three-dimensional position on the measurement target 500 between two positions included in a plurality of first three-dimensional positions of the first three-dimensional model. Of course, the generation unit 204 may interpolate a second three-dimensional position on the measurement target 500 between three or more positions. Also, the generation unit 204 may detect a missing part of the first position information and interpolate a third three-dimensional position on the measurement target 500 at the detected missing part.

[0084] Next, the alignment process (S111) by the generation unit 204 will be described with reference to FIG. 6. FIG. 6 is a flowchart showing an example of the detailed process of the alignment process.

[0085] The generation unit 204 generates a third three-dimensional model using the multi-viewpoint images (S121). In the present disclosure, a three-dimensional model generated using multi-viewpoint images is referred to as a third three-dimensional model. Note that the generation unit 204 may generate, as the third three-dimensional model, a three-dimensional model including only the three-dimensional point cloud of the contour portion of the measurement target 500 in the generation of the third three-dimensional model, or may generate, as the third three-dimensional model, a three-dimensional model including the three-dimensional point cloud of the contour portion of the measurement target 500 and the three-dimensional point cloud of the objects around the measurement target 500.

[0086] Here, the generation of the third three-dimensional model (i.e., three-dimensional reconstruction) using the multi-viewpoint images obtained by the camera 101 in the present disclosure is defined. An image obtained by photographing a measurement target 500 existing in the real space from a plurality of different viewpoints by one or more cameras is referred to as a multi-viewpoint image. The multi-viewpoint image may be an image group including a plurality of frames obtained by photographing a moving image while moving one or more cameras, or may be an image group including a plurality of still images obtained by imaging from a plurality of positions with one or more cameras, or may be an image group including a plurality of still images obtained by photographing with a plurality of fixed cameras installed at a plurality of positions. Further, it may be an image group obtained by combining two or more of these image groups. That is, the multi-viewpoint image includes a plurality of two-dimensional images obtained by photographing the same measurement target 500 from different viewpoints. Reconstructing the measurement target 500 in the three-dimensional space using this multi-viewpoint image is called three-dimensional reconstruction. Alternatively, generating the measurement target 500 in the three-dimensional space using the multi-viewpoint image is called three-dimensional model generation.

[0087] FIG. 7 is a diagram showing the mechanism of three-dimensional reconstruction.

[0088] The generation unit 204 reconstructs the points on the image plane into the world coordinate system using the camera parameters. The measurement target 500 reconstructed in the three-dimensional space is called a three-dimensional model. The three-dimensional model of the measurement target 500 is represented by, for example, a set of third three-dimensional points indicating the third three-dimensional positions of a plurality of measurement points on the measurement target 500 (the surface of the measurement target 500) reflected in the multi-viewpoint images. The set of three-dimensional points is referred to as a three-dimensional point cloud. The three-dimensional position indicated by each three-dimensional point in the three-dimensional point cloud is represented by, for example, the three-dimensional coordinates of three-value information consisting of the X component, the Y component, and the X component of the three-dimensional coordinate space composed of the XYZ axes. Note that the three-dimensional model may include not only the three-dimensional coordinates but also information representing the color of each point or the surface shape of each point and its periphery.

[0089] At this time, the generation unit 204 may acquire in advance the camera parameters of each camera, or may estimate them simultaneously with the creation of the three-dimensional model. The camera parameters include internal parameters such as the focal length and the image center of the camera, and external parameters indicating the three-dimensional position and orientation of the camera.

[0090] FIG. 7 shows an example of a typical pinhole camera model. In this model, the lens distortion of the camera is not considered. When considering the lens distortion, the generation unit 204 uses the corrected position obtained by normalizing the position of the point in the image plane coordinates by the distortion model.

[0091] To actually calculate the three-dimensional position, the generation unit 204 uses two or more images with different viewpoints for which the camera parameters are specified among the multi-viewpoint images. The method for calculating the three-dimensional position will be described with reference to FIG. 8. FIG. 8 is a diagram for explaining the method for calculating the three-dimensional position using the multi-viewpoint images.

[0092] The generation unit 204 sets one of the multi-viewpoint images as the reference image 512, and sets the other images as the reference images 511 and 513. The generation unit 204 calculates three-dimensional points corresponding to each pixel of the reference image 512 by using the multi-viewpoint images. Specifically, the generation unit 204 identifies the correspondence relationship of each pixel among the multi-viewpoint images, and performs triangulation using the pixels in the identified correspondence relationship and the camera parameters to calculate the distance from each viewpoint to the measurement target 500. When the generation unit 204 executes processing for each pixel of the reference image 512, it searches for the pixel corresponding to the pixel 522 to be processed from the reference images 511 and 513. When the generation unit 204 acquires the pixels 521 and 523 corresponding to the pixel 522 from the reference images 511 and 513, it can calculate the three-dimensional position of the measurement point 501 by triangulation based on the positions and orientations (postures) of the cameras that captured each image of the multi-viewpoint images. Note that the positions and orientations (postures) of the cameras that captured each image of the multi-viewpoint images are indicated by the external parameters among the camera parameters.

[0093] As the number of reference images increases, the number of triangulation times for one pixel of the reference image 512 increases, so the accuracy of the three-dimensional position of the measurement point 501 improves. For example, even for the same three-dimensional position of the measurement point 501, the three-dimensional points using the reference image 512 and the reference image 511 in FIG. 8 and the three-dimensional points using the reference image 512 and the reference image 513 are slightly different in position. Therefore, calculating the three-dimensional position of one measurement point 501 using two or more three-dimensional points improves the accuracy compared to calculating the three-dimensional position of one measurement point 501 by adopting only one of the three-dimensional points. For example, the generation unit 204 calculates a plurality of candidates for the three-dimensional points of the measurement point 501, and finally calculates the three-dimensional points of the measurement point 501 with high accuracy by a method of estimating from the average point and the degree of variation.

[0094] FIG. 9 is a diagram showing the epipolar constraint of the pair of feature points between two images.

[0095] When a two-dimensional point m obtained by imaging a three-dimensional point M in a three-dimensional space in image 531 is used as a feature point, an example of obtaining a feature point corresponding to the two-dimensional point m from image 532 using epipolar constraint will be described. First, using the external parameters of each camera, the optical center C of the camera that captured image 531 and the optical center C' of the camera that captured image 532 are obtained. Then, using the optical center C of the camera and the coordinates of the two-dimensional point m in image 531, a straight line 533 in the three-dimensional space passing through the optical center C and the two-dimensional point m is calculated. Next, using the straight line 533 and the external parameters of the camera that captured image 532, an epipolar line 534, which is a line corresponding to the straight line 533 on image 532, is calculated. And a three-dimensional point candidate can be obtained by triangulating the feature points on the epipolar line 534 in image 532. That is, all the feature points on the epipolar line 534 can be used as candidate points for specifying a two-dimensional point m' corresponding to the two-dimensional point m on the straight line 533.

[0096] Figure 10 is a diagram for explaining a method of estimating camera parameters and a method of generating a third three-dimensional model.

[0097] In the method of estimating camera parameters and the method of generating a third three-dimensional model, using the epipolar constraint described with reference to FIG. 9, the coordinates and orientation of the camera in the world coordinate system O w are calculated, and further, the three-dimensional position of a point on an image captured by the camera in the world coordinate system O w is calculated. The internal parameters of the camera are known, and an example of estimating the external parameters of the camera and generating a third three-dimensional model of the measurement object 500 using three frames (image 541, image 542, and image 543) will be described.

[0098] In order to obtain the camera parameters of each camera, it is necessary to calculate the rotation matrices R1, R2, R3 and the translation vectors T1, T2, T3 of the cameras in the world coordinate system with the origin at 0. First, a method for calculating the rotation matrix and the translation vector of the camera that captured Image 541 and Image 542 will be described. When a point m1 = (u1, v1, 1) on Image 541 corresponds to a point m2 on Image 542, an epipolar equation that satisfies (Equation 1) holds between the two.

[0099]

Number

[0100] Here, F is called the Fundamental matrix (F matrix). The generation unit 204 can obtain each point as a point m1 = (x1, y1, z1) and m2 = (x2, y2, z2) in each camera coordinate system by the conversion formula shown in (Equation 2) using the internal parameter K of each camera. The epipolar equation can be rewritten as (Equation 3).

[0101]

Number

[0102]

Number

[0103] Here, E is called the Essential matrix (E matrix). The generation unit 204 can calculate each element of the E matrix using a plurality of corresponding points. Also, after the generation unit 204 calculates each element of the F matrix using a plurality of corresponding points such as the points m1 and m2 between the images, it may obtain the E matrix by the conversion formula of (Equation 4).

[0104] E = K -1 FK (Equation 4)

[0105] By decomposing this E matrix, the generation unit 204 can obtain the rotation matrix and translation vector from the image 541 to the image 542 in the world coordinate system. When the position of the first camera in the world coordinate system and the inclination of the first camera with respect to each axis of the world coordinate system are known, the generation unit 204 can use the relative relationship between the first camera and the second camera to obtain the positions and postures of the first camera and the second camera in the world coordinate system. The generation unit 204 may calculate the position and posture of the first camera in the world coordinate system using information of cameras other than the video (for example, information obtained by sensors such as a gyro sensor or an acceleration sensor provided in the camera), or may measure it in advance. Also, with the camera coordinate system of the first camera as the world coordinate system, the positions and postures of other cameras may be calculated.

[0106] When considering the lens distortion of the camera, the generation unit 204 corrects the position of the point on the image using a distortion model, and obtains the F matrix or the E matrix using the corrected position. As an example, the generation unit 204 uses a radial distortion model of the lens shown in (Equation 5).

[0107] u undistorted = u(1 + k1r 2 + k2r 4 + k3r 6 ) v undistorted = v(1 + k1r 2 + k2r 4 + k3r 6 ) r 2 = u 2 + v 2 (Equation 5)

[0108] Also, the generation unit 204 can obtain the coordinates of the three-dimensional point M of the corresponding point on the world coordinate system of the corresponding point by the triangle formed by the rotation matrix and the translation vector of the image 541 and the image 542.

[0109] In addition, the above geometric relationship can be extended to three viewpoints. When adding Image 543 to Images 541 and 542, the generation unit 204 calculates the E matrix for Images 542 and 543, and Images 541 and 543 respectively, and obtains the relative rotation matrix and translation vector between each camera. By integrating these, the generation unit 204 can calculate the rotation matrix and translation vector in the world coordinate system of the camera of Image 543.

[0110] Alternatively, the rotation matrix and translation vector of Image 543 may be calculated from corresponding points between Image 543 and Image 541, and between Image 543 and Image 542 respectively. Specifically, corresponding points are obtained between Image 541 and Image 543, and between Image 542 and Image 543. Here, assuming that point m3 on Image 543 corresponding to point m1 on Image 541 and point m2 on Image 542 is obtained, since the coordinates of the three-dimensional point M of this corresponding point can be acquired, the correspondence between the points on Image 543 and the coordinates in the three-dimensional space can be obtained. At this time, (Equation 6) holds.

[0111]

Equation

[0112] Here, P is called the Perspective matrix (P matrix). Since the relationship of (Equation 7) holds for the P matrix, E matrix, and internal matrix, the generation unit 204 can obtain the E matrix of Image 543, and thereby obtain the rotation matrix and translation vector.

[0113] P = KE (Equation 7)

[0114] Even when the internal parameters are unknown, after the generation unit 204 calculates the F matrix or P matrix, it is possible to obtain the internal matrix and E matrix by dividing the F matrix and P matrix under the constraints that the internal matrix is an upper triangular matrix and the E matrix is a positive definite symmetric matrix.

[0115] Return to the description of FIG. 6. After step S121, the generation unit 204 performs matching between the first three-dimensional model and the third three-dimensional model, and calculates coordinate axis difference information indicating differences in scale, orientation, and position on each coordinate axis (S122). Specifically, the generation unit 204 uses the first position information included in the first three-dimensional model and the third position information included in the third three-dimensional model to perform matching to identify a plurality of first three-dimensional positions among the first position information and a plurality of third three-dimensional positions among the third position information that are in a corresponding relationship with each other. Then, the generation unit 204 calculates the coordinate axis difference information between the first position information and the third position information using the matching result. The difference in scale on the coordinate axis is, for example, the ratio of the sizes of the first three-dimensional model and the third three-dimensional model. The difference in position is, for example, the difference in distance between a point on the first three-dimensional model and a point on the third three-dimensional model corresponding to a specific point of the measurement target 500. The difference in orientation is, for example, the difference between a specific orientation of the first three-dimensional model and a specific orientation of the third three-dimensional model corresponding to a specific direction of the measurement target.

[0116] In the matching process of matching the first three-dimensional model and the third three-dimensional model, the generation unit 204 performs the matching process using either one of two methods. As the first method, the generation unit 204 performs a matching process of aligning a part of the first position information and a part of the third position information so that the error between a part of the first position information of the first three-dimensional model and a part of the third position information of the third three-dimensional model is minimized. Also, as the second method, the generation unit 204 performs a matching process of aligning a part of the first position information and a part of the third position information so that the error between all of the first position information of the first three-dimensional model and all of the third position information of the third three-dimensional model is minimized. For example, when the reliability of the third three-dimensional model indicates a reliability higher than a predetermined reliability, the generation unit 204 may perform the matching process using the first method, and when the reliability of the third three-dimensional model indicates a reliability lower than the predetermined reliability, the generation unit 204 may perform the matching process using the second method.

[0117] For the reliability of the third three-dimensional model, the following indicators may be used. In calculating the indicators, the generation unit 204 uses the camera parameters of the camera that captured the multi-viewpoint images on which the third three-dimensional model is based, and reprojects each of the plurality of third three-dimensional positions indicated by the third position information onto each image of the multi-viewpoint images. Then, the generation unit 204 calculates the reliability indicated by the error between the position of the pixel on each image on which each third three-dimensional position is calculated and the position of the reprojected pixel. The smaller the calculated reliability value is, the higher the reliability means.

[0118] FIG. 11 is a diagram for explaining a first method of the matching process. FIG. 11(a) shows a first three-dimensional model 551 generated by the measuring instrument 100, and FIG. 11(b) shows a third three-dimensional model 552 generated using multi-viewpoint images captured by a plurality of cameras 101. In the first method, the three-dimensional model generation device 200 accepts manual selection of at least three three-dimensional points included in the first three-dimensional model 551 and at least three three-dimensional points included in the third three-dimensional model 552 that are in a corresponding relationship with each other. Specifically, the three-dimensional model generation device 200 accepts from the user an input indicating that the three-dimensional point 561 and the three-dimensional point 571 are in a corresponding relationship, the three-dimensional point 562 and the three-dimensional point 572 are in a corresponding relationship, and the three-dimensional point 563 and the three-dimensional point 573 are in a corresponding relationship, via an input IF (not shown) such as a touch panel, a keyboard, or a mouse. Thereby, the generation unit 204 can identify that the three-dimensional points 561 to 563 of the first three-dimensional model 551 and the three-dimensional points 571 to 573 of the third three-dimensional model 552 are in a corresponding relationship according to the accepted input, and thus can calculate the coordinate axis difference information.

[0119] FIG. 12 is a diagram for explaining a second method of the matching process. FIG. 12(a) shows a first three-dimensional model 551 generated by the measuring instrument 100, and FIG. 12(b) shows a third three-dimensional model 552 generated using multi-viewpoint images captured by a plurality of cameras 101. In the second method, the generation unit 204 uses, for example, the following ICP (Iterative Closest Point) calculation formula (Equation 8) to minimize the error between all the three-dimensional points included in the first three-dimensional model 551 and all the three-dimensional points included in the third three-dimensional model 552. By aligning the scale, position, and orientation on one coordinate axis with the other, the coordinate axis difference information between the first position information and the third position information is calculated. Note that in the second method, instead of calculating the coordinate axis difference information, the three-dimensional coordinate axes of the first three-dimensional model 551 and the third three-dimensional model 552 may be aligned.

[0120]

Equation

[0121] Note that although the generation unit 204 uses all the three-dimensional points included in the first three-dimensional model 551 and all the three-dimensional points included in the third three-dimensional model 552, it is not necessary to use all the three-dimensional points. The ICP matching process may be performed between some of the three-dimensional points at a predetermined ratio of the whole. Note that points showing a reliability higher than a predetermined reliability due to the reprojection error may be extracted, and the ICP matching process may be performed using the extracted three-dimensional points.

[0122] Note that when the generation unit 204 uses an image captured by the built-in camera of the measuring instrument 100 or an image captured at the same position as the camera for generating the third three-dimensional model, the image captured by the built-in camera of the measuring instrument 100 is common to the first three-dimensional model and the third three-dimensional model. That is, since the correspondence between the first three-dimensional model and the third three-dimensional model can be specified without performing the matching process in step S122, the matching process in step S122 may not be performed.

[0123] That is, in this case, the receiving unit 201 acquires the image captured by the built-in camera of the measuring instrument 100, or the image captured at the same position as the camera, and the second positional relationship that is the positional relationship between the first three-dimensional model and the image. The image captured by the built-in camera of the measuring instrument 100 is an example of a specific image in which the second positional relationship that is the positional relationship with the first three-dimensional model is specified. The specific image may be, in addition to the image captured by the built-in camera of the measuring instrument 100, an image captured at the same position as the built-in camera of the measuring instrument 100, or an image captured from a position where the positional relationship with the measuring instrument 100 is specified. The generation unit 204 generates a third three-dimensional model using the multi-viewpoint image and the specific image. The generation unit 204 specifies the first relationship that is the positional relationship between the first three-dimensional model and the multi-viewpoint image using the third three-dimensional model and the second positional relationship.

[0124] According to this, by generating the third three-dimensional model using, together with the multi-viewpoint image, the specific image in which the second positional relationship with the measuring instrument has already been specified, the first positional relationship between the first three-dimensional model and the multi-viewpoint image can be easily specified.

[0125] Returning to the description of FIG. 6. After step S122, the generation unit 204 uses the calculated coordinate axis difference information to convert the coordinate axis of the first three-dimensional model to the coordinate axis of the third three-dimensional model (S123). Thereby, the generation unit 204 aligns the three-dimensional coordinate axis of the first three-dimensional model and the three-dimensional coordinate axis of the third three-dimensional model. Therefore, the generation unit 204 can specify the first positional relationship between the first three-dimensional model and the multi-viewpoint image. Note that the generation unit 204 may convert the coordinate axis of the third three-dimensional model to the coordinate axis of the first three-dimensional model.

[0126] The details of the high-definition processing (S112) by the generation unit 204 will be described. The high-definition processing is performed by the following three methods. In the high-definition processing, any one of the three methods may be used, or a combination of two or more may be used.

[0127] A first method of the refinement process (S112) by the generation unit 204 will be described with reference to FIGS. 13 and 14. FIG. 13 is a flowchart showing the first method of the refinement process. FIG. 14 is a diagram for explaining the first method of the refinement process.

[0128] In the first method, for each of a plurality of first three-dimensional positions included in the first three-dimensional model, the generation unit 204 uses the first positional relationship and the multi-viewpoint images to change the first color information corresponding to the first three-dimensional position to second color information based on the pixels of the multi-viewpoint image corresponding to the first three-dimensional position. Thereby, the generation unit 204 generates a second three-dimensional model with the color information of the first three-dimensional model refined to a high precision.

[0129] Specifically, the generation unit 204 performs Loop 1 including the following step S131 and step S132 for each of a plurality of three-dimensional points 551a indicating a plurality of first three-dimensional positions included in the first three-dimensional model 551.

[0130] The generation unit 204 projects the three-dimensional point 551a of the first three-dimensional position to be processed onto the image 581 captured at the position (viewpoint) closest to the first three-dimensional position among the plurality of images 581 to 583 of the multi-viewpoint image (S131). Thereby, the generation unit 204 identifies the pixel 591 in the image 581 captured at the position closest to the first three-dimensional position to be processed among the multi-viewpoint images, which is the pixel 591 capturing the three-dimensional point 551a.

[0131] Note that although the generation unit 204 selects, as the image onto which the first three-dimensional position to be processed is projected, the image captured at the position closest to the first three-dimensional position among the multi-viewpoint images, the present invention is not limited to this. The generation unit 204 may calculate the normal vector in the first three-dimensional model of the first three-dimensional position to be processed, and select, from among the multi-viewpoint images, the image captured in the shooting direction with the smallest difference from the calculated normal vector as the image onto which the first three-dimensional position to be processed is projected.

[0132] Next, the generation unit 204 changes the first color information of the first three-dimensional position to the second color information indicated by the pixel value of the pixel 591 specified in step S131 (S132).

[0133] Further, when it is detected that a plurality of first three-dimensional positions are projected onto one pixel, the generation unit 204 may change the first color information of the first three-dimensional position that is closest to the position of the camera 101 when the image having the pixel is taken, to the second color information indicated by the pixel value of the pixel. In this case, among the plurality of first three-dimensional positions, for the first three-dimensional position that is second closest to the position of the camera 101 when the image having the pixel is taken, as the image that projects the first three-dimensional position, it may be selected from among the plurality of images obtained by removing the above image from the multi-viewpoint image by the same method as in step S131. If there are third and subsequent first three-dimensional positions, they can be selected by the same method.

[0134] Thereby, using the multi-viewpoint image in which the first positional relationship with the first three-dimensional model is specified, the first color information of the first three-dimensional model can be changed to the second color information with higher accuracy than the first color information.

[0135] Next, a second method of the refinement process (S112) by the generation unit 204 will be described with reference to FIGS. 15 and 16. FIG. 15 is a flowchart showing the second method of the refinement process. FIG. 16 is a diagram for explaining the second method of the refinement process.

[0136] In the second method, the generation unit 204 interpolates the second three-dimensional position on the measurement target 500 between two positions included in a plurality of first three-dimensional positions included in the first three-dimensional model, using the first positional relationship and the multi-viewpoint image. Thereby, the generation unit 204 generates a second three-dimensional model including the plurality of first three-dimensional positions and the interpolated second three-dimensional position.

[0137] Specifically, the generation unit 204 projects a first three-dimensional point group 553 indicating a plurality of first three-dimensional positions included in the first three-dimensional model 551 onto a plurality of images 581 to 583 of the multi-viewpoint image (S141). The generation unit 204 projects the first three-dimensional point group 553 onto all the images 581 to 583 of the multi-viewpoint image.

[0138] Next, for each of the plurality of images 581 to 583 included in the multi-viewpoint image, the generation unit 204 generates a triangle group having a plurality of pixels onto which the first three-dimensional point group is projected as vertices (S142). Specifically, as shown in FIG. 16, the generation unit 204 generates a plurality of triangles 602 by connecting a plurality of pixels 593 onto which the first three-dimensional point group 553 is projected in the image 581 with line segments 601.

[0139] The generation unit 204 performs a loop 2, which is a double loop for each of the plurality of images 581 to 583 included in the multi-viewpoint image and for each triangle 602 of the triangle group generated on the image. Loop 2 includes the following steps S143 to S145.

[0140] The generation unit 204 calculates the texture intensity inside the triangle 602 to be processed, and based on the calculated texture intensity, determines whether the triangle 602 is a flat part or a texture part (S143).

[0141] When the generation unit 204 determines that the triangle 602 to be processed is a flat part (flat part in S143), it calculates the three-dimensional points inside the triangle by linearly interpolating the inside of the triangle in the three-dimensional space defined by the three first three-dimensional positions corresponding to the vertices of the triangle 602 (S144).

[0142] When the generation unit 204 determines that the triangle 602 to be processed is a texture part (texture part in S143), for each pixel inside the triangle 602, similar points are detected from the images of the multi-viewpoint image, and triangulation is performed between the detected similar points to calculate the three-dimensional point corresponding to the pixel (S145). The triangulation can use the method described with reference to FIGS. 8 and 9. In step S145, the generation unit 204 may interpolate the three-dimensional position corresponding to the inside of the triangle 602 among the third three-dimensional models generated using the multi-viewpoint image.

[0143] Thereby, using the multi-viewpoint image in which the first positional relationship with the first three-dimensional model is specified, the three-dimensional position between the two first three-dimensional positions of the first three-dimensional model can be interpolated. Therefore, a second three-dimensional model with a higher density than the first three-dimensional model can be generated.

[0144] Next, a third method of the high-definition processing (S112) by the generation unit 204 will be described with reference to FIGS. 17 and 18. FIG. 17 is a flowchart showing the third method of the high-definition processing. FIG. 18 is a diagram for explaining the third method of the high-definition processing.

[0145] In the third method, the generation unit 204 detects the missing part 554 of the first position information included in the first three-dimensional model 551A, and uses the first positional relationship and the multi-viewpoint image to interpolate the third three-dimensional position on the measurement target 500 in the detected missing part 554. Thereby, the generation unit 204 generates a second three-dimensional model including a plurality of first three-dimensional points and the interpolated third three-dimensional position.

[0146] Specifically, the generation unit 204 projects the first three-dimensional point group 553A indicating a plurality of first three-dimensional positions included in the first three-dimensional model 551A onto a plurality of images 581 to 583 of the multi-viewpoint image (S151). The generation unit 204 projects the first three-dimensional point group 553A onto all the images 581 to 583 of the multi-viewpoint image.

[0147] Next, the generation unit 204 performs loop 3 including the following steps S152 to S155 for each of the plurality of images 581 to 583 included in the multi-viewpoint image.

[0148] For each of the plurality of pixels constituting the image 581 to be processed, the generation unit 204 detects the missing part 554 of the first position information by detecting a pixel in which the first three-dimensional point group is not projected onto the region 603 within a certain distance r1 or less from the pixel (S152). For example, the generation unit 204 detects, as the missing part 554, a region in the three-dimensional space corresponding to the region 603 within a certain distance r1 or less from the detected pixel. Finally, the generation unit 204 detects, as the missing part 554, a region in the three-dimensional space corresponding to the sum region of the plurality of regions. Note that in FIG. 18, one region 603 is illustrated.

[0149] The generation unit 204 calculates the texture intensity of the region 603 corresponding to the missing part 554, and determines whether the region 603 is a flat part or a texture part based on the calculated texture intensity (S153).

[0150] When the generation unit 204 determines that the region 603 corresponding to the missing part 554 is a flat part (flat part in S153), the generation unit 204 calculates the three-dimensional points inside the missing part 554 by linearly interpolating the inside of the missing part 554 in the three-dimensional space defined by a plurality of first three-dimensional positions corresponding to the first three-dimensional point group projected around the missing part on the image to be processed (S153).

[0151] When the generation unit 204 determines that the region 603 corresponding to the missing part 554 is a texture part (texture part in S153), for each pixel inside the missing part 554, the generation unit 204 detects similar points from the other images 582 and 583 of the multi-viewpoint image, and calculates the three-dimensional points corresponding to the pixels by performing triangulation with the detected similar points (S155). The triangulation can use the method described with reference to FIGS. 8 and 9. Note that in step S155, the generation unit 204 may interpolate the three-dimensional positions corresponding to the inside of the missing part 554 in the third three-dimensional model generated using the multi-viewpoint image.

[0152] Thus, even when a missing part occurs in the first three-dimensional model due to occlusion or the like during measurement by the measuring instrument 100, the three-dimensional position of the missing part of the first three-dimensional model can be interpolated using the multi-viewpoint image in which the first positional relationship with the first three-dimensional model is specified.

[0153] [Effects, etc.] The three-dimensional data generation method according to the present disclosure is a three-dimensional model generation method executed by a three-dimensional model generation device 200 as an information processing device, and includes a first acquisition step (S101) of acquiring a first three-dimensional model having first position information indicating a plurality of first three-dimensional positions on a measurement target 500 by emitting an electromagnetic wave and acquiring a reflected wave in which the emitted electromagnetic wave is reflected by the measurement target 500 from a measuring instrument 100, a second acquisition step (S101) of acquiring multi-viewpoint images taken by one or more cameras 101 from a plurality of viewpoints different from the measurement target 500, and a generation step (S104) of generating a second three-dimensional model by refining the first three-dimensional model using the multi-viewpoint images.

[0154] According to this, at least one of the first position information and the first color information of the first three-dimensional model including the high-precision position information obtained by the measuring instrument 100 is refined using the multi-viewpoint images taken by the easily portable camera 101. For this reason, the generation accuracy of the three-dimensional model can be improved, and the processing time of the generation process of the three-dimensional model can be shortened.

[0155] Further, in the three-dimensional data generation method according to the present disclosure, in the generation step (S104), a third three-dimensional model is generated using the multi-viewpoint images (S121), the first positional relationship between the first three-dimensional model and the multi-viewpoint images is specified by aligning the three-dimensional coordinate axes of the first three-dimensional model and the three-dimensional coordinate axes of the third three-dimensional model (S123), and the second three-dimensional model is generated by refining the first three-dimensional model using the specified first positional relationship and the multi-viewpoint images (S112).

[0156] Therefore, in order to refine the first three-dimensional model using the multi-viewpoint image in which the first positional relationship with the first three-dimensional model is specified, the first three-dimensional model can be refined more effectively.

[0157] [Modification Example] In the above embodiment, the first three-dimensional model is assumed to include the first color information. However, the present invention is not limited to this, and the first three-dimensional model may not include the first color information. That is, the first three-dimensional model only needs to include the first position information. In this case, in the three-dimensional model generation device 200, the generation unit 204 adds, for each of the plurality of first three-dimensional positions indicated by the first position information, the first positional relationship specified in step S111 and the second color information based on the pixels of the multi-viewpoint image corresponding to the first three-dimensional position as the color information corresponding to the first three-dimensional position, thereby generating a second three-dimensional model. Therefore, high-precision color information can be added to the first three-dimensional model using the multi-viewpoint image in which the first positional relationship with the first three-dimensional model is specified.

[0158] (Others) As described above, the three-dimensional model generation method and the like according to the present disclosure have been described based on the above embodiments. However, the present disclosure is not limited to the above embodiments.

[0159] For example, in the above-described embodiment and modification example, the second three-dimensional model is generated by changing the first three-dimensional model. However, the second three-dimensional model may be generated by changing the first three-dimensional model with the third three-dimensional model generated from the multi-viewpoint image. Further, the second three-dimensional model may be generated based on these three-dimensional models without changing the first three-dimensional model and the third three-dimensional model.

[0160] For example, in the above embodiment, it was described that each processing unit included in the three-dimensional model generation device or the like is realized by a CPU and a control program. For example, the components of the processing unit may each be composed of one or more electronic circuits. Each of the one or more electronic circuits may be a general-purpose circuit or a dedicated circuit. The one or more electronic circuits may include, for example, a semiconductor device, an IC (Integrated Circuit), or an LSI (Large Scale Integration). The IC or LSI may be integrated on one chip or on a plurality of chips. Here, although it is called an IC or LSI, the name may change depending on the degree of integration, and it may be called a system LSI, a VLSI (Very Large Scale Integration), or a ULSI (Ultra Large Scale Integration). Also, an FPGA (Field Programmable Gate Array) programmed after the manufacture of the LSI can be used for the same purpose.

[0161] In addition, the general or specific aspects of the present disclosure may be realized by a system, a device, a method, an integrated circuit, or a computer program. Alternatively, it may be realized by a computer-readable non-transitory recording medium such as an optical disk, an HDD (Hard Disk Drive), or a semiconductor memory in which the computer program is stored. Also, it may be realized by any combination of a system, a device, a method, an integrated circuit, a computer program, and a recording medium.

[0162] In addition, forms obtained by applying various modifications that can be conceived by those skilled in the art to each embodiment, and forms realized by arbitrarily combining the components and functions in the embodiment without departing from the spirit of the present disclosure are also included in the present disclosure.

Industrial Applicability

[0163] The present disclosure can be applied to a three-dimensional model generation device or a three-dimensional model generation system, and can be applied to, for example, figure creation, terrain or building structure recognition, human behavior recognition, or free viewpoint video generation, etc.

Explanation of Signs

[0164] 100 Measuring instrument 101 Camera 111 Laser irradiation unit 112 Laser light receiving unit 200 Three-dimensional model generation device 201 Receiver 202 Storage unit 203 Acquisition unit 204 Generation unit 205 Output unit 400 Three-dimensional model generation system 500 Measurement target 501 Measurement point 511, 513 Reference image 512 Reference image 521~523, 591, 593, 594 Pixel 531, 532, 541~543, 581~583 Image 533 Straight line 534 Epipolar line 551, 551A First three-dimensional model 551a Three-dimensional point 552 Third three-dimensional model 553, 553A Three-dimensional point cloud 554 Defective part 561~563, 571~573 Three-dimensional point 601 Line segment 602 Triangle

Claims

1. A three-dimensional model generation method executed by an information processing apparatus, comprising: a first acquisition step of acquiring a first three-dimensional model generated by a measuring instrument that emits electromagnetic waves and acquires a reflected wave obtained by reflecting the electromagnetic waves from a measurement target, the first three-dimensional model having first position information indicating a plurality of first three-dimensional positions on the measurement target; a second acquisition step of acquiring multi-viewpoint images of the measurement target taken by one or more cameras from a plurality of different viewpoints; a generation step of generating a second three-dimensional model of the measurement target based on the multi-viewpoint images and the first three-dimensional model; in the generation step, generating the second three-dimensional model by refining color information included in the first three-dimensional model using the multi-viewpoint images; the second three-dimensional model has color information assigned to units finer than regions where color information is assigned in the first three-dimensional model Three-dimensional model generation method.

2. In the generation step, the positional relationship between the multi-viewpoint images and the first three-dimensional model is specified in a three-dimensional coordinate system, and the second three-dimensional model is generated based on the multi-viewpoint images, the first three-dimensional model, and the positional relationship. The three-dimensional model generation method according to claim 1.

3. In the generation step, generating a third three-dimensional model using the multi-viewpoint images; aligning the three-dimensional coordinate axes of the first three-dimensional model and the three-dimensional coordinate axes of the third three-dimensional model to specify a first positional relationship between the first three-dimensional model and the multi-viewpoint images; generating the second three-dimensional model by refining the first three-dimensional model using the specified first positional relationship and the multi-viewpoint images. The three-dimensional model generation method according to claim 1 or 2.

4. In the generation step, for each of the plurality of first three-dimensional positions, the second three-dimensional model is generated by adding, as color information corresponding to the first three-dimensional position, second color information based on pixels of the multi-viewpoint image corresponding to the first three-dimensional position, using the first positional relationship and the multi-viewpoint images. The three-dimensional model generation method according to claim 3.

5. The first three-dimensional model further has first color information indicating the color of the measurement target at each of the plurality of first three-dimensional positions, generated using an image of the measurement target. Each of the plurality of images included in the multi-viewpoint image is an image with a higher resolution than the image captured by the measuring instrument. In the generation step, for each of the plurality of first three-dimensional positions, by using the first positional relationship and the multi-viewpoint image to change the first color information corresponding to the first three-dimensional position to second color information based on the pixels of the multi-viewpoint image corresponding to the first three-dimensional position, the second three-dimensional model is generated. The three-dimensional model generation method according to claim 3.

6. In the generation step, by using the first positional relationship and the multi-viewpoint image to interpolate a second three-dimensional position on the measurement target between two or more positions included in the plurality of first three-dimensional positions, the second three-dimensional model including the plurality of first three-dimensional positions and the interpolated second three-dimensional position is generated. The three-dimensional model generation method according to any one of claims 3 to 5.

7. In the generation step, a missing portion of the first position information is detected, and by using the first positional relationship and the multi-viewpoint image to interpolate a third three-dimensional position on the measurement target at the detected missing portion, the second three-dimensional model including the plurality of first three-dimensional positions and the interpolated third three-dimensional position is generated. The three-dimensional model generation method according to any one of claims 3 to 6.

8. In the generation step, a third three-dimensional model is generated using the multi-viewpoint image, and the second three-dimensional model is generated by refining the third three-dimensional model using the first three-dimensional model. The three-dimensional model generation method according to claim 1.

9. Furthermore, including a third acquisition step of acquiring a specific image in which a second positional relationship with the first three-dimensional model is specified and the second positional relationship, In the generation step, a third three-dimensional model is generated using the multi-viewpoint image and the specific image, the first positional relationship between the first three-dimensional model and the multi-viewpoint image is specified by using the third three-dimensional model and the second positional relationship, and the second three-dimensional model is generated by refining the first three-dimensional model using the specified first positional relationship and the multi-viewpoint image. The three-dimensional model generation method according to claim 1 or 2.

10. A first three-dimensional model generated by a measuring instrument that emits electromagnetic waves and acquires a reflected wave obtained by reflecting the electromagnetic waves from a measurement target, the first three-dimensional model having first position information indicating a plurality of first three-dimensional positions on the measurement target, and a first acquisition unit that acquires the first three-dimensional model; A second acquisition unit that acquires multi-viewpoint images of the measurement target taken by one or more cameras from a plurality of different viewpoints; A generation unit that generates a second three-dimensional model of the measurement target based on the multi-viewpoint images and the first three-dimensional model, wherein the generation unit generates the second three-dimensional model by refining color information included in the first three-dimensional model using the multi-viewpoint images, and the second three-dimensional model has color information assigned in units finer than the regions where color information is assigned in the first three-dimensional model An information processing apparatus.

11. A program for causing a computer to execute a three-dimensional model generation method, wherein the three-dimensional model generation method includes a first acquisition step of acquiring a first three-dimensional model generated by a measuring instrument that emits electromagnetic waves and acquires a reflected wave obtained by reflecting the electromagnetic waves from a measurement target, the first three-dimensional model having first position information indicating a plurality of first three-dimensional positions on the measurement target; a second acquisition step of acquiring multi-viewpoint images of the measurement target taken by one or more cameras from a plurality of different viewpoints; and a generation step of generating a second three-dimensional model of the measurement target based on the multi-viewpoint images and the first three-dimensional model, wherein in the generation step, the second three-dimensional model is generated by refining color information included in the first three-dimensional model using the multi-viewpoint images, and the second three-dimensional model has color information assigned in units finer than the regions where color information is assigned in the first three-dimensional model A program.

Images