Three-dimensional model generation method and three-dimensional model generation device

Abstract

The three-dimensional model generation method includes: a first three-dimensional model generation step (step S105), generating a first three-dimensional model of a specified area based on multiple first frames; a determination step (step S106), projecting the first three-dimensional model to at least one second frame, and determining multiple unprojected pixels in at least one second frame to which the first three-dimensional model is not projected; and a second three-dimensional model generation step (step S107), generating a second three-dimensional model based on the multiple unprojected pixels.

Description

Technical Field

[0001] The invention relates to a three-dimensional model generating method and a three-dimensional model generating device. Background Art

[0002] In recent years, there is a technology that uses multi-viewpoint images obtained by photographing a subject such as a person or a building from multiple viewpoints to measure the three-dimensional shape of the subject and generate a three-dimensional model of the subject (for example, see Patent Document 1).

[0003] In the technology disclosed in Patent Document 1, after geometrically estimating the camera parameters (camera position, orientation, focal length in pixel units, lens distortion, etc.) of each image constituting a multi-viewpoint image when it is captured, each pixel on each image is projected into a three-dimensional space using the estimated camera parameters, thereby generating a three-dimensional model of the subject.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2015-056142 Summary of the invention

[0007] Problems to be solved by the invention

[0008] In the process of generating a three-dimensional model, it is desirable that the processing time be short.

[0009] The present invention provides a three-dimensional model generation method and the like capable of shortening the processing time of the generation process of the three-dimensional model.

[0010] Means for solving problems

[0011] A three-dimensional model generation method according to one form of the present invention comprises: a first three-dimensional model generation step of generating a first three-dimensional model of a specified area based on a plurality of first frames; a determination step of projecting the first three-dimensional model onto at least one second frame and determining a plurality of unprojected pixels in the at least one second frame to which the first three-dimensional model is not projected; and a second three-dimensional model generation step of generating a second three-dimensional model based on the plurality of unprojected pixels.

[0012] In addition, a three-dimensional model generation device according to one form of the present invention comprises: a first three-dimensional model generation unit, which generates a first three-dimensional model of a specified area based on multiple first frames; a determination unit, which projects the first three-dimensional model to at least one second frame and determines multiple unprojected pixels in the at least one second frame to which the first three-dimensional model is not projected; and a second three-dimensional model generation unit, which generates a second three-dimensional model based on the unprojected pixels.

[0013] In addition, another form of the three-dimensional model generation method of the present invention includes: a first three-dimensional model generation step, generating a first three-dimensional model of a specified area based on multiple first frames; a determination step, projecting the first three-dimensional model to at least one second frame, and determining a plurality of projection pixels in the at least one second frame where the first three-dimensional model is projected; a judgment step, judging whether at least one second shooting position where the at least one second frame is captured is closer to the specified area than any one of the multiple first shooting positions where the multiple first frames are captured; and an updating step, updating the first three-dimensional model based on the plurality of projection pixels before the first three-dimensional model is projected, when it is judged that the at least one second shooting position is closer to the specified area than any one of the multiple first shooting positions.

[0014] In addition, the present invention can also be implemented as a program that causes a computer to execute the steps included in the above-mentioned three-dimensional model generation method. In addition, the present invention can also be implemented as a non-temporary recording medium such as a CD-ROM that can be read by a computer and records the program. In addition, the present invention can also be implemented as information, data or signals representing the program. Moreover, these programs, information, data and signals can also be distributed via a communication network such as the Internet.

[0015] Effects of the Invention

[0016] According to the present invention, it is possible to provide a three-dimensional model generation method and the like that can shorten the processing time of the three-dimensional model generation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a diagram for explaining an overview of the three-dimensional model generation method according to the first embodiment.

[0018] Figure 2 This is a block diagram showing a characteristic structure of the three-dimensional model generating device according to the first embodiment.

[0019] Figure 3 This is a block diagram showing a detailed configuration of a generation unit included in the three-dimensional model generation device according to the first embodiment.

[0020] Figure 4 This is a flowchart showing the processing executed by the three-dimensional model generation device according to the first embodiment.

[0021] Figure 5A It is a diagram for explaining a first example of the photographing sequence of a camera.

[0022] Figure 5B This is a diagram for explaining a first example of the processing sequence of the three-dimensional model generating device with respect to the imaging sequence of the camera.

[0023] Fig. 6AIt is a diagram for explaining the second example of the photographing sequence of the camera.

[0024] Figure 6B This is a diagram for explaining a second example of the processing sequence of the three-dimensional model generation device with respect to the imaging sequence of the camera.

[0025] Fig. 7A It is a diagram for explaining the third example of the photographing sequence of the camera.

[0026] Figure 7B This is a diagram for explaining a third example of the processing sequence of the three-dimensional model generating device with respect to the imaging sequence of the camera.

[0027] Fig. 8A It is a diagram for explaining the fourth example of the photographing sequence of the camera.

[0028] Figure 8B This is a diagram for explaining a fourth example of the processing sequence of the three-dimensional model generating device with respect to the imaging sequence of the camera.

[0029] Fig. 9 This is a flowchart showing details of processing executed by a frame set generation unit included in the three-dimensional model generation device according to the first embodiment.

[0030] Fig.10 This is a flowchart showing the details of the processing executed by the generation unit included in the three-dimensional model generation device according to the first embodiment.

[0031] Fig.11 This is a diagram for explaining the processing position of a frame in the generation process executed by the three-dimensional model generation unit included in the three-dimensional model generation device according to the first embodiment.

[0032] Fig.12 It is a diagram for explaining the change of the shooting position of the camera.

[0033] Fig.13 It is a diagram for explaining the change of the imaging range of the camera.

[0034] Fig.14 This is a diagram for explaining a specific example of the generation process executed by the three-dimensional model generation unit included in the three-dimensional model generation device according to the first embodiment.

[0035] Fig.15 This is a block diagram showing a characteristic structure of a three-dimensional model generating device according to the second embodiment.

[0036] Fig.16 This is a block diagram showing the detailed structure of a generation unit included in the three-dimensional model generation device according to the second embodiment.

[0037] Fig.17This is a flowchart showing the details of the processing executed by the three-dimensional model generation unit of the second embodiment.

[0038] Fig.18 This is a diagram for explaining the processing position of a frame in the process executed by the updating unit included in the three-dimensional model generation device according to the second embodiment. DETAILED DESCRIPTION

[0039] (Origin of the present invention)

[0040] In 3D model generation (restoration) technology, when restoring a 3D shape using continuously captured frames (images), there are repeated parts of the subject in multiple frames. Since multiple 3D model generation processes are performed on the subject, unnecessary processing time is increased.

[0041] Therefore, the present invention provides a three-dimensional generation method and the like that can shorten the processing time of the generation process of the three-dimensional model.

[0042] A three-dimensional model generation method according to one form of the present invention comprises: a first three-dimensional model generation step of generating a first three-dimensional model of a specified area based on a plurality of first frames; a determination step of projecting the first three-dimensional model onto at least one second frame and determining a plurality of unprojected pixels in the at least one second frame to which the first three-dimensional model is not projected; and a second three-dimensional model generation step of generating a second three-dimensional model based on the plurality of unprojected pixels.

[0043] Thus, since differential three-dimensional models of the generated three-dimensional models are sequentially generated, models of overlapping portions of the three-dimensional models are not generated, thereby shortening the processing time for generating the three-dimensional models.

[0044] In addition, for example, the three-dimensional model generation method also includes a frame set generation step, which generates a frame set consisting of multiple frames contained in the multiple shooting frames based on the position and posture of the camera in each of the multiple shooting frames, and the frame set includes the multiple first frames and the at least one second frame.

[0045] Thus, since the three-dimensional model is generated using the frames of the frame set consisting of the frames selected from the plurality of frames based on the position and posture of the camera, it is possible to generate a frame set from which unnecessary frames such as a predetermined area not captured are removed. Therefore, since the three-dimensional model generation process for the unnecessary frames is not performed, the processing time can be shortened.

[0046] In addition, for example, in the frame set generating step, the multiple frames are arranged in the frame set in the order of frames, and in the determining step, the third three-dimensional model is projected onto the at least one second frame, the third three-dimensional model is an already generated three-dimensional model including the first three-dimensional model, and a plurality of unprojected pixels to which the third three-dimensional model is not projected in the at least one second frame to which the third three-dimensional model is projected are determined, and the determining step and the second three-dimensional model generating step are repeatedly performed in the order of the frames.

[0047] Thus, projection and generation of the second three-dimensional model are repeatedly performed for each frame. Therefore, the existing three-dimensional model including the first three-dimensional model and the generated second three-dimensional model is projected onto the frame, so that unprojected pixels that are not projected can be determined. Thus, the processing time can be further shortened.

[0048] Furthermore, for example, in the frame set generation step, the frame set is generated in which the plurality of frames are arranged in order from near to far from the prescribed area in terms of the position of the camera when each of the plurality of frames in the frame set was captured.

[0049] Thus, the second three-dimensional model can be generated by sequentially using frames taken by a camera close to the specified area. The closer the frame is to the subject, the higher the accuracy, that is, the higher the image quality. Therefore, the processing time for generating the three-dimensional model can be shortened, and the accuracy of the three-dimensional model can be improved.

[0050] In addition, for example, in the frame set generation step, the following frame set is generated, in which, when the direction from the camera toward the specified area is set as the positive direction of the reference axis, the frame set arranges the multiple frames in the order from near to far from the specified area in the positive direction of the reference axis according to the position of the camera when each frame in the multiple frames in the frame set is shot.

[0051] Thereby, a plurality of frames can be arranged in the order of frames captured by a camera close to a predetermined area based on the reference axis direction.

[0052] Furthermore, for example, in the frame set generating step, the positive direction of the reference axis is calculated based on the optical axis direction of the camera for each frame when the plurality of frames are captured.

[0053] Thus, for example, since the reference axis direction is calculated based on the optical axis direction of the camera, the frame set can be made to exclude unnecessary frames that hardly include the specified area through simple processing. Therefore, the processing time for unnecessary frames when generating a three-dimensional model can be shortened.

[0054] Furthermore, for example, in the frame set generating step, a representative frame is selected from the plurality of frames, and the optical axis direction of the camera which photographed the representative frame is calculated as the positive direction of the reference axis.

[0055] Therefore, compared to the case where the reference axis direction is calculated based on the optical axis direction of each camera that has taken multiple frames, since the reference axis direction is calculated based on the optical axis direction of the camera that has taken representative frames, the processing time required to calculate the reference axis direction can be shortened by selecting only an appropriate number of representative frames.

[0056] Furthermore, for example, in the frame set generation step, it is determined whether each of the plurality of frames satisfies a first condition, and a frame set is generated from a plurality of frames satisfying the first condition among the plurality of frames.

[0057] Thus, by appropriately setting the first condition, the frame set can exclude unnecessary frames such as frames on which all the first three-dimensional models are projected or frames that hardly include the specified area, thereby shortening the processing time for unnecessary frames when generating the three-dimensional model.

[0058] Furthermore, for example, in the frame set generating step, it is determined as the first condition that an angle formed by the positive direction of the reference axis and the optical axis direction of a camera when capturing the plurality of frames included in the frame set is equal to or smaller than a first angle.

[0059] This makes it possible to eliminate unnecessary frames that hardly include the predetermined area from the frame set by simple processing, thereby shortening the processing time for unnecessary frames when generating a three-dimensional model.

[0060] Furthermore, for example, the predetermined area is an area including at least one of a stationary object in stillness and a moving object in motion as a subject.

[0061] Thus, since the object is included in the predetermined area, when measuring the distance between the predetermined area and the camera, for example, the distance can be measured with reference to the object, so the distance can be calculated with high accuracy.

[0062] Furthermore, for example, the at least one second frame is identical to at least one of the plurality of first frames.

[0063] Thus, for example, when determining the second frame from a plurality of frames, it is not necessary to perform a process of excluding the first frame, and thus the processing time is shortened.

[0064] Furthermore, for example, the first frame and the second frame are different from each other.

[0065] Thus, when generating the second three-dimensional model, the second three-dimensional model is not projected onto the first frame, and thus the processing time for generating the second three-dimensional model can be shortened.

[0066] In addition, a three-dimensional model generation device according to one form of the present invention comprises: a first three-dimensional model generation unit, which generates a first three-dimensional model of a specified area based on multiple first frames; a determination unit, which projects the first three-dimensional model to at least one second frame and determines multiple unprojected pixels in the at least one second frame to which the first three-dimensional model is not projected; and a second three-dimensional model generation unit, which generates a second three-dimensional model based on the unprojected pixels.

[0067] Thus, the 3D model generation device sequentially generates differential 3D models of the generated 3D models, and thus does not generate models of overlapping portions of the 3D models. Therefore, the 3D model generation device can shorten the processing time for generating the 3D model.

[0068] In addition, another form of the three-dimensional model generation method of the present invention includes: a first three-dimensional model generation step, generating a first three-dimensional model of a specified area based on multiple first frames; a determination step, projecting the first three-dimensional model to at least one second frame, and determining a plurality of projection pixels in the at least one second frame where the first three-dimensional model is projected; a judgment step, judging whether at least one second shooting position where the at least one second frame is captured is closer to the specified area than any one of the multiple first shooting positions where the multiple first frames are captured; and an updating step, updating the first three-dimensional model based on the plurality of projection pixels before projecting the first three-dimensional model, when it is judged that the at least one second shooting position is closer to the specified area than any one of the multiple first shooting positions.

[0069] Thus, a three-dimensional model is generated using frames captured by a camera in the order along the time series of the captured frames. Therefore, by processing the frames in the order of capturing, a frame set in which the frame order is rearranged is not generated, and thus a three-dimensional model can be generated in a short time. In addition, a new three-dimensional model can be generated in which pixels projected on a frame in a three-dimensional model generated once are updated with pixels of higher precision than that frame (i.e., pixels of a frame captured by a camera closer to the specified area). Thus, a high-precision three-dimensional model can be generated.

[0070] In addition, for example, the determination step may also include a second three-dimensional model generation step, wherein the second three-dimensional model generation step determines a plurality of unprojected pixels of the first three-dimensional model that are not projected in the at least one second frame, and generates a second three-dimensional model based on the plurality of unprojected pixels.

[0071] Thus, the second three-dimensional model can be generated using non-projected pixels, which are pixels that do not exist in the first three-dimensional model.

[0072] Hereinafter, various embodiments of the three-dimensional model generation method of the present invention are described in detail using the accompanying drawings. In addition, each embodiment described below represents a specific example of the present invention. Therefore, the numerical values, shapes, materials, configurations of constituent elements, connection methods, steps, and the order of steps shown in the following embodiments are examples and are not intended to limit the present invention.

[0073] In addition, each figure is a schematic diagram, and is not necessarily a strictly illustrated figure. In addition, in each figure, the same reference numerals are attached to substantially the same structure, and the repeated description may be omitted or simplified.

[0074] (Implementation Method 1)

[0075] [summary]

[0076] First, refer to Figure 1 , an overview of the three-dimensional model generation method of implementation mode 1 is described.

[0077] Figure 1 This is a diagram for explaining an overview of the three-dimensional model generation method according to the first embodiment. Figure 2 This is a block diagram showing a characteristic structure of the three-dimensional model generating device 100 according to the first embodiment.

[0078] The three-dimensional model generation system 400 comprises: a camera group 300, which is composed of multiple cameras 301 as camera devices for photographing a specified area; an estimation device 200, which estimates the positions and postures of the multiple cameras 301 based on multiple frames (in other words, images or video data) photographed by the multiple cameras 301; and a three-dimensional model generation device 100, which inputs the multiple frames photographed by the multiple cameras 301 and the estimation results of the estimation device 200.

[0079] In addition, the posture of the camera 301 indicates at least one of the shooting direction of the camera 301 and the tilt of the camera 301 .

[0080] The prescribed area is an area including a stationary object or a moving object such as a person, or both. In other words, the prescribed area is an area including at least one of a stationary object and a moving object as a subject. Examples of prescribed areas including stationary objects and moving objects include sports games such as basketball, or streets where people or cars exist. In addition, the prescribed area is not limited to a specific object that is a subject, but may also include a scene such as a landscape. Figure 1The case where the subject 500A is a house is exemplified. In the following, not only a specific object to be a subject but also a predetermined area including a landscape or the like will be simply referred to as a subject.

[0081] The plurality of cameras 301 each capture an image of a subject, and outputs the captured images to the estimation device 200. In the first embodiment, the camera group 300 includes two or more cameras 301. The plurality of cameras 301 capture the same subject from different viewpoints.

[0082] In addition, the 3D model generation system 400 may include a plurality of cameras 301 or a single camera 301. For example, the 3D model generation system 400 may generate a plurality of frames (in other words, multi-viewpoint video data) with different viewpoints while moving the single camera 301. That is, each of the plurality of frames is a frame captured (generated) by the camera 301 with at least one of the position and posture of the camera 301 being different from each other.

[0083] The camera 301 may be a camera that generates a two-dimensional image or a camera that includes a three-dimensional measurement sensor that generates a three-dimensional model. In the first embodiment, each of the plurality of cameras 301 is a camera that generates a two-dimensional image.

[0084] One or more cameras 301 are connected to the estimation device 200 itself or to a hub (not shown) such as a communication device or a server via wires or wirelessly, so that each camera 301 can input the frames it captures to the estimation device 200 .

[0085] In addition, the frames captured by one or more cameras 301 may be output in real time, or may be temporarily recorded in an external storage device such as a memory or a cloud server and then output from the external storage device.

[0086] In addition, the one or more cameras 301 can be fixed cameras such as surveillance cameras, mobile cameras such as camcorders, smart phones or wearable cameras, or mobile cameras such as drones with shooting functions.

[0087] The estimation device 200 is a device that estimates the positions, postures, and camera parameters of the plurality of cameras 301 based on a plurality of frames acquired from one or more cameras 301 .

[0088] Here, the camera parameters are parameters representing the characteristics of the camera 301 , and are one or more parameters including internal parameters such as the focal length and image center of the camera 301 , and external parameters representing the position (more specifically, three-dimensional position) and posture of the camera 301 .

[0089] In addition, the estimation method of the estimation device 200 estimating the position and attitude of the camera 301 is not particularly limited. The estimation device 200 estimates the position and attitude of the plurality of cameras 301 using, for example, Visual-SLAM (Simultaneous Localization and Mapping) technology. Alternatively, the estimation device 200 estimates the position and attitude of each of the one or more cameras 301 using, for example, Structure-From-Motion technology.

[0090] In addition, the estimation device 200 calculates the distance between the camera 301 and the subject as a camera parameter based on the position and posture of the camera 301 estimated using the above-mentioned technology. In addition, the three-dimensional model generation system 400 may also include a distance measuring sensor to measure the distance between the camera 301 and the subject.

[0091] The estimation device 200 is connected to the 3D model generation device 100 itself or to a hub (not shown) such as a communication device or a server via wire or wireless, so as to be able to input a plurality of frames received from the camera 301 and estimation results into the 3D model generation device 100 .

[0092] The estimation device 200 inputs the plurality of frames received from the camera 301 and the estimated camera parameters thereof to the three-dimensional model generation device 100 .

[0093] Furthermore, the estimation result of the estimation device 200 may be output in real time, or the estimation result may be temporarily recorded in an external storage device such as a memory or a cloud server and then output from the estimation device.

[0094] In addition, the three-dimensional model generation device 100 may store the camera parameters in advance. In this case, the three-dimensional model generation system 400 may not include the estimation device 200. In addition, the plurality of cameras 301 may be connected to the three-dimensional model generation device 100 in a wireless or wired manner so as to be communicable therewith.

[0095] In addition, the multiple frames captured by the camera 301 may be directly input to the 3D model generation apparatus 100. In this case, the camera 301 is connected to the 3D model generation apparatus 100 itself, or to a hub (not shown) such as a communication device or a server, for example, by wire or wireless connection.

[0096] The estimation device 200 includes at least a computer system including, for example, a control program, a processing circuit such as a processor or a logic circuit for executing the control program, and a recording device such as an internal memory or an accessible external memory for storing the control program.

[0097] The three-dimensional model generation device 100 is a device that generates a model of a subject in a (virtual) three-dimensional space based on the positions, postures, and camera parameters of each of the plurality of cameras 301 and a plurality of frames, that is, performs three-dimensional model generation processing.

[0098] Here, the plurality of frames are multi-viewpoint images. Multi-viewpoint images are images obtained by moving the camera 301 while photographing a subject existing in the real space, or by photographing a subject existing in the real space from different viewpoints using a plurality of cameras 301. That is, the plurality of frames are images with different viewpoints. In other words, the estimation device 200 estimates the positions, postures (shooting directions), and camera parameters of the plurality of cameras 301 based on the plurality of frames with different viewpoints obtained from more than one camera 301.

[0099] In addition, the object generated by restoring its shape in a virtual three-dimensional space based on the frame of the real object photographed is called a three-dimensional model. The three-dimensional model of the object is a collection of points representing the three-dimensional positions of multiple points on the object photographed in multiple two-dimensional images from multiple viewpoints, that is, in which at least one of the position and posture of the camera 301 photographing the object is different.

[0100] The three-dimensional position is represented by three-value information consisting of an X component, a Y component, and a Z component of a three-dimensional space consisting of mutually orthogonal X-axis, Y-axis, and Z-axis. In addition, the information included in the multiple points representing the three-dimensional position includes not only the three-dimensional position (i.e., information representing the coordinates) but also information representing the color of each point, information representing the surface shape of each point and its surroundings, and the like.

[0101] The three-dimensional model generation device 100 at least includes a computer system, which includes, for example, a control program, a processing circuit such as a processor or a logic circuit that executes the control program, and a recording device such as an internal memory or an accessible external memory that stores the control program. In addition, the three-dimensional model generation device 100 may also be implemented, for example, by hardware installation of the processing circuit, or by executing a software program stored in the memory or distributed from an external server by the processing circuit, or by a combination of these hardware installations and software installations.

[0102] [Structure of 3D model generation device]

[0103] Next, refer to Figure 2 and Figure 3 , the detailed structure of the three-dimensional model generating device 100 is described.

[0104] The three-dimensional model generation device 100 is a device that generates a three-dimensional model based on a plurality of frames and includes a receiving unit 110 , a storage unit 120 , an acquiring unit 130 , a generating unit 140 , and an output unit 150 .

[0105] The receiving unit 110 receives from the estimation device 200 a plurality of images captured by the plurality of cameras 301, and the position, posture, and camera parameters of each of the plurality of cameras 301 estimated by the estimation device 200. The receiving unit 110 is, for example, a communication interface for communicating with the estimation device 200. When the three-dimensional model generation device 100 and the estimation device 200 perform wireless communication, the receiving unit 110 includes, for example, an antenna and a wireless communication circuit. Alternatively, when the three-dimensional model generation device 100 and the estimation device 200 perform wired communication, the receiving unit 110 includes, for example, a connector connected to a communication line.

[0106] The receiving unit 110 outputs the received frames captured by the plurality of cameras 301 and the positions, postures, and camera parameters of the one or more cameras 301 estimated by the estimation device 200 to the storage unit 120 .

[0107] The storage unit 120 is a memory that stores a plurality of frames captured by one or more cameras 301 received by the receiving unit 110, and the position, posture, and camera parameters of each of the one or more cameras 301 estimated by the estimation device 200. The storage unit 120 stores, for example, a control program executed by each processing unit included in the three-dimensional model generation device 100.

[0108] The storage unit 120 is implemented by, for example, a HDD (Hard Disk Drive), a flash memory, or the like.

[0109] The acquisition unit 130 acquires from the storage unit 120 a plurality of frames captured by one or more cameras 301 and stored in the storage unit 120 , as well as the position, posture, and camera parameters of each of the one or more cameras 301 estimated by the estimation device 200 , and outputs them to the processing unit of the generation unit 140 .

[0110] Furthermore, the receiving unit 110 may directly output the plurality of frames received from the one or more cameras 301 , and the position, posture, and camera parameters of each of the one or more cameras 301 estimated by the estimation device 200 to the generating unit 140 .

[0111] The generation unit 140 is a processing unit that generates a three-dimensional model of the subject based on the plurality of images captured by the plurality of cameras 301 input from the acquisition unit 130 and the position, posture, and camera parameters of each of the plurality of cameras 301 estimated by the estimation device 200 .

[0112] Figure 3 This is a block diagram showing a detailed configuration of the generation unit 140 included in the three-dimensional model generation device 100 according to the first embodiment.

[0113] The generating unit 140 includes a frame set generating unit 141 , a determining unit 142 , and a three-dimensional model generating unit 143 .

[0114] The frame set generation unit 141 is a processing unit that generates a frame set, which is composed of a plurality of frames selected from a plurality of frames (shooting frames) based on the position and posture of the camera 301 in each of the plurality of frames (shooting frames), that is, a plurality of frames included in the plurality of shooting frames. There is one data (position and posture data) representing the position and posture of the camera relative to one frame. That is, the frame set generation unit 141 generates a frame set composed of frames selected from a plurality of frames based on a plurality of position and posture data representing the position and posture of the camera 301 in each of the plurality of frames when shooting the plurality of frames. The frame set includes a plurality of first frames and a second frame. The first frame is a frame used by the first three-dimensional model generation unit 144 to generate a first three-dimensional model. The second frame is a frame used to generate a second three-dimensional model.

[0115] Furthermore, at least one second frame may be the same as at least one frame among the plurality of first frames. Alternatively, the first frame and the second frame may be different from each other.

[0116] For example, the frame set generation unit 141 generates a multi-viewpoint frame set (hereinafter also referred to as a frame set) by rearranging the multiple frames in a prescribed order based on the multiple frames captured by one or more cameras 301 input from the acquisition unit 130 and the position, posture and camera parameters of each of the one or more cameras 301 estimated by the estimation device 200.

[0117] The determination unit 142 is a processing unit that projects the first three-dimensional model onto at least one second frame and determines a plurality of unprojected pixels in at least one second frame where the first three-dimensional model is not projected. Specifically, the determination unit 142 projects the first three-dimensional model generated by the first three-dimensional model generation unit 144 described later onto at least one second frame, where the at least one second frame is a frame included in a plurality of frames other than the first frame, and determines the unprojected pixels, where the first three-dimensional model is not projected in at least one second frame where the first three-dimensional model is projected. More specifically, the determination unit 142 projects the already generated three-dimensional model (hereinafter also referred to as the existing three-dimensional model) onto the frame in the order of the frame set generated by the frame set generation unit 141. For example, the determination unit 142 determines the unprojected pixels of the frame where the three-dimensional model is projected, where the three-dimensional model is not projected, by projecting the existing three-dimensional model onto the frame. That is, the determination unit 142 determines the pixels of the difference between the existing three-dimensional model and the frame.

[0118] The three-dimensional model generation unit 143 is a processing unit that generates a three-dimensional model based on a plurality of frames. For example, the three-dimensional model generation unit 143 first generates a three-dimensional model (a first three-dimensional model), and regenerates a three-dimensional model (hereinafter also referred to as a second three-dimensional model or a new three-dimensional model) for pixels in the frame obtained by projecting the generated three-dimensional model by the determination unit 142 and to which the three-dimensional model is not projected. The three-dimensional model generation unit 143 outputs the new three-dimensional model to the storage unit 120, for example. The storage unit 120 stores the new three-dimensional model input from the three-dimensional model generation unit 143.

[0119] The three-dimensional model generating unit 143 includes a first three-dimensional model generating unit 144 and a second three-dimensional model generating unit 145 .

[0120] The first three-dimensional model generation unit 144 is a processing unit that generates a first three-dimensional model of a predetermined area based on a plurality of first frames. Specifically, the first three-dimensional model generation unit 144 generates a first three-dimensional model of the subject based on a plurality of frames (first frames) among the plurality of frames.

[0121] The second three-dimensional model generating unit 145 is a processing unit that generates a second three-dimensional model based on the unprojected pixels identified by the identifying unit 142 .

[0122] For example, the determination unit 142 further projects the existing three-dimensional model onto a frame in the frame set to which the three-dimensional model has not been projected, and determines the unprojected pixels in the frame to which the three-dimensional model has not been projected. Here, in the case where the second three-dimensional model generation unit 145 has not generated the second three-dimensional model, the existing three-dimensional model is the first three-dimensional model. In addition, in the case where the second three-dimensional model generation unit 145 generates the second three-dimensional model, the existing three-dimensional model includes the first three-dimensional model and the second three-dimensional model generated by the second three-dimensional model generation unit 145. The second three-dimensional model generation unit 145 further generates a new three-dimensional model for the unprojected pixels in the frame to which the determination unit 142 has projected the existing three-dimensional model, to which the existing three-dimensional model has not been projected.

[0123] The determination unit 142 and the 3D model generation unit 143 repeatedly project the 3D model onto the frame in the order of the frame set, thereby determining unprojected pixels in the frame where the 3D model is not projected, and generating a 3D model for the determined unprojected pixels.

[0124] In addition, the three-dimensional model first projected to the frame is generated, for example, based on the frame set by the three-dimensional model generation unit 143. For example, the three-dimensional model generation unit 143 first generates a three-dimensional model using the first and second frames in the frame set, and projects the generated three-dimensional model to the third frame, thereby determining unprojected pixels on which the three-dimensional model is not projected among the pixels of the third frame.

[0125] For example, the frame set generation unit 141 arranges a plurality of frames in a frame set in a frame order. For example, the frame set generation unit 141 generates a frame set in which a plurality of frames are arranged in a frame order based on a prescribed condition. In this case, for example, the determination unit 142 projects an already generated three-dimensional model including a first three-dimensional model, that is, an existing three-dimensional model (third three-dimensional model) onto at least one second frame, and determines unprojected pixels to which the existing three-dimensional model is not projected in at least one second frame to which the existing three-dimensional model is projected. The determination unit 142 and the second three-dimensional model generation unit 145 repeatedly perform the above-mentioned processing in the order of frames. The existing three-dimensional model includes the first three-dimensional model and the generated second three-dimensional model.

[0126] In addition, the prescribed conditions may be set arbitrarily in advance. For example, the frame set generation unit 141 calculates the positive direction of the reference axis based on the optical axis direction of each camera 301 when shooting a plurality of frames. That is, the frame set generation unit 141 determines the frame order based on the calculated positive direction of the reference axis and generates a frame set of the determined frame order.

[0127] The optical axis direction is, for example, a direction orthogonal to the light incident surface of the sensor provided by the camera 301. Alternatively, the optical axis direction is, for example, the optical axis direction of the lens provided by the camera 301. The positive direction of the optical axis is, for example, the normal direction of the light incident surface of the sensor provided by the camera 301. The negative direction of the optical axis is, for example, the direction in which light is incident on the camera 301.

[0128] For one frame, there is one data indicating the optical axis direction of the camera (optical axis data). That is, the frame set generation unit 141 generates a frame set consisting of frames selected from multiple frames based on multiple optical axis data indicating the optical axis direction of the camera 301 for each frame when the multiple frames are captured.

[0129] For example, the frame set generation unit 141 selects a representative frame from a plurality of frames, the representative frame being one of the plurality of frames, and calculates the optical axis direction of the camera 301 that captured the selected representative frame as the positive direction of the reference axis. In addition, the representative frame may be one or more than two. When there are more than two representative frames, the frame set generation unit 141 may set the average value in the optical axis direction of the camera 301 that captured each of the two or more selected representative frames as the reference axis direction.

[0130] In addition, for example, the frame set generation unit 141 generates a frame set in which the frames are arranged in the order in which they are photographed at a position close to the subject in the reference axis direction. In other words, the frame set generation unit 141 generates a frame set in which the frames are arranged in the order from near to far from the prescribed area according to the position of the camera 301 of each frame when the plurality of frames in the frame set are photographed. Specifically, in the frame set generation unit 141, a frame set is generated in which, when the direction of the prescribed area viewed from the camera 301, that is, the direction from the camera 301 toward the prescribed area is set as the positive direction of the reference axis, the frame set arranges the plurality of frames in the order from near to far from the prescribed area in the positive direction of the reference axis according to the position of the camera 301 of each frame when the plurality of frames in the frame set are photographed.

[0131] In addition, the frame set generation unit 141 may generate a frame set using all frames included in the plurality of frames stored in the storage unit 120 , or may generate a frame set using a part of the plurality of frames.

[0132] For example, the frame set generation unit 141 determines whether each of the plurality of frames satisfies the first condition, and generates a frame set using the frames satisfying the first condition among the plurality of frames.

[0133] In addition, the prescribed first condition can be arbitrarily set in advance. For example, the frame set generation unit 141 determines the case where the angle formed by the positive direction of the reference axis and the optical axis direction of the camera 301 when shooting multiple frames included in the frame set is less than the prescribed first angle as the prescribed first condition. That is, when the angle formed by the reference axis direction and the optical axis direction of the camera 301 is less than the first angle, the frame set generation unit 141 includes the frames shot by the camera 301 in the frame set. On the other hand, when the angle formed by the reference axis direction and the optical axis direction of the camera 301 is greater than the first angle, the frame set generation unit 141 does not include the frames shot by the camera 301 in the frame set.

[0134] Refer again Figure 2 The output unit 150 is a processing unit that outputs the three-dimensional model stored in the storage unit 120. The output unit 150, for example, merges the existing three-dimensional model and the new three-dimensional model stored in the storage unit 120, generates a three-dimensional model, and outputs it. The output unit 150, for example, includes an antenna, a wireless communication circuit, a connector, etc. for connecting to a display device such as a display (not shown) in a wired or wireless manner. The output unit 150 outputs the merged three-dimensional model to the display device, so that the display device displays the three-dimensional model.

[0135] [Processing steps of the three-dimensional model generation device]

[0136] Next, refer to Figures 4 to 18 , describing in detail the processing steps of three-dimensional model generation performed by the three-dimensional model generation device 100.

[0137] <Summary>

[0138] Figure 4 This is a flowchart showing the processing executed by the three-dimensional model generating device 100 according to the first embodiment.

[0139] First, the receiving unit 110 receives information indicating the position and posture of the camera 301 , camera parameters, and a plurality of frames (captured frames) estimated by the estimation device 200 (step S101 ). The receiving unit 110 outputs the received data to the storage unit 120 .

[0140] Next, the storage unit 120 stores the information indicating the position and posture of the camera 301 , the camera parameters, and the plurality of frames (photographed frames) received by the receiving unit 110 in step S101 (step S102 ).

[0141] Next, the acquisition unit 130 inputs the information indicating the position and posture of the camera 301 , the camera parameters, and a plurality of frames (captured frames) stored in the storage unit 120 to the generation unit 140 (step S103 ).

[0142] Next, the frame set generation unit 141 generates a frame set including a plurality of frames, wherein the frame set determines the frame order based on a plurality of data indicating the position and posture of the camera 301 when each of the plurality of frames (captured frames), camera parameters, and the plurality of frames (captured frames) (step S104).

[0143] Next, the first three-dimensional model generation unit 144 generates a three-dimensional model (first three-dimensional model) based on the frame set (step S105 ). For example, the first three-dimensional model generation unit 144 generates the first three-dimensional model using any two frames (first frames) in the frame set.

[0144] Next, the determination unit 142 projects the first three-dimensional model generated in step S105 onto a frame (second frame) (step S106 ). Specifically, the determination unit 142 projects the first three-dimensional model onto at least one second frame included in the frame set excluding the first frame.

[0145] Next, the second three-dimensional model generation unit 145 generates a three-dimensional model (second three-dimensional model) based on the unprojected pixels in the second frame to which the first three-dimensional model was not projected in step S106 (step S107 ).

[0146] Next, the output unit 150 combines and outputs the first three-dimensional model and the second three-dimensional model (step S108 ).

[0147] <Frame Order Determination Process>

[0148] Next, we will refer to Figure 5A to Figure 8B Description Figure 4The frame set generation process (more specifically, the frame order determination process) is performed by the frame set generation unit 141 in step S104.

[0149] The frame set generation unit 141 may generate a frame set using all frames included in the storage unit 120 , or may generate a frame set by selecting at least three frames from among all frames.

[0150] Furthermore, the frame set may be a frame sequence in which the position and attitude of the camera 301 move in the negative direction of the optical axis direction of the camera 301 (in other words, the shooting direction).

[0151] Furthermore, the generation section 140 (more specifically, the frame set generation section 141 ) may generate a frame set by extracting frames from the plurality of frames stored in the storage section 120 .

[0152] For example, when the acquisition unit 130 acquires multiple frames from the storage unit 120 in a random frame order, the frame set generation unit 141 generates the following frame set, wherein the frame order of the frame set is rearranged in the order from small to large according to the position and posture of the camera 301 that shot the multiple frames in the negative direction of the camera optical axis direction, that is, in the order from near to far from the subject according to the position of the camera 301 when shooting the frames.

[0153] Furthermore, the frame set generation unit 141 may add identification information such as an index number indicating the frame order to each frame of the frame set individually, or may add the identification information to the frame set collectively.

[0154] In addition, the t frame, t+1 frame, t+2 frame, t+3 frame, t+4 frame, and t+5 frame to be described below are frames captured by the camera 301 in a time series in this order. Figure 5A to Figure 8B In the following, the case where there is an object as a subject in the direction of the camera optical axis in each figure will be described. Figure 5A to Figure 8B In the figure, the direction of the camera optical axis is consistent with the direction of the reference axis.

[0155] <Example 1>

[0156] Figure 5A This is a diagram for explaining a first example of the photographing procedure of the camera 301 .

[0157] Figure 5B This is a diagram for explaining a first example of the frame order determined by the three-dimensional model generation device 100 with respect to the imaging order of the camera 301 .

[0158] The first example is an example in which the camera 301 photographs the subject while being away from the subject. In this case, the generation unit 140 (more specifically, the frame set generation unit 141) generates a frame set, and the frame set is as follows: Figure 5BAs in the processing order of the images, the order of frames is set to be consistent with the camera movement direction, that is, the order in which the subjects are photographed.

[0159] In this way, when the camera 301 captures multiple frames with the camera 301 being far away from the subject, the frame set generation unit 141 can generate frame sets in the order of capturing the frames, thereby generating frame sets in which the frames are arranged in the order from near to far from the subject.

[0160] <Example 2>

[0161] Fig. 6A This is a diagram for explaining a second example of the photographing sequence of the camera 301 .

[0162] Figure 6B This is a diagram for explaining a second example of the frame order determined by the three-dimensional model generation device 100 with respect to the imaging order of the camera 301 .

[0163] The second example is an example in which the camera 301 photographs the subject while approaching the subject. In this case, the frame set generation unit 141 generates a frame set, and the frame set is as follows: Figure 6B As shown in the processing order of , the order of frames is set to be the opposite order to the camera movement direction, that is, the opposite order to the order in which the subjects are photographed.

[0164] In this way, when the camera 301 captures multiple frames in which the camera 301 is close to the subject, the frame set generation unit 141 generates a frame set rearranged in the order opposite to the shooting order, thereby being able to generate a frame set in which the frames are arranged in the order of the frames generated by the camera 301 from near to far from the subject.

[0165] As described above in the first and second examples, when the position change of the camera 301 when capturing a frame is constant relative to the subject and time information indicating the time when the frame is captured is attached to the frame, the frame set generation unit 141 can generate a frame set that rearranges the frames based on the time information.

[0166] <Example 3>

[0167] Fig. 7A This is a diagram for explaining the third example of the imaging sequence of the camera 301.

[0168] Figure 7B This is a diagram for explaining a third example of the frame order determined by the three-dimensional model generation device 100 with respect to the imaging order of the camera 301 .

[0169] The third example is an example in which the camera 301 captures the subject while moving without changing the distance to the subject. In this case, the frame set generation unit 141 can generate a frame set by rearranging the order of frames in an arbitrary order. Figure 7B A process of generating a frame set in which the order of frames is set to coincide with the camera movement direction, that is, the order in which the subjects are photographed is shown.

[0170] In this way, when capturing multi-viewpoint images with the camera 301 positioned parallel to the subject, that is, with the distance between the subject and the camera 301 unchanged, the frame set generation unit 141 generates a frame set from a plurality of frames regardless of the capturing order.

[0171] <Example 4>

[0172] Fig. 8A This is a diagram for explaining the fourth example of the photographing sequence of the camera 301.

[0173] Figure 8B This is a diagram for explaining a fourth example of the frame order determined by the three-dimensional model generation device 100 with respect to the imaging order with the camera 301 .

[0174] The fourth example is an example in which the camera 301 moves and photographs the subject while randomly changing the distance to the subject. Figure 8B In the processing sequence shown in FIG. 1 , the frame set generation unit 141 generates a frame set by rearranging the frame order in the order from the frame captured at a position close to the subject, in other words, in the order from the frame located on the positive side in the camera optical axis direction. Figure 8B In the illustrated example, the frame set generation unit 141 generates a frame set in the order of t frame, t+3 frame, t+1 frame, t+4 frame, and t+2 frame.

[0175] By performing such processing, the frame set generation unit 141 can generate a frame set in which the order of frames is rearranged in the order from near to far of the position of the camera 301 from the subject, regardless of the order in which the frames are captured.

[0176] <Frame Determination Processing>

[0177] Next, we will refer to Fig. 9 The following describes the procedure of the frame identification process performed by the frame set generation unit 141 .

[0178] As described above, the frame set generation unit 141 generates a frame set in which a plurality of frames are rearranged to form a sequence of frames captured by the camera 301 at positions from near to far from the subject, that is, a sequence of frames in which the position of the camera 301 at each of the plurality of frames is from near to far from a specified area.

[0179] Here, for example, when generating a three-dimensional model, the three-dimensional model generation unit 143 may generate the three-dimensional model without using all frames of the plurality of captured frames (in other words, all captured frames captured by the camera 301) stored in the storage unit 120. That is, the generation unit 140 (more specifically, the frame set generation unit 141) may generate a frame set by extracting a part of the frames from the plurality of frames (captured frames). In other words, the frame set generation unit 141 may determine (in other words, select) a part of the frames from all the frames captured by the camera 301 and generate a frame set.

[0180] Fig. 9 This is a flowchart showing an example of frame set generation processing executed by the three-dimensional model generation device 100 according to the first embodiment.

[0181] First, the frame set generation unit 141 calculates the reference axis direction (more specifically, the positive direction of the reference axis) from a plurality of frames (step S201 ).

[0182] In step S201, for example, the frame set generation unit 141 calculates the optical axis direction of the camera 301 in each of the frames when each frame is captured based on the camera parameters, and calculates the reference axis direction based on the calculated optical axis directions. For example, the frame set generation unit 141 calculates the average position of each position of the camera 301 when each frame is captured. Furthermore, the frame set generation unit 141 calculates, for example, the average value (average angle) of the angles formed by the optical axis direction of the camera 301 relative to a predetermined axis arbitrarily determined in advance when each frame is captured. For example, the frame set generation unit 141 calculates the direction of an axis that passes through the calculated average position and is inclined relative to the predetermined axis by the calculated average angle as the reference axis direction.

[0183] Alternatively, the frame set generation unit 141 calculates the optical axis direction of the camera 301 for each frame, and calculates the reference axis direction using the least square method for the plurality of optical axis directions. In this case, the frame set generation unit 141 may remove the optical axis direction having an error greater than a predetermined threshold.

[0184] Alternatively, in step S201 , for example, the frame set generation unit 141 determines a representative frame to be a representative from a plurality of frames, and sets the optical axis direction of the camera 301 that captured the determined representative frame as the reference axis direction.

[0185] The representative frame may be determined by any method. For example, the frame set generation unit 141 sets the frame captured by the camera 301 closest to the subject among the cameras 301 that captured multiple frames as the representative frame. Alternatively, the frame set generation unit 141 sets the frame captured by the camera 301 that is located at an intermediate distance from the subject among the cameras 301 that captured multiple frames as the representative frame.

[0186] Next, the frame set generation unit 141 starts a repetitive process of selecting a frame (step S202 ). Specifically, the frame set generation unit 141 repeatedly executes the process of steps S203 to S205 for each of the plurality of frames stored in the storage unit 120 .

[0187] The frame set generation unit 141 determines whether the angle formed by the reference axis (more specifically, the positive direction of the reference axis) calculated in step S201 and the optical axis direction of the camera 301 that captured the frame is equal to or smaller than the first angle (step S203 ).

[0188] When the frame set generation unit 141 determines that the angle formed by the reference axis calculated in step S201 and the optical axis direction of the camera 301 that captured the frame is less than the first angle (No in step S203), the frame is set as non-selected (step S205). That is, when it is No in step S203, the frame set generation unit 141 does not select the frame and does not add the frame to the frame set to be generated. In addition, setting it as non-selected means that the frame set generation unit 141 does not use the frame in the processing performed in step S207 later, for example, it is not necessary to delete the frame from the storage unit 120.

[0189] On the other hand, when the frame set generation unit 141 determines that the angle formed by the reference axis calculated in step S201 and the optical axis direction of the camera 301 that captured the frame is not less than the first angle (Yes in step S203), the frame is selected (step S204). Here, selection means that the frame set generation unit 141 uses the corresponding frame for processing performed in step S207 thereafter.

[0190] The frame set generation unit 141 performs the repetitive processing of steps S203 to S205 for each of the plurality of frames, that is, after performing the repetitive processing of selecting or non-selecting a frame, the repetitive processing ends (step S206 ).

[0191] Next, the frame set generation unit 141 rearranges the selected frames in the order of frames captured at positions close to the subject along the reference axis calculated in step S201, that is, in the order of frames in which the position of the camera 301 of each frame when capturing the plurality of frames is from near to far from the prescribed area, and generates a frame set (step S207). In step S207, for example, the frame set generation unit 141 associates the rearranged plurality of frames by assigning consecutive numbers in the order of the frames.

[0192] The threshold value of the first angle may be stored in advance in the storage unit 120, for example. The first angle may be set arbitrarily, and its value is not particularly limited. The frame set generation unit 141 may select a representative frame based on, for example, the position of the camera 301, rather than the angle with the reference axis.

[0193] In addition, the frame set generation unit 141 may store the generated frame set in the storage unit 120 , or may later output the generated frame set to the 3D model generation unit 143 which generates a new 3D model based on the frame set.

[0194] In addition, in step S203, it is described that the angle is less than the first angle, but this is not strictly necessary. For example, when it is described as "less than the first angle", it may also mean that it is smaller than the first angle.

[0195] <3D Model Generation Process>

[0196] Next, refer to Figure 10 to Figure 13 , the three-dimensional model generation process performed by the three-dimensional model generation device 100 is described.

[0197] Fig.10 This is a flowchart showing the three-dimensional model generation process executed by the three-dimensional model generation device 100 according to the first embodiment.

[0198] First, the first three-dimensional model generating unit 144 generates a three-dimensional model (first three-dimensional model) using a plurality of frames (first frames) in a frame set (step S301 ).

[0199] Next, the generation unit 140 starts repetitive processing for the number of frames included in the frame set (step S302 ). Specifically, the generation unit 140 repeatedly executes the processing of steps S302 to S306 for each of the plurality of frames included in the frame set except the frame used in step S301 .

[0200] Next, the determination unit 142 projects the already generated three-dimensional model (the existing three-dimensional model, i.e., the third three-dimensional model) onto at least one frame (the second frame) in the frame set (step S303). In step S303, the determination unit 142 projects the existing three-dimensional model onto the frames of the frame set according to the frame order determined by the frame set generation unit 141. The existing three-dimensional model is initially only the first three-dimensional model generated in step S301, but by repeating the processing of steps S303 to S305, the new three-dimensional model (the second three-dimensional model) generated in step S305 described later is repeatedly added to the already generated three-dimensional model.

[0201] Next, the identification unit 142 identifies unprojected pixels on which the three-dimensional model (specifically, three-dimensional points constituting the three-dimensional model) is not projected in the second frame on which the three-dimensional model is projected (step S304 ).

[0202] Next, the second 3D model generation unit 145 generates a new 3D model (second 3D model) based on the unprojected pixels determined in step S304 (step S305). That is, the second 3D model generation unit 145 generates a second 3D model composed of points that do not exist in the existing 3D model.

[0203] Next, the generator 140 ends the repetitive processing below the number of frames of the frame set (step S306). That is, regarding the processing after the second round in the loop processing of steps S303 to S305, in step S303, the second three-dimensional model generated before step S305 is included in the existing three-dimensional model.

[0204] Fig.11 This is a diagram for explaining the processing position of the generation process executed by the generation unit 140 included in the three-dimensional model generation device 100 according to the first embodiment.

[0205] Fig.12 It is a diagram for explaining changes in the shooting position of the camera 301.

[0206] Fig.13 It is a diagram for explaining changes in the imaging range of the camera 301.

[0207] also, Fig.11 The frames shown in (a) to 11(c) are respectively represented by Fig.12 and Fig.13 The frames captured by the camera 301 are shown. Fig.12 and Fig.13 is a diagram showing the same situation. Fig.12 It is a stereogram. Fig.13 is a top view. Fig.13 , the three-dimensional model generation range (ie, the imaging range) at time t and time t+1 is indicated by being shaded.

[0208] like Fig.11 As shown in (a), for example, it is assumed that no existing three-dimensional model is projected on any part of the frame at time t. In this case, the second three-dimensional model generation unit 145 generates a second three-dimensional model for all pixels of the frame at time t.

[0209] In addition, if Fig.11 As shown in (b), for example, assume that for the frame at time t+1, a 3D model is projected on a portion of the portion indicated by the shadow. Specifically, assume Fig.11 The shaded part in (b) is projected on Fig.11 In this case, the second three-dimensional model generation unit 145 generates a pixel position of the second three-dimensional model on the frame at time t shown in (a). Fig.11 The second three-dimensional model is generated using pixels of the unshaded portion shown in (b).

[0210] In addition, if Fig.11 As shown in (c) of FIG. 1 , for example, assume that for the frame at time t+2, an existing three-dimensional model is projected on a portion of the portion indicated by the shadow. Specifically, assume Fig.11 The shaded part in (c) is projected on Fig.11 The second three-dimensional model generated on the frame at time t shown in (a) and Fig.11 In this case, the second three-dimensional model generation unit 145 generates a pixel position of the second three-dimensional model on the frame at time t+1 shown in (b). Fig.11 The second three-dimensional model is generated by using pixels at positions not shaded (ie, non-projected pixels) shown in (c).

[0211] In this way, the second three-dimensional model generation unit 145 generates the second three-dimensional model in the order of frames in the frame set generated by the frame set generation unit 141 .

[0212] Next, refer to Fig.14 A method for generating a three-dimensional model is described.

[0213] When generating the first three-dimensional model, the three-dimensional model generating unit 143 generates existing three-dimensional points using, for example, two frames.

[0214] Fig.14 This is a diagram for explaining a process procedure of generating a three-dimensional model by the three-dimensional model generating unit 143 .

[0215] like Fig.14 As shown, for example, the first three-dimensional model generating unit 144 generates a first three-dimensional model of the object 500 using two frames, namely, a reference frame 501 and a visible frame 502 .

[0216] The reference frame 501 and the visible frame 502 only need to include the object 500 and the common part of the object 500 (for example, Fig.14 The feature point 503 shown in the figure is sufficient and is not particularly limited.

[0217] The first three-dimensional model generation unit 144 generates a first three-dimensional model by using, for example, the first frame of the frame set as the reference frame 501 and the second frame of the frame set as the visible frame 502. For example, as shown in the following formula (1), the first three-dimensional model generation unit 144 calculates the Normalized Cross Correlation (NCC) between small areas as N(I, J) by using a combination of the reference frame 501 and the visible frame 502, and generates matching information indicating the result of matching between frames. In addition, the first three-dimensional model generation unit 144 generates a three-dimensional model by performing triangulation based on information indicating the position and posture of the camera 301 in each of the reference frame 501 and the visible frame 502 when the reference frame 501 and the visible frame 502 were respectively photographed, and matching information between frames.

[0218]

Mathematical formula 1

[0219]

[0220] In addition, I xy and J xy represents the pixel values ​​in the small area of ​​frame I and frame J. In addition,

[0221]

Mathematical formula 2

[0222]

[0223] as well as

[0224]

Mathematical formula 3

[0225]

[0226] are the average values ​​of the pixel values ​​in the small areas of frame I and frame J respectively.

[0227] In addition, when the second three-dimensional model generating unit 145 generates the second three-dimensional model, the same steps are performed as when the first three-dimensional model generating unit 144 generates the first three-dimensional model. Specifically, for example, when the first three-dimensional model generating unit 144 generates the first three-dimensional model by using the first frame in the frame set as the reference frame 501 and the second frame in the frame set as the visible frame 502, the second three-dimensional model generating unit 145 projects the first three-dimensional model onto the visible frame by using the third frame in the frame set as the new visible frame.

[0228] For example, as shown in the above formula (1), the second three-dimensional model generation unit 145 calculates the NCC between the small areas as N(I, J) by combining the reference frame 501 (for example, the first frame of the frame set) and the pixels in the newly used visible frame that are not projected with the first three-dimensional model, and generates matching information indicating the matching result obtained by matching between the frames. In addition, the three-dimensional model generation unit 143 generates the second three-dimensional model by performing triangulation based on the information indicating the position and posture of the camera 301 when each of the reference frame 501 and the newly used visible frame was photographed and the matching information between the frames.

[0229] [Effects, etc.]

[0230] As described above, the three-dimensional model generation method of implementation mode 1 includes: a first three-dimensional model generation step (step S105), generating a first three-dimensional model of a specified area based on multiple first frames; a determination step (step S106), projecting the first three-dimensional model to at least one second frame, and determining multiple unprojected pixels in at least one second frame to which the first three-dimensional model is not projected; and a second three-dimensional model generation step (step S107), generating a second three-dimensional model based on multiple unprojected pixels.

[0231] Thus, since the differential three-dimensional models of the generated three-dimensional models are generated sequentially, the models of the overlapping parts in each three-dimensional model are not generated. Therefore, the processing time for generating the three-dimensional model can be shortened. In addition, according to the three-dimensional model generation method of embodiment 1, the first three-dimensional model is projected onto at least one second frame. Therefore, when shooting at least one second frame, even if a part of the subject is set outside the shooting area, the three-dimensional model representing the part of the subject will not be lost. For example, if a subject that is difficult to keep still is shot only once and multiple first frames are generated, a three-dimensional model of the subject can be generated, and based on at least one second frame, a three-dimensional model of subjects other than the subject can be generated.

[0232] In addition, for example, the three-dimensional model generation method of embodiment 1 also includes a frame set generation step (step S104), which generates a frame set consisting of multiple frames contained in multiple shooting frames based on the position and posture of the camera 301 in each of the multiple shooting frames, and the frame set includes multiple first frames and one or more second frames.

[0233] Thus, since the three-dimensional model is generated using the frames of the frame set consisting of the frames selected from the plurality of frames based on the position and posture of the camera 301, it is possible to generate a frame set that excludes unnecessary frames such as a predetermined area that is not photographed. Therefore, since the three-dimensional model generation process is not performed for unnecessary frames, the processing time can be shortened.

[0234] In addition, for example, in the frame set generating step, a plurality of frames are arranged in a prescribed frame set in frame order, and in the determining step, an already generated three-dimensional model including the first three-dimensional model, i.e., a third three-dimensional model, is projected onto at least one second frame, and a plurality of unprojected pixels of the third three-dimensional model that are not projected in at least one second frame on which the third three-dimensional model is projected are determined, and the above-mentioned determining step and the above-mentioned second three-dimensional model generating step are repeatedly performed in frame order.

[0235] Thus, the projection and the generation of the second three-dimensional model are repeated for each frame. Therefore, the existing three-dimensional model including the first three-dimensional model and the generated second three-dimensional model is projected onto the frame, so that pixels that are not projected can be extracted. Thus, the processing time can be further shortened.

[0236] Furthermore, for example, in the frame set generation step, a frame set is generated in which a plurality of frames are arranged in order from near to far from the prescribed area when the position of the camera 301 for each frame when the plurality of frames in the frame set were captured.

[0237] Thus, the second three-dimensional model can be generated by sequentially using the frames starting from the frames captured by the camera 301 close to the specified area. The closer the frame captured by the camera 301 is to the subject, the higher the accuracy, that is, the higher the image quality. Therefore, the processing time for generating the three-dimensional model can be shortened, and the accuracy of the three-dimensional model can be improved.

[0238] In addition, for example, in the frame set generation step, when the direction from the camera 301 toward the specified area is set as the positive direction of the reference axis, a frame set is generated in which the frames are arranged in order from near to far from the specified area in the positive direction of the reference axis according to the position of the camera 301 of each frame when the multiple frames in the frame set are captured.

[0239] Thus, based on the reference axis direction, a plurality of frames can be arranged in the order of frames captured by the camera 301 close to the predetermined area.

[0240] Furthermore, for example, in the frame set generation step, the positive direction of the reference axis is calculated based on the optical axis direction of the camera 301 for each frame when a plurality of frames are captured.

[0241] Thus, for example, since the optical axis direction of the camera 301 becomes the reference axis direction, the frame set can be made to exclude unnecessary frames that hardly include the specified area through simple processing. Therefore, the processing time for unnecessary frames when generating a three-dimensional model can be shortened.

[0242] Furthermore, for example, in the frame set generation step, a representative frame is selected from a plurality of frames, and the optical axis direction of the camera 301 that has photographed the representative frame is calculated as the positive direction of the reference axis.

[0243] Therefore, compared with the case where the reference axis direction is calculated based on the optical axis direction of the camera 301 that has taken multiple frames, the reference axis direction is calculated based on the optical axis direction of the camera 301 that has taken representative frames. Therefore, by only selecting an appropriate number of representative frames, the processing time required to calculate the reference axis direction can be shortened.

[0244] For example, in the frame set generation step, it is determined whether each of the plurality of frames satisfies the first condition, and a frame set is generated using frames satisfying the first condition among the plurality of frames.

[0245] Thus, by appropriately setting the first condition, unnecessary frames such as frames on which all the first three-dimensional models are projected or frames that hardly include a predetermined area can be excluded from the frame set. Therefore, the processing time for unnecessary frames when generating a three-dimensional model can be shortened.

[0246] Furthermore, for example, in the frame set generation step, it is determined as a first condition that the angle formed by the positive direction of the reference axis and the optical axis direction of the camera 301 when the plurality of frames included in the frame set are captured is equal to or smaller than a first angle.

[0247] This makes it possible to eliminate unnecessary frames that hardly include the predetermined area from the frame set by simple processing, thereby shortening the processing time for unnecessary frames when generating a three-dimensional model.

[0248] In addition, for example, the predetermined area is an area including at least one of a stationary object in stationary state and a moving object in motion as a subject.

[0249] Thus, since the object is included in the predetermined area, when measuring the distance between the predetermined area and the camera 301, for example, the distance can be measured with reference to the object, so the distance can be calculated with high accuracy.

[0250] Furthermore, for example, at least one second frame is identical to at least one frame among the plurality of first frames.

[0251] Thus, for example, when determining the second frame from a plurality of frames, it is not necessary to perform a process of excluding the first frame, and thus the processing time is shortened.

[0252] In addition, for example, the first frame and the second frame are different from each other.

[0253] Therefore, since the second three-dimensional model is not projected onto the first frame when the second three-dimensional model is generated, the processing time for generating the second three-dimensional model can be shortened.

[0254] In addition, the three-dimensional model generation device 100 of embodiment 1 includes: a first three-dimensional model generation unit 144, which generates a first three-dimensional model of a specified area based on multiple first frames; a determination unit 142, which projects the first three-dimensional model to at least one second frame and determines unprojected pixels to which the first three-dimensional model is not projected; and a second three-dimensional model generation unit 145, which generates a second three-dimensional model based on the unprojected pixels.

[0255] Thus, the 3D model generation device 100 sequentially generates differential 3D models of the generated 3D models, and therefore does not generate models of overlapping portions of the 3D models. Therefore, the 3D model generation device 100 can shorten the processing time for generating the 3D model.

[0256] (Implementation Method 2)

[0257] Next, a three-dimensional model generation device according to Embodiment 2 will be described. In the description of Embodiment 2, substantially the same components as those of Embodiment 1 are denoted by the same reference numerals, and the description may be simplified or partially omitted.

[0258] [structure]

[0259] First, refer to Fig.15 , the structure of the three-dimensional model generation device of embodiment 2 is explained.

[0260] Fig.15 This is a block diagram showing a characteristic structure of a three-dimensional model generating device 100a according to the second embodiment. Fig.16 This is a block diagram showing a characteristic structure of the generation unit 140a included in the three-dimensional model generation device 100a according to the second embodiment.

[0261] The three-dimensional model generation system 400 a according to the second embodiment includes a camera group 300 , an estimation device 200 , and a three-dimensional model generation device 100 a .

[0262] The three-dimensional model generating device 100 a includes a receiving unit 110 , a storage unit 120 , an acquiring unit 130 , a generating unit 140 a , and an output unit 150 .

[0263] The generating unit 140 a includes a frame set generating unit 141 a , a determining unit 142 a , a three-dimensional model generating unit 143 a , a determining unit 146 , and an updating unit 147 .

[0264] The frame set generation unit 141a is a processing unit that determines the frame order of a plurality of frames and generates a frame set. Here, the frame set generation unit 141a generates a frame set that maintains the frame order stored in the storage unit 120, that is, the shooting order of the camera 301.

[0265] The determination unit 142a is a processing unit that projects an image onto an existing three-dimensional model in the order of the frame set generated by the frame set generation unit 141a. The determination unit 142a projects the first three-dimensional model onto at least one second frame, and determines a plurality of projection pixels of the first three-dimensional model projected in at least one second frame. Specifically, the determination unit 142a projects the first three-dimensional model onto at least one second frame, the at least one second frame being a frame included in a plurality of frames (more specifically, a frame set) other than the first frame, and determines pixels of one or more second frames onto which the first three-dimensional model is projected, i.e., projection pixels, and pixels of the first three-dimensional model projected onto one or more second frames. That is, the determination unit 142a determines the projection pixels onto which the first three-dimensional model is projected for the frame onto which the first three-dimensional model is projected. In addition, the determination unit 142a determines the pixels (i.e., three-dimensional points) of the first three-dimensional model projected onto the frame.

[0266] Furthermore, the identification unit 142 a identifies a plurality of non-projected pixels onto which the first three-dimensional model is not projected in at least one second frame.

[0267] The three-dimensional model generation unit 143a is a processing unit that generates a three-dimensional model from a plurality of frames. The three-dimensional model generation unit 143a includes a first three-dimensional model generation unit 144a and a second three-dimensional model generation unit 145a.

[0268] The first 3D model generation unit 144a is a processing unit that generates a first 3D model of a predetermined area based on a plurality of first frames. Specifically, the first 3D model generation unit 144a generates a first 3D model of the subject based on at least two first frames among the plurality of frames.

[0269] The second three-dimensional model generating unit 145 a is a processing unit that generates a second three-dimensional model based on the plurality of unprojected pixels determined by the determining unit 142 a .

[0270] The determination unit 146 is a processing unit that determines whether at least one second shooting position of at least one second frame is closer to a specified area than any one of the multiple first shooting positions of multiple first frames. Specifically, the determination unit 146 determines whether the position of the camera 301 that shoots at least one second frame containing the projection pixels determined by the determination unit 142a is closer to the subject than the position of the camera 301 when the frame used to generate the pixels of the first three-dimensional model is shot, and the pixels of the first three-dimensional model are pixels determined by the determination unit 142a. That is, for the pixels projected onto the existing three-dimensional model in the frame projected by the determination unit 142a, the determination unit 146 determines which of the distances between the camera 301 and the subject is closer, that is, the frame that shoots the pixels of the existing three-dimensional model corresponding to the projected pixels and the frame that projects the existing three-dimensional model.

[0271] When the determination unit 146 determines that at least one second shooting position is closer to the specified area than any one of the plurality of first shooting positions, the updating unit 147 updates the first three-dimensional model based on the plurality of projection pixels determined by the determination unit 142a. More specifically, the updating unit 147 updates the first three-dimensional model so that, for pixels in each of the plurality of frames where the positions of the three-dimensional points of the three-dimensional model overlap, the pixels of the frame shot by the camera 301 that is closest to the subject determined by the determination unit 146 (i.e., the pixel values ​​before the first three-dimensional model is projected) are used.

[0272] For example, when the determination unit 146 determines that the position of the camera 301 that captured at least one second frame is closer to the subject than the position of the camera 301 when the frame for generating the pixels of the first three-dimensional model was captured, the updating unit 147 generates a new three-dimensional model by updating (in other words, changing) the pixels of the first three-dimensional model determined by the determination unit 142a to the projected pixels of at least one second frame determined by the determination unit 142a. That is, the updating unit 147 generates a new three-dimensional model using the pixels of the frame captured by the camera 301 that is closer to the subject, between the camera 301 that captured the frame for generating the pixels of the existing three-dimensional model and the camera 301 that captured the frame containing the pixels of the projected existing three-dimensional model.

[0273] For example, the determination unit 142a projects the existing three-dimensional model onto a frame in the frame set where the three-dimensional model has not been projected, and determines the pixels of the frame where the three-dimensional model is projected, i.e., the projection pixels. Here, when the second three-dimensional model generation unit 145a has not generated the second three-dimensional model, the existing three-dimensional model is the first three-dimensional model. In addition, when the second three-dimensional model generation unit 145 has generated the second three-dimensional model, the existing three-dimensional model is the second three-dimensional model generated by the second three-dimensional model generation unit 145.

[0274] The determination unit 142a and the three-dimensional model generation unit 143a repeatedly project the three-dimensional model onto the frame in the order of the frame set, thereby extracting the projection pixels of the existing three-dimensional model projected in the frame and the corresponding pixels of the existing three-dimensional model, and generating a new three-dimensional model using pixels of the frames captured by the camera 301 that captured the frames for generating the pixels of the existing three-dimensional model and the camera 301 that captured the frames containing the projection pixels of the projected existing three-dimensional model.

[0275] [Processing steps]

[0276] Next, refer to Fig.17 and Fig.18 , and detailed description of the processing steps of the three-dimensional model generation device of implementation mode 2.

[0277] Fig.17 This is a flowchart showing the three-dimensional model generation process executed by the three-dimensional model generation device 100 a according to the second embodiment.

[0278] First, the acquisition unit 130 acquires information indicating the position and posture of the camera 301, camera parameters, and multiple frames from the storage unit 120, and inputs them to the generation unit 140a (step S401). Specifically, similarly to the first embodiment, the receiving unit 110 first receives information indicating the position and posture of the camera 301, camera parameters, and multiple frames estimated by the estimation device 200, and outputs these data to the storage unit 120. In addition, the storage unit 120 also stores the information indicating the position and posture of the camera 301, camera parameters, and multiple frames received by the receiving unit 110. Then, the acquisition unit 130 executes step S401.

[0279] Next, the frame set generation unit 141a generates a frame set in the order of frames captured by the camera 301 (step S402). For example, the camera 301 associates the captured frames with the time information indicating the time when the frame was captured, and sends it to the three-dimensional model generation device 100a via the estimation device 200. The frame set generation unit 141a generates a frame set in the order of frames captured by the camera 301 based on the time information associated with each frame.

[0280] Next, the first three-dimensional model generation unit 144a generates a three-dimensional model (first three-dimensional model) based on the frame set (step S403). For example, the first three-dimensional model generation unit 144a generates the first three-dimensional model using any two frames (first frames) in the frame set.

[0281] Next, the generation unit 140a starts repetitive processing for a number of frames equal to or less than the number of frames included in the frame set (step S404). Specifically, the generation unit 140a repeatedly executes the processing of steps S405 to S408 for each of the plurality of frames included in the frame set.

[0282] Next, the determination unit 142a projects the existing three-dimensional model onto at least one frame (the second frame) in the frame set (step S405).

[0283] Next, the determination unit 142a extracts pixels of the three-dimensional points of the projected existing three-dimensional model in the frame projected with the existing three-dimensional model (step S406). More specifically, in step S406, the determination unit 142a projects the existing three-dimensional model to at least one second frame of the frames included in the frame set other than the first frame, and determines the projection pixels of at least one second frame of the projected existing three-dimensional model.

[0284] Next, the determination unit 146 determines whether the projection pixel determined by the determination unit 142a in step S406 is the nearest neighboring point (step S407). Specifically, the determination unit 146 determines whether at least one second shooting position at which at least one second frame is shot is closer to the specified area than any one of the plurality of first shooting positions at which a plurality of first frames are shot. More specifically, the determination unit 146 determines whether the position of the camera 301 that shoots the second frame containing the extracted pixel determined by the determination unit 142a in step S406 is closer to the subject than the position of the camera 301 when the frame used to generate the pixel of the existing three-dimensional model is shot, the pixel of the existing three-dimensional model being the pixel determined by the determination unit 142a in step S406. The determination unit 146 determines that the pixel contained in the frame shot by the camera 301 that is closer to the subject is the nearest neighboring point.

[0285] When the determination unit 146 determines that the projection pixel of the second frame determined in step S407 is the nearest neighbor (Yes in step S407), that is, when it is determined that at least one second shooting position is closer to the specified area than any of the plurality of first shooting positions, the updating unit 147 updates the existing three-dimensional model based on the projection pixel before the first three-dimensional model is projected (step S408). That is, the updating unit 147 updates the pixels of the existing three-dimensional model determined in step S406 to the projection pixels of the second frame determined in step S406.

[0286] On the other hand, when the determination unit 146 determines that the projection pixel of the second frame determined in step S407 is not the nearest neighbor (No in step S407), the updating unit 147 does not change the pixel of the existing three-dimensional model to the projection pixel of the second frame. In other words, the updating unit 147 does not perform the process of updating the pixel of the existing three-dimensional model.

[0287] Next, the identification unit 142 a identifies unprojected pixels on which the three-dimensional model is not projected (specifically, three-dimensional points constituting the three-dimensional model) in the second frame on which the three-dimensional model is projected (step S409 ).

[0288] Next, the second 3D model generation unit 145a generates a new 3D model (second 3D model) based on the unprojected pixels determined in step S409 (step S410). That is, the second 3D model generation unit 145a generates a second 3D model composed of points that do not exist in the existing 3D model.

[0289] Next, the generation unit 140 a ends the repetitive processing of the frame set (step S411 ).

[0290] Next, the output unit 150 combines and outputs the first three-dimensional model and the second three-dimensional model (step S412 ).

[0291] In addition, in order to perform the determination of step S407, the determination unit 146 may maintain information indicating the distance between the camera 301 and the three-dimensional point in the frame used to generate the existing three-dimensional model, or may maintain, in each three-dimensional point (pixel) of the existing three-dimensional model, the ID of the camera 301 that captured the frame containing each pixel, that is, information indicating the distance between the camera 301 and the subject when the frame was captured.

[0292] Fig.18 1 is a diagram for explaining the processing position of a frame in the process performed by the updating unit 147 included in the three-dimensional model generating device 100a according to the second embodiment. Fig.18 In FIG. 1 , the replacement range of pixels at time t+1 and time t+2, that is, the pixel positions in the frame where the existing three-dimensional model is projected, are indicated by hatching.

[0293] In addition, Fig.18 In the example shown, the frame at time t+2, the frame at time t+1, and the frame at time t are arranged in this order from near to far distances from the subject.

[0294] First, if Fig.18 As shown in (a), for example, it is assumed that no existing three-dimensional model is projected on any part of the frame at time t. In this case, the second three-dimensional model generation unit 145a generates a second three-dimensional model for all pixels of the frame at time t.

[0295] In addition, if Fig.18 As shown in (b), for example, assume that for the frame at time t+1, an existing three-dimensional model is projected at the portion indicated by the shadow (in this example, all frames at time t+1). Specifically, assume Fig.18 The shaded part in (b) is the projected area containing Fig.18 In this case, the updating unit 147 replaces the three-dimensional points (pixels) of the existing three-dimensional model generated (updated) using the frame at time t projecting the existing three-dimensional model with the pixels of the frame at time t+1, i.e., updates the model.

[0296] In addition, if Fig.18 As shown in (c), for example, assume that for the frame at time t+2, the existing three-dimensional model is projected at the position indicated by the shadow (in this example, all frames at time t+2). Specifically, assume Fig.18 The shaded part in (c) is projected by Fig.18 In this case, the updating unit 147 replaces the 3D points (pixels) of the existing 3D model with the pixels of the frame at time t+2.

[0297] Thus, for example, the updating unit 147 appropriately updates the pixels of the existing three-dimensional model based on the position of the camera 301 that captured the frame. For example, when the camera 301 moves in the direction of the camera optical axis (in other words, the positive direction of the reference axis) and captures the scene (subject), the updating unit 147 replaces the pixels of the existing three-dimensional model with a new three-dimensional model that generates all projected points in the current processing frame.

[0298] [Effects, etc.]

[0299] As described above, the three-dimensional model generation method of implementation mode 2 includes: a first three-dimensional model generation step (step S403), generating a first three-dimensional model of a specified area based on multiple first frames; a determination step (step S406), projecting the first three-dimensional model to at least one second frame (step S404), and determining multiple projection pixels of the projected first three-dimensional model in at least one second frame; a judgment step (step S407), judging whether at least one second shooting position of at least one second frame is closer to the specified area than any one of the multiple first shooting positions of the multiple first frames; and an updating step (step S408), when it is judged that at least one second shooting position is closer to the specified area than any one of the multiple first shooting positions (yes in step S407), updating the first three-dimensional model based on multiple projection pixels before the projected first three-dimensional model.

[0300] Thus, a three-dimensional model is generated using the frames captured by the camera 301 in the order along the time series of the captured frames. Therefore, by processing the frames in the order of capturing, a frame set in which the frame order is rearranged is not generated, and thus a three-dimensional model can be generated in a short time. In addition, a new three-dimensional model can be generated in which, for the pixels projected on the frame in the newly generated three-dimensional model, a high-precision pixel (i.e., the pixel of the frame captured by the camera 301 which is closer to the specified area) is used compared to the frame. Thus, a high-precision three-dimensional model can be generated.

[0301] In addition, for example, in the determining step, a plurality of unprojected pixels in at least one second frame to which the first three-dimensional model is not projected are determined (step S409). In addition, for example, the three-dimensional model generation method of embodiment 2 further includes a second three-dimensional model generation step of generating a second three-dimensional model based on the plurality of unprojected pixels (step S410).

[0302] Thereby, it is possible to generate the second three-dimensional model using non-projected pixels that are pixels not present in the first three-dimensional model.

[0303] (Other embodiments)

[0304] As mentioned above, the three-dimensional model generation method and the like of the present invention have been described based on the above-mentioned first and second embodiments, but the present invention is not limited to the above-mentioned first and second embodiments.

[0305] For example, in the above-mentioned embodiment, it is explained that the processing unit such as the generation unit 140 possessed by the three-dimensional model generation device is implemented by a CPU and a control program. For example, the constituent elements of the processing unit can also be respectively composed of one or more electronic circuits. One or more electronic circuits can be general-purpose circuits or dedicated circuits. In one or more electronic circuits, for example, semiconductor devices, ICs (Integrated Circuits), or LSIs (Large Scale Integrations) can also be included. ICs or LSIs can be integrated on one chip or on multiple chips. Here, they are called ICs or LSIs, but the naming method changes according to the degree of integration, and may be called system LSI, VLSI (Very Large Scale Integration) or ULSI (Ultra Large Scale Integration). In addition, FPGAs (Field Programmable Gate Arrays) programmed after the manufacture of LSIs can also be used for the same purpose.

[0306] In addition, all or specific forms of the present invention may also be implemented by a system, device, method, integrated circuit, or computer program. Alternatively, it may be implemented by a non-temporary recording medium readable by a computer such as an optical disk, HDD (Hard Disk Drive) or semiconductor memory storing the computer program. In addition, it may also be implemented by any combination of a system, device, method, integrated circuit, computer program, and recording medium.

[0307] In addition, various modifications that can be conceived by those skilled in the art to the embodiments and arbitrarily combining components and functions in the embodiments within the scope not departing from the gist of the present invention are also included in the present invention.

[0308] Industrial Applicability

[0309] The present invention can be applied to a three-dimensional model generation device or a three-dimensional model generation system, for example, it can be applied to generating a figure, identifying the structure of a terrain or a building, identifying human behavior, or generating a free viewpoint image.

[0310] Description of Reference Numerals

[0311] 100, 100a Three-dimensional model generating device

[0312] 110 Receiving Department

[0313] 120 Storage

[0314] 130 Acquisition Department

[0315] 140, 140a Generation unit

[0316] 141, 141a frame set generation unit

[0317] 142, 142a Determination

[0318] 143, 143a Three-dimensional model generation unit

[0319] 144, 144a First three-dimensional model generating unit

[0320] 145, 145a Second three-dimensional model generating unit

[0321] 146 Judgment Department

[0322] 147 Update Department

[0323] 150 Output

[0324] 200 Prediction device

[0325] 300 Camera Set

[0326] 301 Camera

[0327] 400, 400a 3D model generation system

[0328] 500, 500A Subject

[0329] 501 Reference Frame

[0330] 502 Visible Frame

[0331] 503 feature points

Claims

1. A three-dimensional model generation method, wherein: include: a first three-dimensional model generating step of generating a first three-dimensional model of a predetermined area based on a plurality of first frames; A determining step of projecting the first three-dimensional model onto at least one second frame, and determining a plurality of unprojected pixels in the at least one second frame onto which the first three-dimensional model is not projected, the at least one second frame being different from the plurality of first frames; as well as The second three-dimensional model generating step generates a second three-dimensional model based on the plurality of unprojected pixels.

2. The three-dimensional model generation method according to claim 1, wherein: Further including: a frame set generating step of generating a frame set consisting of a plurality of frames included in the plurality of captured frames based on the position and posture of the camera of each frame when the plurality of captured frames are captured; The frame set includes the plurality of first frames and the at least one second frame.

3. The three-dimensional model generation method according to claim 2, wherein: In the frame set generation step, the plurality of frames are arranged in the frame set in a frame order, In the determining step, a third three-dimensional model is projected onto the at least one second frame, and a plurality of unprojected pixels on which the third three-dimensional model is not projected in the at least one second frame on which the third three-dimensional model is projected are determined, wherein the third three-dimensional model is a generated three-dimensional model including the first three-dimensional model. The determining step and the second three-dimensional model generating step are repeatedly performed in the frame sequence.

4. The three-dimensional model generation method according to claim 3, wherein: In the frame set generation step, the frame set is generated in which the frames are arranged in the order of the position of the camera when each of the frames in the frame set is captured from near to far from the prescribed area.

5. The three-dimensional model generation method according to claim 4, wherein: In the frame set generation step, the frame set is generated as follows, in which, when the direction from the camera toward the specified area is set as the positive direction of the reference axis, the multiple frames are arranged in the order from near to far from the specified area in the positive direction of the reference axis according to the position of the camera when each frame in the multiple frames in the frame set is shot.

6. The three-dimensional model generation method according to claim 5, wherein: In the frame set generation step, the positive direction of the reference axis is calculated based on the optical axis direction of the camera when each of the plurality of frames is captured.

7. The three-dimensional model generation method according to claim 5, wherein: In the frame set generation step, selecting a representative frame from the plurality of frames, The optical axis direction of the camera that captured the representative frame is calculated as the positive direction of the reference axis.

8. The three-dimensional model generation method according to claim 5, wherein: In the frame set generation step, determining whether each of the plurality of frames satisfies a first condition, The frame set is generated from a plurality of frames satisfying the first condition among the plurality of frames.

9. The three-dimensional model generation method according to claim 8, wherein: In the frame set generation step, The first condition is determined to be that an angle formed by the positive direction of the reference axis and the optical axis direction of a camera when the plurality of frames included in the frame set are captured is equal to or smaller than a first angle.

10. The three-dimensional model generation method according to claim 1, wherein: The predetermined area is an area including at least one of a stationary object in stationary state and a moving object in motion as a subject.

11. A three-dimensional model generating device, wherein: have: A first three-dimensional model generating unit generates a first three-dimensional model of a predetermined area based on a plurality of first frames; a determining unit, projecting the first three-dimensional model onto at least one second frame, and determining a plurality of unprojected pixels in the at least one second frame onto which the first three-dimensional model is not projected, the at least one second frame being different from the plurality of first frames; as well as The second three-dimensional model generating unit generates a second three-dimensional model based on the unprojected pixels.

12. A method for generating a three-dimensional model, wherein: include: a first three-dimensional model generating step, generating a first three-dimensional model of a specified area based on a plurality of first frames; A determining step of projecting the first three-dimensional model onto at least one second frame, and determining a plurality of projection pixels onto which the first three-dimensional model is projected in the at least one second frame, wherein the at least one second frame is different from the plurality of first frames; a determination step of determining whether at least one second shooting position at which the at least one second frame is shot is closer to the specified area than any one of the plurality of first shooting positions at which the plurality of first frames are shot; as well as The updating step updates the first three-dimensional model based on the plurality of projection pixels before the first three-dimensional model is projected, when it is determined that the at least one second imaging position is closer to the predetermined area than any one of the plurality of first imaging positions.

13. The three-dimensional model generation method according to claim 12, wherein: In the determining step, a plurality of unprojected pixels in the at least one second frame on which the first three-dimensional model is not projected are determined, and The three-dimensional model generation method further includes a second three-dimensional model generation step of generating a second three-dimensional model based on the plurality of unprojected pixels.

Images