Three-dimensional model generating method and three-dimensional model generating device

Abstract

This three-dimensional model generation method includes: a first acquisition step (S101) for acquiring map information that is obtained by camera calibration executed by having a subject photographed by one or more cameras from a plurality of viewpoints, and that contains a plurality of three-dimensional points, each indicating a position on the subject in a three-dimensional space; a second acquisition step (S101) for acquiring a first image of a first viewpoint and a second image of a second viewpoint; a determination step (S104) for determining a search range on the three-dimensional space containing a first three-dimensional point on the subject corresponding to a first point of the first image, on the basis of the map information; a matching step (S105) for searching for a similar point similar to the first point, in a range corresponding to the search range on the second image; and a generation step (S106) for generating a three-dimensional model using searching results from the matching step.

Description

[Technical field]

[0001] The present disclosure relates to a three-dimensional model generating method and a three-dimensional model generating device. [Background technology]

[0002] Patent Document 1 discloses a technique for generating a three-dimensional model of a subject by using a plurality of images obtained by photographing the subject from a plurality of viewpoints. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-130146 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the process of generating a three-dimensional model, it is desirable to improve the generation accuracy of the three-dimensional model and to reduce the processing time.

[0005] The present disclosure provides a three-dimensional model generation method and the like that can improve the generation accuracy of a three-dimensional model and shorten the processing time for the generation process of the three-dimensional model. [Means for solving the problem]

[0006] A three-dimensional model generation method according to one aspect of the present disclosure is a three-dimensional model generation method executed by an information processing device, the three-dimensional model generation method including: a first acquisition step of acquiring map information obtained by camera calibration executed by having one or more cameras capture an image of an object from a plurality of viewpoints, the map information including a plurality of three-dimensional points each indicating a position on the object in a three-dimensional space; a second acquisition step of acquiring a first image of the object captured from the first viewpoint and a second image of the object captured from a second viewpoint; The plurality of three-dimensional points included ina determining step of determining a search range in a three-dimensional space including a first three-dimensional point on the object corresponding to a first point in the first image based on the first point; a matching step of searching for a similar point similar to the first point in a range corresponding to the search range on the second image; and a generating step of generating a three-dimensional model using a search result in the matching step. the search range includes a three-dimensional position on the subject estimated from the map information. .

[0007] A three-dimensional model generating device according to an embodiment of the present disclosure includes a first acquisition unit that acquires map information including a plurality of three-dimensional points obtained by camera calibration performed by having one or more cameras capture an image of an object from a plurality of viewpoints, each of the three-dimensional points indicating a position on the object in a three-dimensional space, a second acquisition unit that acquires a first image of the object captured from the first viewpoint and a second image of the object captured from a second viewpoint, and a second acquisition unit that acquires the map information. The plurality of three-dimensional points included in a determination unit that determines a search range in a three-dimensional space including a first three-dimensional point on the subject corresponding to a first point in the first image based on the above-mentioned, a matching unit that searches for a similar point similar to the first point in a range corresponding to the search range on the second image, and a generation unit that generates a three-dimensional model using a search result in the matching unit. The search range includes a three-dimensional position on the subject estimated from the map information. .

[0008] The present disclosure may be realized as a program that causes a computer to execute steps included in the above-mentioned three-dimensional model generating method. The present disclosure may also be realized as a non-transitory recording medium, such as a CD-ROM, on which the program is recorded and which is readable by a computer. The present disclosure may also be realized as information, data, or signals indicating the program. The program, information, data, and signals may be distributed via a communication network, such as the Internet. Effect of the Invention

[0009] According to the present disclosure, it is possible to provide a three-dimensional model generation method and the like that can improve the generation accuracy of a three-dimensional model and shorten the processing time for the generation process of a three-dimensional model. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram for explaining an overview of a three-dimensional model generating method according to the first embodiment. [Diagram 2] FIG. 2 is a block diagram showing a characteristic configuration of the three-dimensional model generating device according to the first embodiment. [Diagram 3] FIG. 3 is a diagram for explaining a method for generating map information by the estimation device. [Figure 4] FIG. 4 is a diagram for explaining a first interpolation method for interpolating a three-dimensional position between a plurality of three-dimensional points. [Diagram 5] FIG. 5 is a diagram showing an example of meshing the map information in the projection frame onto which the map information shown in FIG. 3 is projected. [Figure 6] FIG. 6 is a diagram for explaining a second interpolation method for interpolating a three-dimensional position between a plurality of three-dimensional points. [Figure 7] FIG. 7 is a diagram showing an example of dividing the projection frame onto which the map information shown in FIG. 3 is projected into a plurality of regions. [Figure 8] FIG. 8 is a flowchart showing an example of the operation of the three-dimensional model generating device. [Figure 9] FIG. 9 is a diagram for explaining the matching process when there is no limit to the search range. [Figure 10] FIG. 10 is a diagram for explaining the matching process when the search range is limited. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] (Background to this disclosure) In the technology disclosed in Patent Document 1, a three-dimensional model is generated by searching for similarities between multiple images. In general, when searching for similarities between one pixel of one image and another image, an epipolar line on the other image is calculated based on the geometric constraints of the camera, and all pixels on the epipolar line are searched for. For this reason, there is room for improving the processing speed of searching for similarities. In addition, if a similar subject is present on the epipolar line, the wrong similarities may be searched for, which may lead to a decrease in search accuracy.

[0012] Therefore, the present disclosure provides a three-dimensional model generation method and the like that can improve the generation accuracy of a three-dimensional model and shorten the processing time for the three-dimensional model.

[0013] A three-dimensional model generation method according to one embodiment of the present disclosure is a three-dimensional model generation method executed by an information processing device, and includes: a first acquisition step of acquiring map information obtained by camera calibration performed by having one or more cameras photograph a subject from multiple viewpoints, and including a plurality of three-dimensional points each indicating a position on the subject in three-dimensional space; a second acquisition step of acquiring a first image of the subject photographed from a first viewpoint and a second image of the subject photographed from a second viewpoint; a determination step of determining a search range in three-dimensional space including a first three-dimensional point on the subject corresponding to a first point in the first image based on the map information; a matching step of searching for a similar point similar to the first point in a range corresponding to the search range on the second image; and a generation step of generating a three-dimensional model using the search results in the matching step.

[0014] According to this, a search range is determined based on map information, and a similar point similar to the first point on the first image is searched for in a range corresponding to the search range on the second image limited by the search range. In this way, since a search for similar points is performed in a range where similar points are likely to exist based on map information, the accuracy of searching for similar points can be improved and the time required for the search process can be shortened. Therefore, the accuracy of generating a three-dimensional model can be improved and the processing time of the generation process of the three-dimensional model can be shortened.

[0015] Also, for example, the search range may be a range in a shooting direction from the first viewpoint, and in the matching step, an epipolar line corresponding to the first point in the second image may be limited to a length according to the search range, and a similar point that is similar to the first point may be searched for on the epipolar line of the second image.

[0016] According to this, a similar point that is similar to the first point is searched for on an epipolar line whose length is limited according to the search range, thereby improving the accuracy of searching for similar points and shortening the time required for the search process.

[0017] Also, for example, the determination step may include an interpolation step of using the plurality of three-dimensional points included in the map information to interpolate a three-dimensional point at which the subject is estimated to exist between two points included in the plurality of three-dimensional points, and in the determination step, the search range may be determined using the plurality of three-dimensional points interpolated in the interpolation step.

[0018] According to this, a 3D point where the subject is estimated to exist is interpolated between two of the multiple 3D points, so that the search range can be determined with high accuracy.

[0019] Also, for example, the interpolation step may include interpolating a 3D point where the subject is estimated to be present in an area defined by three points included in the plurality of 3D points.

[0020] According to this, a 3D point where a subject is estimated to exist is interpolated in a region defined by three of a plurality of 3D points, so that the search range can be determined with high accuracy.

[0021] Also, for example, in the determination step, a three-dimensional position based on the first viewpoint and a three-dimensional position on the subject corresponding to each of a plurality of regions on a projected image in which the plurality of three-dimensional points are projected onto the first image may be estimated to generate a distance image including the estimated three-dimensional positions of each region, and the search range may be determined based on the three-dimensional position of each region in the distance image.

[0022] According to this, since the three-dimensional position is calculated for each of a plurality of regions on the projected image, it is possible to calculate as many three-dimensional positions as are necessary for matching.

[0023] Also, for example, in the determining, the search range determined for each of the plurality of regions in the distance image may be widened as the estimation accuracy of the three-dimensional position of the corresponding region decreases.

[0024] According to this, the search range is widened as the accuracy of the three-dimensional position on the interpolated three-dimensional shape decreases, so that the search range can be determined according to the accuracy.

[0025] Also, for example, the estimation accuracy of the three-dimensional position may be higher as the distance of the three-dimensional position from the first viewpoint decreases.

[0026] Therefore, the search range can be determined according to the distance of the three-dimensional position from the first viewpoint.

[0027] Also, for example, the estimation accuracy of the three-dimensional position may be higher as the three-dimensional position is closer to at least one of the multiple three-dimensional points.

[0028] Therefore, the search range can be determined according to the distance from at least one of the multiple three-dimensional points of the three-dimensional position.

[0029] Also, for example, the estimation accuracy of the three-dimensional position may be higher as the density of the multiple three-dimensional points in the area in which the three-dimensional position exists increases.

[0030] Therefore, the search range can be determined according to the density of multiple three-dimensional points in the area where the three-dimensional position exists.

[0031] Also, for example, in the camera calibration, the position and orientation of the one or more cameras may be calculated, and the first image and the second image may be included in a plurality of images taken for the camera calibration.

[0032] Moreover, a three-dimensional model generating device according to one embodiment of the present disclosure includes a first acquisition unit that acquires map information obtained by camera calibration performed by having one or more cameras photograph an object from multiple viewpoints, and including multiple three-dimensional points each indicating a position on the object in three-dimensional space; a second acquisition unit that acquires a first image of the object photographed from the first viewpoint and a second image of the object photographed from a second viewpoint; a determination unit that determines a search range in three-dimensional space including a first three-dimensional point on the object corresponding to a first point in the first image based on the map information; a matching unit that searches for a similar point similar to the first point in a range corresponding to the search range on the second image; and a generation unit that generates a three-dimensional model using the search results from the matching unit.

[0033] According to this, a search range is determined based on map information, and a similar point similar to a first point on a first image is searched for in a range corresponding to the search range on a second image restricted between images. In this way, since the search for similar points is performed in a restricted search range, it is possible to improve the accuracy of the search for similar points and to shorten the time required for the search process.

[0034] Each embodiment of the three-dimensional model generating method and the like according to the present disclosure will be described in detail below with reference to the drawings. Note that each embodiment described below shows one specific example of the present disclosure. Therefore, the numerical values, shapes, materials, components, arrangement and connection of the components, steps, order of steps, and the like shown in each embodiment below are merely examples and are not intended to limit the present disclosure.

[0035] In addition, each drawing is a schematic diagram and is not necessarily a precise illustration. In addition, in each drawing, the same reference numerals are used for substantially the same configurations, and duplicated explanations may be omitted or simplified.

[0036] (Embodiment 1) [overview] First, an overview of a three-dimensional model generating method according to the first embodiment will be described with reference to FIG.

[0037] Fig. 1 is a diagram for explaining an overview of a three-dimensional model generating method according to embodiment 1. Fig. 2 is a block diagram showing a characteristic configuration of a three-dimensional model generating device 100 according to embodiment 1.

[0038] In the three-dimensional model generating method, as shown in Fig. 1, a three-dimensional model of a predetermined area is generated from a plurality of images captured from a plurality of different viewpoints using a plurality of cameras 301. Here, the predetermined area is an area including a stationary object or a moving moving object such as a person, or both. In other words, the predetermined area is an area including at least one of a stationary object and a moving moving object as a subject, for example.

[0039] Examples of a predetermined area including stationary objects and moving objects include a venue where a sporting event such as a basketball game is being held, or a space on a road where people or cars are present. The predetermined area may include not only a specific object that is the subject, but also scenery and the like. FIG. 1 illustrates an example in which the subject 500 is a building. In the following description, a predetermined area including not only a specific object that is the subject, but also scenery and the like, is also simply referred to as the subject.

[0040] As shown in FIG. 2, the three-dimensional model generation system 400 includes a camera group 300 including a plurality of cameras 301, an estimation device 200, and a three-dimensional model generation device 100.

[0041] (Multiple Cameras) The multiple cameras 301 are multiple imaging devices that capture an image of a predetermined area. The multiple cameras 301 each capture an image of a subject, and output the captured frames to the estimation device 200. In the first embodiment, the camera group 300 includes two or more cameras 301. The multiple cameras 301 capture the same subject from different viewpoints. In other words, the frames are images.

[0042] Although the three-dimensional model generation system 400 is described as including the camera group 300, the present invention is not limited thereto, and may include a single camera 301. For example, in the three-dimensional model generation system 400, a single camera 301 may be moved to capture an object existing in real space, so as to generate a multi-viewpoint image made up of a plurality of frames having different viewpoints. Each of the plurality of frames is a frame captured (generated) by the camera 301 in which at least one of the position and the orientation of the camera 301 is different from the others.

[0043] In addition, each camera 301 may be a camera that generates a two-dimensional image, or may be a camera equipped with a three-dimensional measurement sensor that generates a three-dimensional model. In the first embodiment, each of the multiple cameras 301 is a camera that generates a two-dimensional image.

[0044] The multiple cameras 301 may be directly connected to the estimation device 200 via wired or wireless communication so that each of them can output frames captured by the cameras 301 to the estimation device 200, or may be indirectly connected to the estimation device 200 via a hub (not shown) such as a communication device or a server.

[0045] Note that the frames captured by the multiple cameras 301 may be output in real time to the estimation device 200. In addition, the frames may be once recorded in an external storage device such as a memory or a cloud server, and then output from the external storage device to the estimation device 200.

[0046] Furthermore, each of the multiple cameras 301 may be a fixed camera such as a surveillance camera, or a mobile camera such as a video camera, smartphone, or wearable camera, or a mobile camera such as a drone with a shooting function.

[0047] (estimation device) The estimation device 200 performs camera calibration by having one or more cameras 301 capture an image of a subject from multiple viewpoints. The estimation device 200 performs camera calibration to estimate the positions and orientations of the multiple cameras 301 based on multiple frames captured by the multiple cameras 301, for example. Here, the orientation of the camera 301 indicates at least one of the shooting direction of the camera 301 and the tilt of the camera 301. The shooting direction of the camera 301 is the direction of the optical axis of the camera 301. The tilt of the camera 301 is the rotation angle around the optical axis of the camera 301 from a reference orientation.

[0048] Specifically, the estimation device 200 estimates camera parameters of the multiple cameras 301 based on multiple frames acquired from the multiple cameras 301. Here, the camera parameters are parameters indicating the characteristics of the camera 301, and are multiple parameters including internal parameters consisting of the focal length and image center of the camera 301, and external parameters indicating the position (more specifically, the three-dimensional position) and orientation of the camera 301. In other words, the position and orientation of each of the multiple cameras 301 can be obtained by estimating the camera parameters of each of the multiple cameras 301.

[0049] Note that there is no particular limitation on the method by which the estimation device 200 estimates the position and orientation of the camera 301. The estimation device 200 may estimate the positions and orientations of the multiple cameras 301 using, for example, a Visual-SLAM (Simultaneous Localization and Mapping) technique. Alternatively, the estimation device 200 may estimate the positions and orientations of the multiple cameras 301 using, for example, a Structure-From-Motion technique.

[0050] Here, a method for generating map information by the estimation device 200 will be described with reference to FIG.

[0051] 3, the estimation device 200 uses the Visual-SLAM technique or the Structure-From-Motion technique to extract characteristic points as feature points 541, 542, and 543 from each of a plurality of frames 531, 532, and 533 captured by a plurality of cameras 301, and performs a feature point search to extract a set of similar points that are similar between a plurality of frames from the extracted plurality of feature points 541, 542, and 543. By performing the feature point search, the estimation device 200 can identify points on the subject 510 that are commonly captured in the plurality of frames 531, 532, and 533, and can therefore use the extracted sets of similar points to determine the three-dimensional coordinates of the points on the subject 510 according to the principle of triangulation.

[0052] In this way, the estimation device 200 can extract a plurality of sets of similar points and estimate the position and orientation of each camera 301 by using the plurality of sets of similar points. In the process of estimating the position and orientation of each camera 301, the estimation device 200 calculates three-dimensional coordinates for each set of similar points and generates map information 520 including a plurality of three-dimensional points indicated by the calculated plurality of three-dimensional coordinates. Each of the plurality of three-dimensional points indicates a position on the subject in three-dimensional space. The estimation device 200 obtains the position and orientation of each camera 301 and the map information as an estimation result. The obtained map information is information with a higher accuracy than a predetermined accuracy because it has been optimized together with the camera parameters. In addition, the map information includes the three-dimensional positions of each of the plurality of three-dimensional points. Note that the map information may include not only the plurality of three-dimensional positions, but also the color of each three-dimensional point, the surface shape around each three-dimensional point, information indicating which frame each three-dimensional point was generated by, and the like.

[0053] Furthermore, in order to speed up the estimation process, the estimation device 200 may generate map information including a sparse three-dimensional point cloud by limiting the number of sets of similar points to a predetermined number. This is because the estimation device 200 can estimate the position and orientation of each camera 301 with sufficient accuracy even with a predetermined number of sets of similar points. Note that the predetermined number may be determined as a number that allows the position and orientation of each camera 301 to be estimated with sufficient accuracy. Furthermore, the estimation device 200 may estimate the position and orientation of each camera 301 using a set of similar points that is similar to each other at a predetermined similarity or higher among the sets of similar points. As a result, the estimation device 200 can limit the number of sets of similar points used in the estimation process to the number of sets that are similar to each other at a predetermined similarity or higher.

[0054] Furthermore, the estimation device 200 may calculate the distance between the camera 301 and the subject as a camera parameter based on the position and orientation of the camera 301 estimated using the above technique. The three-dimensional model generation system 400 may include a distance measuring sensor, and the distance between the camera 301 and the subject may be measured using the distance measuring sensor.

[0055] The estimation device 200 may be directly connected to the three-dimensional model generating device 100 by wired communication or wireless communication, or may be indirectly connected to the estimation device 200 via a hub (not shown) such as a communication device or a server. In this way, the estimation device 200 outputs to the three-dimensional model generating device 100 a plurality of frames received from the plurality of cameras 301 and a plurality of estimated camera parameters of the plurality of cameras 301.

[0056] The estimation result by the estimating device 200 may be output in real time to the three-dimensional model generating device 100. In addition, the estimation result may be once recorded in an external storage device such as a memory or a cloud server, and then output from the external storage device to the three-dimensional model generating device 100.

[0057] The estimation device 200 includes at least a computer system including, for example, a control program, a processing circuit such as a processor or logic circuit that executes the control program, and a recording device such as an internal memory or an accessible external memory that stores the control program.

[0058] (3D model generation device) The three-dimensional model generating device 100 generates a three-dimensional model of a predetermined area based on multiple frames captured by multiple cameras 301 and the estimation results of the estimation device 200. Specifically, the three-dimensional model generating device 100 is a device that executes a three-dimensional model generation process that generates a three-dimensional model of a subject in a virtual three-dimensional space based on the camera parameters of each of the multiple cameras 301 and multiple frames.

[0059] The three-dimensional model of the subject is data including the three-dimensional shape and color of the subject, which is restored in a virtual three-dimensional space from frames in which the actual object of the subject is photographed. The three-dimensional model of the subject is a set of points indicating the three-dimensional positions of each of a plurality of points on the subject captured in a plurality of two-dimensional images captured by a plurality of cameras 301 from multiple viewpoints, i.e., from a plurality of different viewpoints.

[0060] The three-dimensional position is represented by, for example, ternary information consisting of an X component, a Y component, and a Z component that indicate the position on each of the mutually orthogonal X-axis, Y-axis, and Z-axis. Note that the information contained in the multiple points indicating the three-dimensional position may include not only the three-dimensional position (i.e., information indicating coordinates), but also information indicating the color of each point, information indicating the surface shape of each point and its surroundings, etc.

[0061] The three-dimensional model generating device 100 includes at least a computer system including, for example, a control program, a processing circuit such as a processor or logic circuit that executes the control program, and a recording device such as an internal memory that stores the control program or an accessible external memory. The three-dimensional model generating device 100 is an information processing device. The functions of each processing unit of the three-dimensional model generating device 100 may be realized by software or hardware.

[0062] Furthermore, the three-dimensional model generating device 100 may store camera parameters in advance. In this case, the three-dimensional model generating system 400 may not include the estimation device 200. Furthermore, the multiple cameras 301 may be connected to the three-dimensional model generating device 100 wirelessly or via wires so as to be able to communicate with each other.

[0063] Furthermore, a plurality of frames captured by the camera 301 may be directly output to the three-dimensional model generation device 100. In this case, the camera 301 may be directly connected to the three-dimensional model generation device 100 by, for example, wired communication or wireless communication, or may be indirectly connected to the three-dimensional model generation device 100 via a hub (not shown) such as a communication device or a server.

[0064] [Configuration of the 3D model generation device] Next, the configuration of the three-dimensional model generating device 100 will be described in detail with reference to FIG.

[0065] The three-dimensional model generating device 100 is a device that generates a three-dimensional model from a plurality of frames. The three-dimensional model generating device 100 includes a receiving unit 110, a storage unit 120, an acquiring unit 130, a determining unit 140, a generating unit 150, and an output unit 160.

[0066] The receiving unit 110 receives a plurality of frames captured by the plurality of cameras 301 and an estimation result by the estimation device 200 from the estimation device 200. As a result, the receiving unit 110 acquires a first frame (first image) of the subject captured from a first viewpoint and a second frame (second image) of the subject captured from a second viewpoint. That is, the plurality of frames received by the receiving unit 110 include the first frame and the second frame. The receiving unit 110 outputs the received plurality of frames and the estimation result to the storage unit 120. The receiving unit 110 is, for example, a communication interface for communicating with the estimation device 200. When the three-dimensional model generating device 100 and the estimation device 200 communicate wirelessly, the receiving unit 110 includes, for example, an antenna and a wireless communication circuit. Alternatively, when the three-dimensional model generating device 100 and the estimation device 200 communicate wiredly, the receiving unit 110 includes, for example, a connector connected to a communication line and a wired communication circuit. The receiving unit 110 is an example of a first acquisition unit and a second acquisition unit. In this manner, the first acquisition unit and the second acquisition unit may be realized by one processing unit, or may be realized by two independent processing units. Note that the receiving unit 110 may receive multiple frames from multiple cameras 301 without going through the estimation device 200.

[0067] The storage unit 120 stores a plurality of frames and estimation results received by the receiving unit 110. The storage unit 120 also stores a search range calculated by the determining unit 140. The storage unit 120 may store processing results of processing units included in the three-dimensional model generating device 100. The storage unit 120 stores, for example, control programs executed by each processing unit included in the three-dimensional model generating device 100. The storage unit 120 is realized by, for example, a hard disk drive (HDD), a flash memory, etc.

[0068] The acquisition unit 130 acquires map information of the estimation result stored in the storage unit 120 from the storage unit 120, and outputs the map information to the determination unit 140. The acquisition unit 130 also acquires a plurality of frames and camera parameters of each camera 301 of the estimation result stored in the storage unit 120 from the storage unit 120, and outputs the map information to the generation unit 150.

[0069] The three-dimensional model generating device 100 may not include the storage unit 120 and the acquisition unit 130. In this case, the receiving unit 110 may output map information from the estimation result received from the estimation device 200 to the determining unit 140. The receiving unit 110 may also output a plurality of frames received from a plurality of cameras 301 and camera parameters of each camera 301 from the estimation result received from the estimation device 200 to the generating unit 150.

[0070] The determination unit 140 determines a search range to be used for searching for a plurality of similar points between a plurality of frames, based on the map information acquired from the storage unit 120 by the acquisition unit 130. The search range is a range in a three-dimensional space including a first three-dimensional point on the subject corresponding to a first point on the first frame. The search range can also be said to be a range in a three-dimensional space in which the first three-dimensional point is likely to exist. The search range is also a range in a shooting direction from a first viewpoint at which the first frame is shot. The determination unit 140 uses the plurality of three-dimensional points included in the map information to interpolate a three-dimensional point where the subject is estimated to exist between the plurality of three-dimensional points.

[0071] As described above, the map information represents the three-dimensional shape of the subject as a sparse three-dimensional point cloud. The determining unit 140 estimates a rough three-dimensional position of the subject's surface by filling in the spaces between the multiple three-dimensional points included in the sparse three-dimensional point cloud with multiple planes, and generates the estimation result as three-dimensional information. In this way, the determining unit 140 densifies the sparse three-dimensional point cloud included in the map information into a dense three-dimensional point cloud.

[0072] Specifically, for each of a plurality of pixels on the projection frame onto which a plurality of three-dimensional points are projected onto the first frame, the determination unit 140 estimates a three-dimensional position on the subject that corresponds to the pixel and is based on a first viewpoint from which the first frame was captured. In this way, the determination unit 140 generates a distance image including a plurality of pixels, each of which includes the estimated three-dimensional position.

[0073] Note that the multiple frames used for camera calibration and the multiple frames used for generating the 3D model may be the same or different, i.e., the first and second frames may be included in the multiple frames captured for camera calibration.

[0074] There are two possible methods for interpolating a plurality of three-dimensional points on a plurality of planes. First, the first interpolation method will be described with reference to Fig. 4. Fig. 4 is a diagram for explaining the first interpolation method for interpolating a three-dimensional position between a plurality of three-dimensional points.

[0075] In the first interpolation method, the determination unit 140 uses a plurality of three-dimensional points included in the map information to mesh the plurality of three-dimensional points, thereby interpolating a three-dimensional position on the subject between the plurality of three-dimensional points. For example, as shown in (a) of FIG. 4, the determination unit 140 acquires a projected frame 610 in which a plurality of three-dimensional points 601 included in the map information are projected onto a first frame of the plurality of frames. The determination unit 140 may acquire the projected frame 610 by projecting the plurality of three-dimensional points 601 onto the first frame, or, if the map information includes information indicating which of the frames the plurality of three-dimensional points 601 correspond to the feature points of, the plurality of three-dimensional points 601 may be associated with the feature points on the first frame using the information to acquire the projected frame 610.

[0076] Next, as shown in (b) of Fig. 4, the determination unit 140 meshes the map information on the projection frame 610 onto which the map information is projected, to form a plurality of meshes 611. Each mesh is a polygon having a plurality of 3D points 601 as vertices and surrounded by a plurality of sides 612 connecting two adjacent points among the plurality of 3D points.

[0077] Then, as shown in (c) of FIG. 4, the determination unit 140 calculates a plane equation for each mesh 611, and generates a three-dimensional shape 621 in which each mesh 611 is interpolated with a plane 622. After that, the determination unit 140 calculates, for each pixel in the projection frame 610, a three-dimensional position on the three-dimensional shape 621 that corresponds to the pixel and is a three-dimensional position based on the first viewpoint at which the first frame was captured, thereby generating a distance image 620 in which a three-dimensional position is indicated for each pixel. In this way, the determination unit 140 interpolates a plane 622 for each mesh 611 that is an area defined by three points among a plurality of three-dimensional points 601 included in the map information. In this way, the determination unit 140 interpolates a three-dimensional point where it is estimated that the subject is present.

[0078] The equation of the plane is calculated as follows. The determination unit 140 calculates the equation of the plane from the vertices of the mesh 611, that is, the multiple three-dimensional points 601. When the vertices of the mesh 611 are points A, B, and C, the determination unit 140 calculates the normal vector of the plane from the cross product shown in Equation 1.

number

[0079]

number

[0080] From equation 1, the equation of the plane is expressed as equation 2.

[0081] lX+mY+nZ=D (Formula 2)

[0082] Here, D is a value calculated from the values ​​of l, m, and n and the three-dimensional coordinates of a three-dimensional point in a plane. d ,Y d ,Z d ), then D is expressed by Equation 3.

[0083] D=lX d +mY d +nZ d (Formula 3)

[0084] Next, the three-dimensional coordinates are transformed into the image coordinate system of the first frame using equations 4 and 5.

[0085] x=X / Z (formula 4) y=Y / Z (Equation 5)

[0086] Z can be expressed by formula 6 using formulas 3, 4 and 5.

[0087] 1 / Z=lx / D+my / D+n / D (Equation 6)

[0088] Here, X, Y, and Z indicate a three-dimensional position of the mesh 611 in the plane 622, and x and y indicate the image coordinates of the mesh 611 of the first frame in the plane 622, that is, the two-dimensional position. Note that in the calculation by the determination unit 140 described above, it is assumed that the camera that captures the first frame is a pinhole camera.

[0089] The plane 622 is mainly a triangle and a quadrangle. The determining unit 140 may exclude an erroneous mesh from the generated meshes. For example, when the three-dimensional distance of an edge of a mesh (i.e., the length of the edge in three-dimensional space) is greater than an arbitrary value, the determining unit 140 may exclude a mesh having an edge whose three-dimensional distance is greater than an arbitrary value as an erroneous mesh.

[0090] 5, for example, in projected frame 551 in which map information shown in Fig. 3 is projected onto the first frame, determining unit 140 meshes map information 520, calculates a plane equation of each mesh 552, and estimates the three-dimensional shape of subject 510. Then, determining unit 140 calculates, for each pixel in projected frame 551, the distance from the first viewpoint to the point on the three-dimensional shape that corresponds to that pixel, and determines a search range based on the calculated distance.

[0091] Next, the second interpolation method will be described with reference to Fig. 6. Fig. 6 is a diagram for explaining the second interpolation method for interpolating a three-dimensional position between a plurality of three-dimensional points.

[0092] In the second interpolation method, the determination unit 140 divides the first frame into a plurality of regions using pixel values ​​of each pixel in the first frame among the plurality of frames, and for each of the divided regions, uses a plurality of three-dimensional points projected onto the first frame to interpolate a three-dimensional position on the subject between the plurality of three-dimensional points. For example, as shown in (a) of FIG. 6, the determination unit 140 acquires a projected frame 710 in which a plurality of three-dimensional points 701 included in the map information are projected onto the first frame among the plurality of frames. The determination unit 140 may acquire the projected frame 710 by projecting the plurality of three-dimensional points 701 onto the first frame, or, if the map information includes information indicating which feature points of which frame the plurality of three-dimensional points 701 correspond to, may acquire the projected frame 710 by associating the feature points on the first frame with the plurality of three-dimensional points 701 using the information.

[0093] Next, as shown in FIG. 6B, the determination unit 140 divides the projection frame 710 onto which the map information is projected into a plurality of regions 712 using the pixel values ​​of the pixels of the projection frame 710. For example, the determination unit 140 may determine a group of pixels having similar brightness and similar color as one region using at least one of brightness, color, and edge obtained from the pixel values ​​of each pixel of the projection frame 710, or may determine a region separated by an edge as one region. Similar brightness is a group of brightnesses included in a width of a predetermined brightness range that is smaller than the brightness range from the minimum brightness to the maximum brightness that can be expressed as pixel values. Also, similar colors are a group of colors included in a width of a predetermined color difference range.

[0094] 6(c), the determination unit 140 uses one or more three-dimensional points 701 included in each of the divided regions 712 to estimate a three-dimensional position on the subject at each of the positions of the multiple pixels included in each region 712 in the projection frame 710, thereby generating a distance image 720 in which the three-dimensional position of each pixel is indicated. For example, the determination unit 140 estimates the average of the distances from the first viewpoint to each of the one or more three-dimensional points 701 included in the corresponding region 712 as the three-dimensional position on the subject at each of the positions of the multiple pixels included in each region 712.

[0095] The average distance is calculated as follows.

[0096] One 3D point 701 located in each region 712 is represented as Pn={Xn, Yn, Zn}, where {X, Y, Z} are 3D coordinates in the camera coordinate system, and n={1,...,N}, where N is the number of 3D points included in the map information.

[0097] The average of the distances from the first viewpoint, that is, the camera position, to each of the multiple 3D points 701 in the area 712 is calculated using Equation 7.

[0098]

number

[0099] Next, the image coordinate system of the first frame is transformed into three-dimensional coordinates using equations 8 and 9.

[0100] X=Z mean ×x (Formula 8) Y=Z mean ×y (Formula 9)

[0101] The determination unit 140 uses Equation 7, Equation 8, and Equation 9 to interpolate the three-dimensional position in each region 712. For this reason, the determination unit 140 generates a three-dimensional shape 721 in which each region 712 is interpolated on a plane 722 at a fixed distance from the first viewpoint. Thereafter, the determination unit 140 calculates, for each pixel in the projection frame 710, a three-dimensional position on the three-dimensional shape 721 that corresponds to that pixel and is based on the first viewpoint at which the first frame was captured, thereby generating a distance image 720 in which the three-dimensional position of each pixel is indicated.

[0102] In the above calculation by determining unit 140, it is assumed that the camera that captures the first frame is a pinhole camera.

[0103] In addition, when region 712 contains multiple three-dimensional points 701 and one of the three-dimensional points is a point that is away from a set of other multiple three-dimensional points by an arbitrary value or more, determination unit 140 may exclude the one three-dimensional point that is away by an arbitrary value or more from the distance averaging.

[0104] 3 is projected onto the first frame, into a plurality of regions 562, and estimates the three-dimensional shape of the subject 510 for each of the divided regions 562 using one or more three-dimensional points included in each region. Then, for each pixel of the projected frame 561, the determination unit 140 calculates the distance from the first viewpoint to the point on the three-dimensional shape corresponding to the pixel, and determines a search range based on the calculated distance.

[0105] In addition, when dividing the first frame into a plurality of regions in the second interpolation method, the determination unit 140 may divide the first frame into a plurality of regions using a learning model learned by machine learning. The learning model used in this case may be, for example, a model generated by machine learning using ground truth data obtained by dividing a plurality of learning frames into a plurality of regions for each subject by a person in advance.

[0106] In addition, in both of the above two interpolation methods, the determination unit 140 interpolates the three-dimensional position on the subject in the first frame, that is, the projected frame obtained by projecting a plurality of three-dimensional points included in the map information onto the two-dimensional image, but this is not limited to the above. The determination unit 140 may also interpolate the three-dimensional position on the subject in a three-dimensional space. The three-dimensional position to be interpolated here may be represented by a point group or a plane.

[0107] Furthermore, in either of the above two interpolation methods, when projecting a plurality of three-dimensional points included in the map information onto the first frame, the determination unit 140 may project onto the first frame a point cloud visible from a first viewpoint at which the first frame is captured, but may not need to project a point cloud invisible from the first viewpoint. In other words, the determination unit 140 may project onto the first frame a point cloud on the surface of the subject on the first viewpoint side, but may not need to project a point cloud on the surface of the subject opposite to the first viewpoint.

[0108] The determination unit 140 may also interpolate the three-dimensional position on the subject using a learning model learned by machine learning. The learning model is, for example, a model generated by machine learning using a plurality of learning data, and in this case, each of the plurality of learning data may be data including a combination of acquired three-dimensional point cloud data of the learning subject and a plurality of images of the learning subject taken from different viewpoints. The three-dimensional point cloud data may be data acquired using a distance sensor or the like for each subject. In other words, in machine learning, a learning model is generated for estimating the position of a three-dimensional point on the subject appearing in each of the plurality of images, using the three-dimensional point cloud data as correct answer data.

[0109] In addition, the determination unit 140 may interpolate the three-dimensional point where the subject is estimated to exist on a plane, but this is not limited to the above. For example, the determination unit 140 may interpolate the three-dimensional point where the subject is estimated to exist between two points among the multiple three-dimensional points included in the map information. In other words, the determination unit 140 may interpolate the three-dimensional point where the subject is estimated to exist with a line segment connecting two points.

[0110] Based on the three-dimensional position indicated by each pixel of the distance image obtained by the first or second interpolation method, the determination unit 140 determines a search range including the three-dimensional position on a line connecting the first viewpoint and the three-dimensional position. Specifically, the determination unit 140 determines a range of a predetermined size with the three-dimensional position as the midpoint as the search range. Specifically, the determination unit 140 also obtains a distance from the first viewpoint to a point corresponding to the position of each pixel on the subject, and determines the size of the search range according to the obtained distance. The search range is a search range for searching for a point similar to the point of each pixel of the first frame from a frame different from the first frame.

[0111] The determining unit 140 widens the search range determined for each of a plurality of pixels in the distance image as the estimation accuracy of the three-dimensional position of the corresponding pixel decreases.

[0112] Specifically, the determination unit 140 may determine that the estimation accuracy of the three-dimensional position of the corresponding pixel is lower as the distance from the first viewpoint of the three-dimensional position to be estimated is smaller. This is because one pixel is a two-dimensional area having a spread, and therefore the three-dimensional position to which the one pixel corresponds becomes a smaller area in three-dimensional space as it is closer to the first viewpoint. In other words, the closer to the first viewpoint, the narrower the area in which the estimated three-dimensional position may exist, and therefore the higher the estimation accuracy of the three-dimensional position. In this way, the determination unit 140 determines the search range so that the smaller the distance from the first viewpoint of the three-dimensional position is, the smaller the search range becomes.

[0113] The determination unit 140 may also determine that the estimation accuracy of the three-dimensional position of the corresponding pixel in the first interpolation method is higher as the three-dimensional position of the estimation target is closer to at least one of the multiple three-dimensional points included in the map information. This is because, in the first interpolation method, the three-dimensional shape of the subject is estimated by generating a mesh with multiple three-dimensional points as vertices, so that the closer to the multiple three-dimensional points, the higher the estimation accuracy of the three-dimensional shape of the subject. In other words, the closer to at least one of the multiple three-dimensional points, the more likely the estimated three-dimensional position is to be located at a position closer to the actual three-dimensional position of the subject. In this way, the determination unit 140 determines the search range so that the closer the three-dimensional position is to at least one of the multiple three-dimensional points, the smaller the search range becomes.

[0114] In this case, the determination unit 140 may determine the estimation accuracy of the three-dimensional position by using a distance between the three-dimensional position in the mesh and one of the multiple three-dimensional points located at the multiple vertices of the mesh. The determination unit 140 may determine the estimation accuracy of the three-dimensional position by using an average value of multiple distances between the three-dimensional position in the mesh and each of the multiple three-dimensional points located at the multiple vertices of the mesh.

[0115] For example, the determination unit 140 arranges a predetermined Gaussian distribution that spreads on the distance image with the center of gravity of the pixel where the three-dimensional point is located as the center for each of the multiple three-dimensional points located at the multiple vertices of the mesh. The predetermined Gaussian distribution is arranged so that the average of the predetermined Gaussian distribution coincides with the center of gravity of the pixel. Therefore, the probability density of the predetermined Gaussian distribution is highest at the position of the center of gravity of the pixel, and becomes lower as the pixel is farther away from the position of the center of gravity of the pixel. The determination unit 140 may determine that the estimation accuracy of the three-dimensional position of the corresponding pixel is higher as the probability density of the predetermined Gaussian distribution assigned by arranging the predetermined Gaussian distribution in the distance image is higher.

[0116] The determination unit 140 may arrange a predetermined Gaussian distribution for each of the meshes, the distribution spreading on the distance image with the center of gravity of the mesh as the center. In this case, the predetermined Gaussian distribution is arranged so that the average of the predetermined Gaussian distribution coincides with the center of gravity of the mesh. Therefore, the probability density of the predetermined Gaussian distribution is highest at the position of the center of gravity of the mesh, and decreases as the pixel is farther away from the position of the center of gravity of the pixel. The determination unit 140 may determine that the estimation accuracy of the three-dimensional position of the corresponding pixel is lower as the probability density of the predetermined Gaussian distribution allocated by arranging the predetermined Gaussian distribution in the distance image is higher.

[0117] The determination unit 140 may also determine that the estimation accuracy of the three-dimensional position of the corresponding pixel is higher as the density of the multiple three-dimensional points included in the map information in the region where the three-dimensional position of the estimation target exists is higher. This is because the more the density of the multiple three-dimensional points in the region is higher, the closer the three-dimensional position is to one of the multiple three-dimensional points. For example, in the first interpolation method, the determination unit 140 may determine the density of the multiple three-dimensional points according to the size of the mesh in which the three-dimensional position of the estimation target exists. That is, the determination unit 140 determines that the density is higher as the mesh size is smaller. For example, in the second interpolation method, the determination unit 140 may determine the density of the multiple three-dimensional points according to the number of three-dimensional points included in the region in which the three-dimensional position of the estimation target exists. That is, the determination unit 140 determines that the density is higher as the number of three-dimensional points included in the region is larger. In this way, the determination unit 140 determines the search range so that the search range is smaller as the density of the multiple three-dimensional points in the region in which the three-dimensional position exists is higher.

[0118] As described above, the determining unit 140 determines the estimation accuracy of the three-dimensional position using a plurality of methods. However, it is possible to adopt one of these methods, or to adopt a combination of a plurality of these methods.

[0119] The three-dimensional position calculated for each pixel may be indicated by the distance from the first viewpoint to a point on the three-dimensional shape corresponding to the pixel, or may be indicated by three-dimensional coordinates in a coordinate system with the first viewpoint as the origin. In addition, among the pixels of the distance image, pixels in an area in which the generated three-dimensional shape does not exist in the first frame may not have information indicating a three-dimensional position, or may have information indicating that they do not have information indicating a three-dimensional position. In addition, the determination unit 140 calculates the three-dimensional position on the three-dimensional shape corresponding to each of the pixels included in the projection frame, but is not limited to calculating the three-dimensional position on a pixel-by-pixel basis, and may calculate the three-dimensional position on the three-dimensional shape corresponding to each of the areas on the projection frame. In other words, the determination unit 140 may calculate the three-dimensional position on the three-dimensional shape on an area-by-area basis instead of calculating the three-dimensional position on the three-dimensional shape on a pixel-by-pixel basis. Here, the area is a set of a predetermined number of pixels, for example, a block of a size with a predetermined number of vertical pixels and horizontal pixels.

[0120] The generating unit 150 generates a three-dimensional model of the subject based on the multiple frames acquired from the storage unit 120 by the acquiring unit 130, the camera parameters, and the search range. The generating unit 150 searches for a similar point similar to the first point on the first frame in a range corresponding to the search range on another frame (e.g., the second frame) different from the first frame. The generating unit 150 limits the length of the epipolar line corresponding to the first point in the second frame to a length according to the search range, and searches for a similar point similar to the first point on the epipolar line of the second frame. The generating unit 150 searches for a similar point from the second frame for each of multiple first pixels included in the first frame. As shown in the following formula 10, the generating unit 150 calculates the normalized cross correlation (NCC) between small regions as N(I, J) in a combination between the first frame and multiple frames other than the first frame, and generates matching information indicating the result of matching between the frames.

[0121] Here, the advantages of limiting the search range will be specifically described with reference to Fig. 9 and Fig. 10. Fig. 9 is a diagram for explaining the matching process when there is no limit to the search range. Fig. 10 is a diagram for explaining the matching process when there is a limit to the search range.

[0122] As shown in FIG. 9, when matching is performed between one pixel 572 of a first frame 571 and a frame 581 in an unrestricted search range R1, an epipolar line 582 corresponding to a straight line L1 connecting a first viewpoint V1 and the pixel 572 in the frame 581 exists from one end to the other of the frame 581. The first frame 571 is an image obtained at the first viewpoint V1, and the frame 581 is an image obtained at the second viewpoint V2. The straight line L1 coincides with the shooting direction of the camera 301 of the first viewpoint V1. The pixel 572 corresponds to a point 511 of the object 510. Therefore, a search for a pixel in the frame 581 similar to the pixel 572 is performed on the unrestricted epipolar line 582. Therefore, when there are two or more pixels on the epipolar line 582 that have characteristics similar to the pixel 572, a pixel 583 on the frame 581 that corresponds to a point 512 different from the point 511 of the subject 510 may be erroneously selected as a similar point. This reduces the accuracy of generating a three-dimensional model.

[0123] On the other hand, as shown in FIG. 10, the process by the determination unit 140 determines the search range R2 to be shorter than the search range R1 shown in FIG. 9. Therefore, for one pixel 572 of the first frame 571, matching is performed between the frame 581 and the limited search range R2, and in the frame 581, the epipolar line 584 corresponding to the straight line L1 connecting the first viewpoint V1 and the pixel 572 is shorter than the epipolar line 582 in accordance with the search range R2. Therefore, a search for a pixel in the frame 581 similar to the pixel 572 is performed on the epipolar line 584 that is shorter than the epipolar line 582. Therefore, it is possible to reduce pixels having characteristics similar to the pixel 572, and it is possible to increase the possibility of determining the pixel 585 corresponding to the point 511 of the subject 510 on the frame 581 as a similar point. Therefore, it is possible to improve the generation accuracy of the three-dimensional model. In addition, since the search range can be narrowed, it is possible to shorten the processing time related to the search.

[0124] The generation unit 150 generates a three-dimensional model by performing triangulation using the position and orientation of each camera 301 and the matching information. Note that matching may be performed for all combinations of two frames among the multiple frames.

[0125]

number

[0126] In addition, I xy and J xy are pixel values ​​in the small regions of frames I and J.

number

number

[0127] Then, the generation unit 150 generates a three-dimensional model using the search results in the matching, thereby generating a three-dimensional model including a plurality of three-dimensional points that are more numerous and denser than the plurality of three-dimensional points included in the map information.

[0128] The output unit 160 outputs the three-dimensional model generated by the generation unit 150. The output unit 160 includes, for example, a display device such as a display (not shown), and an antenna, a communication circuit, a connector, and the like for connecting to enable communication via wire or wirelessly. The output unit 160 outputs the integrated three-dimensional model to the display device, thereby causing the display device to display the three-dimensional model.

[0129] [Operation of the 3D model generator] Next, the operation of the three-dimensional model generating device 100 will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the operation of the three-dimensional model generating device 100.

[0130] First, in the three-dimensional model generating device 100, the receiving unit 110 receives a plurality of frames captured by the plurality of cameras 301, the camera parameters of each camera 301, and the map information from the estimation device 200 (S101). Step S101 is an example of a first acquisition step and a second acquisition step. Note that the receiving unit 110 does not have to receive the plurality of frames, the camera parameters, and the map information at one timing, and may receive each at different timings. In other words, the first acquisition step and the second acquisition step may be performed at the same timing or at different timings.

[0131] Next, the storage unit 120 stores the multiple frames captured by the multiple cameras 301, the camera parameters of each camera 301, and the map information received by the receiving unit 110 (S102).

[0132] Next, the acquisition unit 130 acquires the map information stored in the storage unit 120, and outputs the acquired map information to the determination unit 140 (S103).

[0133] The determining unit 140 determines a search range to be used for matching a plurality of points between a plurality of frames based on the map information acquired by the acquiring unit 130 (S104). Step S104 is an example of a determining step. Details of step S104 have been explained in the explanation of the process performed by the determining unit 140, and therefore will not be repeated here.

[0134] Next, the generating unit 150 searches for a similar point that is similar to the first point on the first frame in a range corresponding to the search range on the second frame (S105), and generates a three-dimensional model using the search result (S106). Step S105 is an example of a matching step. Step S106 is an example of a generating step. Details of steps S105 and S106 have been explained in the explanation of the processing performed by the generating unit 150, so they will not be explained here.

[0135] Then, the output unit 160 outputs the three-dimensional model generated by the generation unit 150 (S107).

[0136] [Effects, etc.] The three-dimensional model generation method of this embodiment includes a first acquisition step (S101) of acquiring map information obtained by camera calibration performed by having one or more cameras photograph a subject from multiple viewpoints and including multiple three-dimensional points each indicating a position on the subject in three-dimensional space, a second acquisition step (S101) of acquiring a first image at the first viewpoint and a second image at the second viewpoint, a determination step (S104) of determining a search range in three-dimensional space including a first three-dimensional point on the subject corresponding to a first point in the first image based on the map information, a matching step (S105) of searching for a similar point similar to the first point in a range corresponding to the search range on the second image, and a generation step (S106) of generating a three-dimensional model using the search results in the matching step.

[0137] According to the three-dimensional model generation method, a search range is determined based on map information, and a similar point similar to a first point on one of the first images is searched for in a range corresponding to the search range on the second image limited by the search range. In this way, the three-dimensional model generation method searches for similar points in a range where similar points are likely to exist based on map information, so that the accuracy of searching for similar points can be improved and the time required for the search process can be shortened. Therefore, the accuracy of generating the three-dimensional model can be improved and the processing time of the generation process of the three-dimensional model can be shortened.

[0138] Also, for example, the search range is a range in the shooting direction from a first viewpoint where one first image is shot. In the matching step (S105), the epipolar line corresponding to the first point in the second image is limited to a length according to the search range, and a similar point similar to the first point is searched for on the epipolar line of the second image.

[0139] According to this, a similar point that is similar to the first point is searched for on an epipolar line whose length is limited according to the search range, thereby improving the accuracy of searching for similar points and shortening the time required for the search process.

[0140] Also, for example, the determining step (S104) includes an interpolating step of using a plurality of 3D points included in the map information to interpolate a 3D point where the subject is estimated to exist between two points included in the plurality of 3D points. In the determining step, the search range is determined using the plurality of 3D points interpolated in the interpolating step.

[0141] According to this, a 3D point where the subject is estimated to exist is interpolated between two of the multiple 3D points, so that the search range can be determined with high accuracy.

[0142] Also, for example, the interpolation step interpolates a 3D point where the subject is estimated to be present in an area defined by three points included in the plurality of 3D points.

[0143] According to this, a 3D point where a subject is estimated to exist is interpolated in a region defined by three of a plurality of 3D points, so that the search range can be determined with high accuracy.

[0144] Also, for example, in the determination step (S104), for each of a plurality of regions on the projection frame in which a plurality of three-dimensional points are projected onto a first image, a three-dimensional position on the subject corresponding to the region is estimated based on the first viewpoint at which the first image was captured, and a distance image including the estimated three-dimensional position of each region is generated, and a search range is determined based on the three-dimensional position of each region in the distance image.

[0145] According to this, since the three-dimensional position is calculated for each of a plurality of regions on the projection frame, it is possible to calculate as many three-dimensional positions as are necessary for matching.

[0146] Also, for example, in the determination step (S104), the search range determined for each of a plurality of regions in the distance image is expanded as the estimation accuracy of the three-dimensional position of the corresponding region decreases.

[0147] According to this, the search range is widened as the accuracy of the three-dimensional position on the interpolated three-dimensional shape decreases, so that the search range can be determined according to the accuracy.

[0148] Also, for example, the estimation accuracy of the three-dimensional position is higher as the distance of the three-dimensional position from the first viewpoint decreases, so the search range can be determined according to the distance of the three-dimensional position from the first viewpoint.

[0149] Also, for example, the estimation accuracy of the three-dimensional position is higher as the three-dimensional position is closer to at least one of the multiple three-dimensional points. Therefore, the search range can be determined according to the distance of the three-dimensional position from at least one of the multiple three-dimensional points.

[0150] Also, for example, the estimation accuracy of the three-dimensional position is higher as the density of the multiple three-dimensional points in the area where the three-dimensional position exists is greater. Therefore, it is possible to determine a search range according to the density of the multiple three-dimensional points in the area where the three-dimensional position exists.

[0151] [Variations] In the 3D model generating method according to the above embodiment, a 3D point where a subject is estimated to exist between a plurality of 3D points is interpolated using a plurality of 3D points included in map information, but the interpolation may not be performed. Also, in the matching, a similar point of a first point in a first frame is searched for on an epipolar line in another frame, but the search may be performed without using an epipolar line.

[0152] For example, the three-dimensional model generating device may calculate a first three-dimensional point close to a first three-dimensional position on the subject corresponding to a first point on a first frame, project the first three-dimensional point onto another frame, and search for a similar point to the first point from a predetermined range including the projected first three-dimensional point. The predetermined range may be a range having a two-dimensional spread or a range having a one-dimensional length. If the predetermined range is a range having a one-dimensional length, it may be a range on an epipolar line.

[0153] (Other embodiments) Although the three-dimensional model generating method and the like according to the present disclosure have been described based on the above-mentioned embodiments, the present disclosure is not limited to the above-mentioned embodiments.

[0154] For example, in the above embodiment, each processing unit included in the three-dimensional model generating device or the like is described as being realized by a CPU and a control program. For example, each component of the processing unit may be composed of one or more electronic circuits. Each of the one or more electronic circuits may be a general-purpose circuit or a dedicated circuit. The one or more electronic circuits may include, for example, a semiconductor device, an IC (Integrated Circuit), or an LSI (Large Scale Integration). The IC or LSI may be integrated into one chip or into multiple chips. Here, the IC or LSI is referred to as an IC or LSI, but the name may change depending on the degree of integration, and may be called a system LSI, a VLSI (Very Large Scale Integration), or an ULSI (Ultra Large Scale Integration). In addition, a field programmable gate array (FPGA) that is programmed after the LSI is manufactured can also be used for the same purpose.

[0155] In addition, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, or a computer program. Alternatively, the present disclosure may be realized as a computer-readable non-transitory recording medium such as an optical disk, a hard disk drive (HDD), or a semiconductor memory in which the computer program is stored. Furthermore, the present disclosure may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0156] In addition, the present disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art may think of, and forms realized by arbitrarily combining the components and functions of the embodiments without departing from the spirit of the present disclosure. [Industrial Applicability]

[0157] The present disclosure can be applied to a three-dimensional model generation device or a three-dimensional model generation system, and can be applied to, for example, figure creation, terrain or building structure recognition, human behavior recognition, or free viewpoint video generation. [Explanation of symbols]

[0158] 100 Three-dimensional model generation device 110 Receiving unit 120 Storage section 130 Acquisition Department 140 Decision Section 150 Generation part 160 Output section 200 Estimation device 300 cameras 301 Camera 400 3D model generation system 500, 510 Subject 511, 512 points 520 Map Information 531, 532, 533, 581 frames 541, 542, 543 feature points 551, 561, 610, 710 Projection Frame 552, 611 mesh 562, 712 area 571 1st Frame 572, 583, 585 pixels 582, 584 Epipolar lines 601 3D point 612 sides 620, 720 range images 621, 721 Three-dimensional shape 622, 722 plane 701 3D point R1, R2 search range V1 First View V2 Second Viewpoint

Claims

1. A three-dimensional model generating method executed by an information processing device, comprising: a first acquisition step of acquiring map information obtained by camera calibration performed by having one or more cameras capture images of an object from multiple viewpoints, the map information including multiple three-dimensional points each indicating a position on the object in a three-dimensional space; a second acquisition step of acquiring a first image of the subject captured from a first viewpoint and a second image of the subject captured from a second viewpoint; a determining step of determining a search range in a three-dimensional space including a first three-dimensional point on the subject corresponding to a first point in the first image based on the plurality of three-dimensional points included in the map information; a matching step of searching for a similar point similar to the first point in a range corresponding to the search range on the second image; A generating step of generating a three-dimensional model using the search result in the matching step, The search range includes a three-dimensional position on the subject estimated from the map information. A method for generating three-dimensional models.

2. the search range is a range in an imaging direction from the first viewpoint, In the matching step, an epipolar line corresponding to the first point in the second image is limited to a length corresponding to the search range, and a similar point similar to the first point is searched for on the epipolar line in the second image. The three-dimensional model generating method according to claim 1 .

3. the determining step includes an interpolating step of, using the plurality of three-dimensional points included in the map information, interpolating a three-dimensional point at which the subject is estimated to exist between two points included in the plurality of three-dimensional points; In the determining step, the search range is determined using the plurality of three-dimensional points interpolated in the interpolating step. The three-dimensional model generating method according to claim 1 or 2.

4. The interpolation step includes interpolating a 3D point where the subject is estimated to exist in an area defined by three points included in the plurality of 3D points. The three-dimensional model generating method according to claim 3.

5. In the determining step, generating a distance image including the estimated three-dimensional positions of each region by estimating a three-dimensional position on the subject that corresponds to each region, the three-dimensional position being based on the first viewpoint, for each region on a projected image in which the three-dimensional points are projected onto the first image; The search range is determined based on the three-dimensional position of each region in the range image. The three-dimensional model generating method according to any one of claims 1 to 4.

6. In the determining step, the search range determined for each of the plurality of regions in the distance image is expanded as the estimation accuracy of the three-dimensional position of the corresponding region becomes lower. The three-dimensional model generating method according to claim 5.

7. The estimation accuracy of the three-dimensional position is higher as the distance of the three-dimensional position from the first viewpoint is smaller. The three-dimensional model generating method according to claim 6.

8. The estimation accuracy of the three-dimensional position is higher as the three-dimensional position is closer to at least one of the plurality of three-dimensional points. The three-dimensional model generating method according to claim 6 or 7.

9. The accuracy of estimating the three-dimensional position is higher as the density of the three-dimensional points in the area in which the three-dimensional position exists is higher. The three-dimensional model generating method according to any one of claims 6 to 8.

10. The camera calibration includes calculating positions and orientations of the one or more cameras; The first image and the second image are included in a plurality of images taken for the camera calibration. The three-dimensional model generating method according to any one of claims 1 to 9.

11. A first acquisition unit that acquires map information obtained by camera calibration performed by having one or more cameras capture an object from a plurality of viewpoints, and including a plurality of three-dimensional points each indicating a position on the object in a three-dimensional space; a second acquisition unit that acquires a first image of the subject captured from a first viewpoint and a second image of the subject captured from a second viewpoint; a determination unit that determines a search range in a three-dimensional space including a first three-dimensional point on the subject corresponding to a first point in the first image based on the plurality of three-dimensional points included in the map information; a matching unit that searches for a similar point that is similar to the first point in a range corresponding to the search range on the second image; a generation unit that generates a three-dimensional model using the search result in the matching unit, The search range includes a three-dimensional position on the subject estimated from the map information. 3D model generation device.

Images