3D model generating device, 3D model generating method and program

Abstract

To allow for generating a three-dimensional model regardless of the presence or absence of image data.SOLUTION: A three-dimensional model generation device is provided, comprising: an image data acquisition unit for acquiring image data of an object including time information; a three-dimensional model generation unit configured to generate a three-dimensional model based on the image data of the object acquired by the image data acquisition unit; and a three-dimensional model correction unit configured to estimate a degree of change in the object during a given time period with respect to the three-dimensional model to correct the three-dimensional model.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a three-dimensional model generating device, a three-dimensional model generating method, and a program. [Background technology]

[0002] A technology for easily searching for desired image data from multiple image data is described, for example, in Patent Document 1 below. The technology described in Patent Document 1 stores multiple image data, to which the location where the image was taken and the date and time of taking the image are added as shooting information, in an image exchange server, and searches for desired image data based on the shooting information added to the image data from a terminal device. In addition, a technology for photo mapping multiple images is described, for example, in Patent Document 2 below. The technology described in Patent Document 2 performs photo mapping by using meta information added to multiple photo data to perform organically utilized searches, searches for information about buildings within the field of view in real time, and searches for photos in which a specific person is the subject, using meta information added to multiple photo data. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2004-220420 A [Patent Document 2] JP 2009-176262 A Summary of the Invention [Problem to be solved by the invention]

[0004] There is a technology called photogrammetry that takes multiple photos of a subject while changing the shooting position, and generates a 3D model based on the data of the multiple photos. However, conventional photogrammetry generates a 3D model by processing multiple actual photo data. Therefore, if there is no actual photo data, it is not possible to generate a 3D model.

[0005] The present invention has been made in view of the above, and has an object to make it possible to generate a three-dimensional model regardless of the presence or absence of image data. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the objective, the 3D model generating device of the present invention comprises an image data acquisition unit that acquires image data of an object including time information, a 3D model generation unit that generates a 3D model based on the image data of the object acquired by the image data acquisition unit, and a 3D model correction unit that estimates the degree of change of the object at a predetermined time relative to the 3D model and corrects the 3D model.

[0007] The 3D model generation method of the present invention includes the steps of acquiring image data of an object including time information, generating a 3D model based on the acquired image data of the object, and correcting the 3D model by estimating a degree of change of the object at a predetermined time relative to the 3D model.

[0008] The program of the present invention causes a computer operating as a 3D model generating device to execute the following steps: acquiring image data of an object including time information; generating a 3D model based on the acquired image data of the object; and correcting the 3D model by estimating the degree of change of the object at a predetermined time with respect to the 3D model. Effect of the Invention

[0009] According to the present invention, an effect is achieved in that a three-dimensional model can be generated regardless of the presence or absence of image data. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an image acquisition device in a three-dimensional model generating device according to this embodiment. [Diagram 2]FIG. 2 is a schematic diagram showing a recording format for image data. [Diagram 3] FIG. 3 is a schematic diagram for explaining position information and orientation information for an object. [Figure 4] FIG. 4 is a block diagram showing a three-dimensional model generating device according to this embodiment. [Diagram 5] FIG. 5 is a flowchart showing a three-dimensional model generating method. [Figure 6] FIG. 6 is a block diagram showing a specific configuration of the three-dimensional model generating unit. [Figure 7] FIG. 7 is an explanatory diagram showing the positional relationship between two images to which the photogrammetry principle is applied. [Figure 8] FIG. 8 is an explanatory diagram showing the positional relationship between the two images. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of a 3D model generating device, a 3D model generating method, and a program according to the present invention will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the following embodiments.

[0012] [Image acquisition device] 1 is a block diagram showing an image acquisition device in a 3D model generation device according to this embodiment. The image acquisition device 10 generates format image data (described later) and transmits it to a database 26 in order to cause a 3D model generation device (described later) to generate a 3D model.

[0013] As shown in FIG. 1, the image acquisition device 10 includes a camera (imaging device) 11, a GPS receiver (GNSS signal receiver) 12, a direction sensor 13, a time information acquisition unit 21, a position information acquisition unit 22, a direction information acquisition unit 23, a format unit 24, and a memory unit 25.

[0014] Camera 11 photographs an object. The image data photographed by camera 11 is compressed into a format such as JPEG or PNG to an extent that degradation is not noticeable. Camera 11 sends the photographed image data to format unit 24, and also sends it as trigger information to time information acquisition unit 21, position information acquisition unit 22, and orientation information acquisition unit 23.

[0015] The GPS receiver 12 receives radio waves from satellites to obtain accurate time information and position information. The GPS receiver 12 sends the time information to a time information acquisition unit 21 and sends the position information to a position information acquisition unit 22.

[0016] The time information acquisition unit 21 acquires time information when the camera 11 captured the image data based on time information from the GPS receiving unit 12. The time information acquisition unit 21 sends the time information when the camera 11 captured the image data to the format unit.

[0017] The position information acquisition unit 22 acquires the position information when the camera 11 captured the image data based on the information from the GPS receiving unit 12. The position information acquisition unit 22 sends the position information when the camera 11 captured the image data to the format unit.

[0018] The orientation sensor 13 uses a gyro (angular velocity sensor) to measure the inclination (elevation angle) of the camera 11. The orientation sensor 13 sends the measurement result to the orientation information acquisition unit .

[0019] Based on the measurement result of the orientation sensor 13, the orientation information acquisition unit 23 sends to the format unit the orientation information when the camera 11 captured the image data.

[0020] The image data captured by the camera 11, the time information of the image data acquired by the time information acquisition unit 21, the position information of the image data acquired by the position information acquisition unit 22, and the orientation information of the image data acquired by the orientation information acquisition unit 23 are input to the format unit 24.

[0021] The format unit 24 is set with the number of bytes per parameter, converts various input information into a predetermined format, and generates formatted image data. The formatted image data is data obtained by converting image data, time information of the image data, position information of the image data, and orientation information of the image data into a predetermined format. The format unit 24 sends the formatted image data to the storage unit 25.

[0022] The time information acquisition unit 21, the position information acquisition unit 22, the direction information acquisition unit 23, and the format unit 24 are each configured by an arithmetic circuit such as a CPU (Central Processing Unit).

[0023] The storage unit 25 temporarily stores the formatted image data. The storage unit 25 is an external storage device such as an HDD (Hard Disk Drive) or a memory. The storage unit 25 transmits the formatted image data to a database 26 connected by a network such as the Internet via a transmission unit (not shown). The formatted image data temporarily stored in the storage unit 25 is saved on a large scale in the database 26 on the Internet.

[0024] FIG. 2 is a schematic diagram showing a recording format for image data.

[0025] As shown in FIG. 2, the format image data is composed of a fixed-length portion in the first half and a variable-length portion in the second half. The fixed-length portion has time information, location information, and direction information. The time information is year, month, date, and time information, and is composed of 2 bytes of year data, 1 byte of month data, 1 byte of day data, 1 byte of hour data, 1 byte of minute data, and 1 byte of second data. The variable-length portion is composed of image data. The location information is composed of 8 bytes of latitude data and 8 bytes of longitude data. The location information indicates the position where the object was photographed, and information other than latitude and longitude such as an address, area name, and building name can be used. The location information may be described as an area divided into a map of a predetermined size, and the relative coordinates of the horizontal and vertical axes within the area may be used.

[0026] The orientation information is composed of 8 bytes for horizontal orientation and 8 bytes for elevation angle data. The orientation information of the image data acquired by camera 11 is expressed, for example, using an analog clock, with east being 3 o'clock (90°), south being 6 o'clock (180°), west being 9 o'clock (270°), and north being 12 o'clock (360° (0°)) in a clockwise direction. The elevation angle orientation records the degree of the elevation angle looking up, with the horizontal being 0 degrees and directly above being 90 degrees. The variable length portion is image information, and the image information is a compressed image file in which the image is compressed into JPG or PNG format.

[0027] FIG. 3 is a schematic diagram for explaining position information and orientation information for an object.

[0028] As shown in Fig. 3, map 100 is a division of a city, and map 100 is divided into a number of areas, for example, 100m squares. Fig. 3 shows a state in which a specific section of map 100 is divided into nine areas: area 0010-2324, area 0010-2325, area 0010-2326, area 0010-2455, area 0010-2456, area 0010-2457, area 0010-2586, area 0010-2587, and area 0010-2588.

[0029] For example, the object T is located in an area with an area number of 0010-2456. The camera 11 captures images of the object T at five positions A1, A2, A3, A4, and A5. Position A1 is a front position relative to the object T, position A2 is a left diagonal front position relative to the object T, position A3 is a left diagonal rear position relative to the object T, position A4 is a right diagonal rear position relative to the object T, and position A5 is a right diagonal front position relative to the object T.

[0030] Here, the lines extending from positions A1, A2, A3, A4, and A5 toward the object T are the orientations. The orientation information is horizontal, with the direction directly above the map (due north) being the 12 o'clock direction (0 degrees). For example, the orientation of position A1 is 00 hours, 00 minutes, 00 seconds, and the orientation of position A2 is 02 hours, 30 minutes, 00 seconds. The elevation angle of camera 11, which cannot be shown on map 100, is written in the format as, for example, 12 degrees at position A1, 7 degrees at position A2, 10 degrees at position A3, 25 degrees at position A4, and 35 degrees at position A5.

[0031] [3D model generation device] FIG. 4 is a block diagram showing a three-dimensional model generating device according to this embodiment.

[0032] As shown in FIG. 4, the three-dimensional model generating device 30 includes an image processing device 31, an input device 32, and a display device 33.

[0033] The input device 32 is operated by a user. The input device 32 sends an operation command signal input by the user to the image processing device 31. Here, the operation command signal is a desired location (name and address of an object) for which the user wishes to generate a 3D model, a desired time for which the user wishes to generate a 3D model, etc. The input device 32 is, for example, a keyboard or a touch panel.

[0034] The image processing device 31 is connected to the database 26 via, for example, the Internet through a transmitting / receiving unit (not shown). The image processing device 31 acquires formatted image data stored in the database 26 based on an operation command signal sent from the input device 32, and performs image processing on the formatted image data. That is, the image processing device 31 acquires formatted image data corresponding to the object stored in the database 26 based on the object sent from the input device 32, and performs image processing on the formatted image data. The image processing device 31 sends the image-processed image data to the display device 33.

[0035] The display device 33 displays image data to the user. The display device 33 displays the three-dimensional model and image data processed by the image processing device 31. The display device 33 is, for example, a liquid crystal display or a head-mounted display.

[0036] The image processing device 31 has an image data acquisition unit 34, a three-dimensional model generation unit 35, and a three-dimensional model correction unit 36. The image data acquisition unit 34, the three-dimensional model generation unit 35, and the three-dimensional model correction unit 36 ​​are each configured by a CPU, for example.

[0037] The image data acquisition unit 34 receives an operation command signal input by the user through the input device 32. The image data acquisition unit 34 searches for and acquires a large number of corresponding image data based on the operation command signal. That is, the image data acquisition unit 34 acquires formatted image data of the object input through the input device 32 from the formatted image data stored in the database 26. The image data acquisition unit 34 sends the acquired large number of formatted image data to the 3D model generation unit 35.

[0038] The 3D model generating unit 35 generates a 3D model based on image data included in the multiple formatted image data acquired by the image data acquiring unit 34, time information when the image data included in this formatted image data was captured, position information of the image data, and orientation information of the image data. The 3D model generating unit 35 sends the generated 3D model to the 3D model correcting unit 36.

[0039] The 3D model correction unit 36 ​​corrects the 3D model generated by the 3D model generation unit 35 based on the time information input from the input device 32. That is, the 3D model correction unit 36 ​​estimates the time information input from the input device 32, that is, the degree of change of the object at a specified time, and corrects the 3D model. The 3D model correction unit 36 ​​sends the corrected 3D model to the display device 33.

[0040] At this time, the 3D model generating unit 35 performs averaging processing on the multiple image data having the same time information and orientation information, taking the luminance value of a pixel as the average of the luminance values ​​of the surrounding pixels, and generates a 3D model based on the multiple image data in different orientations that have been averaged. Then, the 3D model correcting unit 36 ​​corrects the 3D model based on the amount of change in the 3D model at different times.

[0041] [3D model generation method] FIG. 5 is a flowchart showing a three-dimensional model generating method.

[0042] As shown in FIG. 4 and FIG. 5, in step S11, the user inputs a desired location for which a 3D model is to be generated as an operation command signal, using the input device 32. The desired location is the name or address of an object. In step S12, the user inputs a desired time for which a 3D model is to be generated as an operation command signal, using the input device 32. The desired time is input in terms of the Gregorian calendar year and the year, month, and day. In step S13, the image data acquisition unit 34 searches and acquires a large number of format image data items corresponding to the desired location and the desired time as search conditions from the database 26. In this case, the image data acquisition unit 34 searches a predetermined period as the desired time. When, for example, July 1, 2000 is input as the desired time, the image data acquisition unit 34 acquires a plurality of format image data items taken during the predetermined period, that is, January 1, 1995 to July 1, 2005.

[0043] In step S14, the 3D model generation unit 35 generates a 3D model based on the image data included in the multiple format image data, the time information of the image data, the position information of the image data, and the orientation information of the image data. The details of the generation of the 3D model in step S14 will be described later. In step S15, the 3D model correction unit 36 ​​estimates the degree of change of the object at a predetermined time for the 3D model and corrects the 3D model. Then, in step S16, the display device 33 displays the corrected 3D model.

[0044] That is, first, the image data acquisition unit 34 acquires a plurality of format image data from the database 26 via the Internet, thereby acquiring a plurality of image data and time information, position information, and orientation information for each image data. The image data acquisition unit 34 acquires a plurality of image data photographed at a plurality of different times and from a plurality of different positions and orientations with respect to the object T.

[0045] Next, the 3D model generating unit 35 performs affine transformation with freedom of enlargement, reduction, rotation, left and right keystone correction, etc., on the multiple image data while maintaining the aspect ratio, and calculates the positioning so that the feature points of the images correspond, that is, by performing matching processing, the difference becomes extremely small. Left and right keystone correction is based on the principle of perspective, in which when a building or the like is photographed from the left side, the left vertical lines of the same height are close and therefore appear long, and the right vertical lines are farther away and therefore appear short. In the case of normal camera photography such as commemorative photos, the left and right directions can be moved probabilistically, so the number of images taken at many angles increases, but the vertical direction is only slightly changed by the elevation angle, and there is not much difference in the up and down of the shooting point. By extracting groups with matching major features, the connection of the images can be understood and the images are expanded in memory. These processes are a photogrammetry technique, which is a technology for generating a 3D shape of the object T from images from multiple angles, and is the principle of triangulation. In order to determine whether or not feature points can be obtained, it is preferable that there are two sets of images in the selected images whose fields of view overlap by about 60%.

[0046] Then, the 3D model correction unit 36 ​​estimates the degree of change in the 3D model generated by the 3D model generation unit 35 so as to correspond to the desired time, and corrects the 3D model. As a result, a 3D model of the object at a predetermined time desired by the user can be generated.

[0047] Here, a specific example of the method for generating a 3D model in step S14 will be described. Fig. 6 is a block diagram showing a specific configuration of a 3D model generating unit, Fig. 7 is an explanatory diagram showing the positional relationship between two images to which the photogrammetry principle is applied, and Fig. 8 is an explanatory diagram showing the positional relationship between the two images.

[0048] As shown in FIG. 6, the three-dimensional model generating unit 35 has an epipolar line direction calculator 41, an epipolar line orthogonal direction calculator 42, a search range determiner 43, a corresponding point detector 44, and a distance calculator 45.

[0049] The epipolar line direction calculator 41 calculates the direction of an epipolar line connecting corresponding pixel points of the image data of the object T based on the image data acquired by the image data acquisition unit 34. The epipolar line direction calculator 41 sends the calculated direction of the epipolar line to the epipolar line orthogonal direction calculator 42.

[0050] The epipolar line orthogonal direction calculator 42 calculates an orthogonal direction perpendicular to the epipolar line and outputs the calculated orthogonal direction relative to the epipolar line to the search range determiner 43.

[0051] The search range determiner 43 determines a two-dimensional search range on the screen so as to include a plurality of pixel points corresponding to each other in the direction of the epipolar line and in the direction perpendicular to the epipolar line. The search range determiner 43 outputs the determined two-dimensional search range to the corresponding point detector 44.

[0052] The corresponding point detector 44 performs a corresponding point search based on the multiple image data acquired by the image data acquisition unit 34 and the determined two-dimensional search range, and obtains a disparity vector. The corresponding point detector 44 sends the obtained disparity vector to the distance calculator 45.

[0053] The distance calculator 45 maps the disparity vector onto an epipolar line to obtain the epipolar line direction component of the disparity vector, and calculates the distance to the object T based on the obtained epipolar line direction component. The distance calculator 45 sends the calculated distance to the object T to the 3D model correction unit 36.

[0054] The generation of a three-dimensional model will be specifically explained below, but here, the generation of a three-dimensional model from two pieces of image data will be explained.

[0055] The image data acquisition unit 34 acquires two pieces of image data from the database 26. For example, as shown in FIG. 3, among the data registered and formatted in area 0010-2456, a photo image having common corresponding points is searched for, with one image as a reference and a second image selected at random as a pair. A three-dimensional point cloud is obtained by triangulating this pair of images, and the relative positions and the point cloud image are expanded by collecting these. In addition, the texture of the part corresponding to this image is associated with each other and managed in memory for later mapping.

[0056] First, two sets of image data are obtained for the object T by the camera 11A for the image of the field of view and the camera 11B for the image of the field of view (see FIG. 7 for both). Next, the corresponding point detector 44 searches for corresponding points of the feature points based on the two sets of image data. The corresponding point detector 44, for example, performs correspondence for each pixel and searches for the position where the difference is minimum. Here, as shown in FIG. 6 and FIG. 7, the cameras 11A and 11B, which are assumed to exist simultaneously at two viewpoints, are disposed in a relationship of Yl=Yr so that the optical axes Ol and Or are included on the same XZ coordinate plane. Using the corresponding points searched for by the corresponding point detector 44, a parallax vector corresponding to the angle difference for each pixel is calculated.

[0057] Since the obtained parallax vector corresponds to the distance from the cameras 11A and 11B in the depth direction, the distance calculator 45 calculates the distance in proportion to the magnitude of the parallax by the perspective law. If it is assumed that the photographer's cameras 11A and 11B only move substantially horizontally, the cameras 11A and 11B are arranged so that their optical axes Ol and Or are included in the same XZ coordinate plane, and the search for corresponding points can be performed only on the scanning lines that are the epipolar lines Epl and Epr. The distance calculator 45 generates a three-dimensional model of the object T using two pieces of image data of the object T and the respective distances from the cameras 11A and 11B to the object T.

[0058] On the other hand, when a point Ql (Xl, Yl) on the left image corresponds to a point Qr (Xr, Yr) on the right image, the disparity vector at the point Ql (Xl, Yl) is Vp (Xl-Xr, Yl-Yr). Here, since the two points Ql and Qr are on the same scanning line (epipolar line), Yl=Yr, and the disparity vector is expressed as Vp (Xl-Xr, 0). The epipolar line direction calculator 41 obtains such a disparity vector Vp for all pixel points on the image, and creates a group of disparity vectors to obtain information on the depth direction of the image. Incidentally, for a set in which the epipolar line is not horizontal, the height of one of the camera positions may be different (with a low probability). In this case, the epipolar line orthogonal direction calculator 42 searches within a rectangle in the epipolar line direction and in a direction orthogonal to the epipolar line, which is about the deviation from the horizontal, in a large search range, compared to a case where a corresponding point search is performed in a large two-dimensional area without considering corresponding point matching, so that the amount of calculation for the minimum rectangle is reduced and becomes rational. Then, as shown in FIG. 8, the search range determiner 43 shows a search range in which the epipolar line direction search range for the minimum rectangle is a~b=c~d, and the orthogonal direction search range is b~c=d~a. In this case, the search width in the epipolar line direction is ΔE, and the search width in the direction T orthogonal to the epipolar line is ΔT. The smallest non-inclined rectangle ABCD that includes the minimum inclined rectangle abcd is the area to be sought.

[0059] In this way, the three-dimensional model generating unit 35 obtains a parallax vector from the corresponding points of the feature points of the multiple cameras 11A and 11B under the epipolar constraint condition, obtains information on the depth direction of each point, maps the texture on the surface of the three-dimensional shape, and generates a three-dimensional model. As a result, the model of the part in the image data used for calculation can reproduce the space seen from the front hemisphere. Also, if there is a part not shown in the image data of the three-dimensional model, and if the line or surface of the surrounding texture is extended to connect, the space between them is interpolated using the same texture. The three-dimensional model generating unit 35 sends the generated three-dimensional model to the three-dimensional model correcting unit 36.

[0060] The method for generating the three-dimensional model is not limited to the above, and other methods may be used.

[0061] Then, when the 3D model correction unit 36 ​​corrects the 3D model generated by the 3D model generation to a 3D model for a time when no image data exists, it calculates the average value of the image data to estimate the degree of change of the object at a specified time when no image data exists, and corrects the 3D model.

[0062] That is, the multiple image data are image data taken during a predetermined period as a desired time (for example, from Jan. 1, 1995 to Jul. 1, 2005). The multiple image data are divided into groups by year, and the average value of the color difference signals of the image data in a predetermined object portion that constitutes the surface of the object that constitutes the target object T is calculated.

[0063] The 3D model correction unit 36 ​​judges whether image data at a desired time of the object for which the user wants to generate a 3D model is present in the multiple acquired image data. If multiple image data for, for example, July 1, 2000 exists as the desired time of the object for which the user wants to generate a 3D model, the 3D model correction unit 36 ​​does not correct the 3D model because the 3D model can be generated on July 1, 2000 based on the multiple image data acquired by the 3D model generation unit 35. In other words, if multiple image data for July 1, 2000 exists, the 3D model generation unit 35 calculates the average value of the multiple image data of the object T from the same time, the same position, and the same direction, and generates a 3D model using a color difference signal of the average value of the image data.

[0064] On the other hand, when there are no multiple image data on July 1, 2000 as the desired time of the object for which the user wants to generate a 3D model, the 3D model correction unit 36 ​​generates a 3D model at the desired time by correcting the 3D model generated by the 3D model generation unit 35 based on multiple image data that are not of the desired time. In this case, the 3D model correction unit 36 ​​corrects the 3D model based on multiple image data before June 30, 2000 and after July 2, 2000. The average value data of the color difference signals of the image data at this time is the average value of the image data in the future direction (before June 30, 2000) and the past direction (after July 2, 2000) of the desired time. The 3D model correction unit 36 ​​corrects the 3D model using the average value of the color difference signals of the image data from different times, the same position, and the same direction. In other words, the 3D model correction unit 36 ​​estimates and synthesizes values ​​obtained by interpolating multiple image data from different times as color difference signals.

[0065] Also, in the case of the image data in the past direction (before June 30, 2000) or the future direction (after July 2, 2000) where July 1, 2000, which is the desired time of the object for which the user wants to generate a 3D model, exists, the average value data of the color difference signal of the image data at this time is the average value of the image data before June 30, 2000, or the average value of the image data after July 2, 2000. The 3D model correction unit 36 ​​estimates and corrects the 3D model in the future direction or the past direction using the average value of the color difference signal of the image data from different times, the same position, and the same direction. That is, the 3D model correction unit 36 ​​estimates and synthesizes a value obtained by interpolating a plurality of image data from different times as a color difference signal. That is, the 3D model correction unit 36 ​​estimates and synthesizes a value obtained by extrapolating the image data based on the amount of change in the average value of the color difference signal every year as a color difference signal.

[0066] Furthermore, when dividing a day (24 hours) into hourly intervals and using color difference signals of image data in each hourly interval, the brightness changes depending on the height of the sun depending on the time, and the reason for removing the influence on the color difference signals is that the 3D model correction unit 36 ​​may obtain weather information from an external source and refer to the weather forecast for the land where the image data was taken, estimate the intensity of sunlight for each hour of the day, and correct the 3D model.

[0067] That is, when there are no multiple image data for July 1, 2000, which is the desired time of the object for which the user wants to generate a 3D model, the 3D model correction unit 36 ​​corrects the 3D model using the average value or amount of change of the image data before June 30, 2000, or the average value or amount of change of the image data after July 2, 2000. At this time, the 3D model correction unit 36 ​​estimates the weather at the desired time based on the weather before or after July 1, 2000, for example. Then, the 3D model correction unit 36 ​​corrects the 3D model using the average value of the image data of the same weather as the weather on July 1, 2000 from the multiple image data before June 30, 2000 and after July 2, 2000.

[0068] In addition, the 3D model correction unit 36 ​​may use image data from a portion of the time when image data exists to perform machine learning using a teacher dataset that takes as input the angle from the camera 11 to the object T at the desired time, July 1, 2000, the weather, and the season based on the date and time, as well as the photographed image data, and outputs corresponding exemplary photographed image data, thereby synthesizing image data of the object T at a specific time, before June 30, 2000 or after July 2, 2000, when no image data exists, using a predicted image.

[0069] In addition, image data used by the 3D model correction unit 36 ​​for correction may be image data that predicts a future object T, that is, future prediction image data. Future prediction image data includes space and image data of moving images such as movies, 3D space and image data created as a virtual future, etc. Image data of the entire configured space can be used, but it also includes partial limited spaces (embedded spaces). It is also possible to perform image processing to generate a composite image from the space and image data of the pre-development city to new city data by extrapolating or interpolating the space information, location information, and orientation information of the space, assuming urban redevelopment or rebuilding of a building. Future prediction image data also includes space and image data created mechanically (such as CAD data) or by artificial intelligence. If the region, building coverage ratio, type of building, etc. are used as input information, spatial image data of a standard structure can be obtained and used.

[0070] That is, the database 26 has not only image data of the object T from the past to the present, but also image data predicted for the future (future predicted image data). The future predicted image data is image data predicting the state of the object T from March 2, 2021 onward, relative to the present day of March 1, 2021. When the user uses the input device 32 to input a future time, for example, March 1, 2031, as the desired time, the image data acquisition unit 34 acquires not only image data before March 1, 2021, but also future predicted image data after March 2, 2021. The three-dimensional model correction unit 36 ​​corrects the three-dimensional model using image data before March 1, 2021 and future predicted image data after March 2, 2021.

[0071] [Effects of this embodiment] In this embodiment, the system includes an image data acquisition unit 34 that acquires image data of an object T including time information, a 3D model generation unit 35 that generates a 3D model based on the image data of the object T acquired by the image data acquisition unit 34, and a 3D model correction unit 36 ​​that estimates the degree of change of the object T at a predetermined time relative to the 3D model and corrects the 3D model.

[0072] Therefore, the 3D model generation unit 35 generates a 3D model based on multiple image data and time information for each image data, and the 3D model correction unit 36 ​​estimates the degree of change of the object T at a predetermined time relative to the 3D model and corrects the 3D model. That is, first, a large number of existing image data are acquired, and a 3D model for a predetermined time is generated from the acquired large number of image data. Next, based on the degree of change (degree of deterioration) of shape, color, etc. estimated from the time in the 3D model at multiple times, the 3D model is corrected to generate a 3D model for a time when no image data exists. As a result, a 3D model can be generated regardless of the presence or absence of image data.

[0073] In this embodiment, the image data of the object T includes position information and orientation information, and the 3D model generating unit 35 generates a 3D model based on the image data of a plurality of objects T whose time information, position information, and orientation information match. Therefore, it is possible to generate a 3D model of the object T at a given time with high accuracy.

[0074] In this embodiment, the 3D model correction unit 36 ​​corrects the 3D model based on the amount of change in the 3D model at different times. Therefore, even if image data at a desired time does not exist, a 3D model at the desired time without image data can be generated based on the amount of change in image data other than the desired time.

[0075] The three-dimensional model generating device according to the present invention has been described above, but the device may be embodied in various different forms other than the above-described embodiment.

[0076] Each component of the illustrated 3D model generating device is a functional concept, and does not necessarily have to be physically configured as illustrated. In other words, the specific form of each device is not limited to that shown in the figure, and all or part of the device may be functionally or physically distributed or integrated in any unit depending on the processing load and usage status of each device.

[0077] The configuration of the 3D model generating device is realized, for example, as software by a program loaded into a memory. In the above embodiment, the functional blocks are described as being realized by the cooperation of these hardware and software. In other words, these functional blocks can be realized in various forms by only hardware, only software, or a combination of both.

[0078] The above-mentioned components include those that a person skilled in the art can easily imagine and those that are substantially the same. Furthermore, the above-mentioned configurations can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the configurations are possible without departing from the scope of the present invention. [Explanation of symbols]

[0079] 10 Image acquisition device 11 Camera (imaging device) 12 GPS receiver 13 Orientation Sensor 21 Time information acquisition unit 22 Location information acquisition section 23 Direction information acquisition unit 24 Format section 25 Memory section 26 Database 30 3D model generation device 31 Image processing device 32 Input Devices 33 Display device 34 Image data acquisition unit 35 3D model generation unit 36 3D model correction section T object

Claims

1. an image data acquisition unit that acquires image data of an object including time information; a three-dimensional model generating unit that generates a three-dimensional model based on the image data of the object acquired by the image data acquiring unit; a three-dimensional model correction unit that estimates a degree of change of the object at a predetermined time with respect to the three-dimensional model and corrects the three-dimensional model; Equipped with The image data of the object includes position information and orientation information; the three-dimensional model generation unit generates a three-dimensional model based on image data of a plurality of objects whose time information, position information, and orientation information match each other; The three-dimensional model correction unit corrects the three-dimensional model based on weather information of a position where the image data of the object was captured. 3D model generation device.

2. The three-dimensional model correction unit corrects the three-dimensional model based on an amount of change in the three-dimensional model at different times. The three-dimensional model generating device according to claim 1 .

3. acquiring image data of an object including time information; generating a three-dimensional model based on the acquired image data of the object; a step of estimating a degree of change of the object at a predetermined time with respect to the three-dimensional model and correcting the three-dimensional model; Including, The image data of the object includes position information and orientation information; generating a three-dimensional model based on image data of a plurality of objects having the same time information, position information, and orientation information; correcting the three-dimensional model based on weather information for a location where the image data of the object was captured; A method for generating 3D models.

4. acquiring image data of an object including time information; generating a three-dimensional model based on the acquired image data of the object; a step of estimating a degree of change of the object at a predetermined time with respect to the three-dimensional model and correcting the three-dimensional model; the image data of the object includes position information and orientation information, generating a three-dimensional model based on image data of a plurality of objects whose time information, position information, and orientation information match, and correcting the three-dimensional model based on weather information at the location where the image data of the object was captured; A program for causing a computer operating as a three-dimensional model generating device to execute the above.

Images