Coordinate identifying device, coordinate identifying method and program

Abstract

To facilitate an estimation for an absolute imaging position of an image of a site.SOLUTION: A coordinate identifying device 100 includes: a relative position estimating unit 14 that estimates a relative imaging position that is not in an actual scale for an image group including a reference image which has an absolute imaging position identified in an actual scale, and a non-reference image (the image of a site) which has an absolute imaging position not identified; and an absolute position estimating unit 15 that estimates the absolute imaging position of the non-reference image using the relative imaging position and the absolute imaging position of the reference image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a coordinate specifying device, a coordinate specifying method, and a program. [Background technology]

[0002] In various plants and other facilities, it is common to process information obtained from cameras and scanners according to respective purposes in order to monitor and inspect the various equipment installed in the facility, and then reflect the results in monitoring, control, etc.

[0003] For example, at nuclear power plants, on-site workers inspect the piping and other equipment installed at the power plant to improve safety and check for differences between design and construction, deterioration of the equipment, etc. However, in order to carry out such work, workers require a lot of inspection time, including traveling to the site and going through entry procedures. In response to this, from the perspective of ensuring plant safety and cybersecurity, technologies are being proposed that enable remote inspection in real time using cameras, IoT (Internet of Things) devices, and wireless communication.

[0004] For example, as this type of technology, a system has been proposed that uses point clouds and images measured by laser scanners, pre-designed CAD (Computer Aided Design) models, etc. to virtually inspect a site in a VR (Virtual Reality) space. Such a system is considered to be effective in reducing the amount of inspection work required.

[0005] In relation to this type of technology, for example, Patent Document 1 describes a technology for estimating and recording the photographing position of an on-site image (inspection image) acquired from a camera or the like in order to improve the efficiency of recording the results of equipment inspection. The technology described in Patent Document 1 can estimate the photographing position of the on-site image (inspection image), so that it is possible to realize a virtual on-site inspection by arranging the inspection image in a VR space using the photographing position information. In order to estimate the absolute photographing position of the on-site image (inspection image), it is preferable to include at least two or more feature points having three-dimensional coordinates in each image. Such feature points can be acquired by attaching a large number of markers to the site and photographing an image so that each marker is captured, or by matching an image acquired using a laser scanner or the like with the inspection image. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2022-137946 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, the conventional technology described in Patent Document 1 has the problem that in order to estimate the absolute shooting positions of all site images (inspection images), a large number of markers must be affixed to the site or distances at the site must be measured using a laser scanner, etc., which requires a great deal of preparation work.

[0008] The present invention has been made to solve the above-mentioned problems, and its main object is to provide a coordinate determination device, a coordinate determination method, and a program that easily estimate the absolute shooting position of an image at a site. [Means for solving the problem]

[0009] In order to achieve the above-mentioned object, the present invention provides a coordinate identification device that includes a relative position estimation unit that estimates a relative shooting position that is not at real scale for a group of images including a reference image whose absolute shooting position at real scale has been identified and a non-reference image whose absolute shooting position has not been identified, and an absolute position estimation unit that estimates the absolute shooting position of the non-reference image using the relative shooting position and the absolute shooting position of the reference image. Other means will be described later. Effect of the Invention

[0010] According to the present invention, it is possible to easily estimate the absolute photographing position of an on-site image, which is a non-reference image. [Brief description of the drawings]

[0011] [Figure 1] 1 is an overall configuration diagram of a coordinate specifying device according to an embodiment; [Figure 2A] FIG. 4 is an explanatory diagram of a reference image database. [Figure 2B] FIG. 4 is an explanatory diagram of a reference image photographing position database. [Figure 2C] FIG. 2 is an explanatory diagram of an image database. [Figure 2D] FIG. 4 is an explanatory diagram of an image relative position database. [Figure 2E] FIG. 4 is an explanatory diagram of an image absolute position database. [Figure 2F] FIG. 1 is an explanatory diagram of a CAD model database. [Figure 2G] FIG. 13 is an explanatory diagram of an image-CAD correspondence relation database. [Diagram 3] 13 is a flowchart showing an operation of an acquisition unit. [Figure 4A] 1 is an explanatory diagram of an image (inspection image) of a site where a measurement position is defined by an acquisition unit. [Figure 4B] FIG. [Diagram 5] 13 is a flowchart showing an operation of a reference image creation unit. [Figure 6A]4 is an explanatory diagram of a reference image acquired in a reference image creation unit; FIG. [Figure 6B] FIG. 13 is an explanatory diagram of an image tilt θ. [Figure 7A] FIG. 1 is an explanatory diagram (1) of the bottom image of a cubic panoramic image. [Figure 7B] FIG. 2 is an explanatory diagram (2) of the bottom image of a cubic panoramic image. [Figure 7C] FIG. 3 is an explanatory diagram of the bottom image of a cubic panoramic image. [Figure 8] 13 is a flowchart showing an operation of a relative position estimation unit. [Figure 9] FIG. 11 is an explanatory diagram of an estimation result of a relative position. [Figure 10] 13 is a flowchart showing an operation of an absolute position estimation unit. [Figure 11] FIG. 4 is an explanatory diagram of the relationship between a relative position and an absolute position. [Figure 12] FIG. 11 is an explanatory diagram of an absolute position estimation result. [Figure 13A] FIG. 1 is an explanatory diagram of the operation of the position correction unit; [Figure 13B] FIG. 13 is an explanatory diagram (2) of the operation of the position correction unit. [Figure 14] 13 is a flowchart showing an operation of an attribute providing unit. [Figure 15A] FIG. 1 is an explanatory diagram (1) of rays generated in the attribute assignment unit. [Figure 15B] FIG. 2 is an explanatory diagram (2) of rays generated in the attribute assignment unit. [Figure 16] 1 is a flowchart (1) showing the operation of the image display unit. [Figure 17] 13 is a flowchart (2) showing the operation of the image display unit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention (hereinafter referred to as "the present embodiment") will be described in detail with reference to the drawings. Note that each drawing is merely a schematic illustration to allow a sufficient understanding of the present invention. Therefore, the present invention is not limited to only the illustrated examples. In addition, in each drawing, common or similar components are given the same reference numerals, and duplicated explanations thereof will be omitted.

[0013] <Configuration of coordinate determination device 100> Hereinafter, the configuration of the coordinate specifying device 100 according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing the configuration of the coordinate specifying device 100 according to this embodiment. The coordinate specifying device 100 is a device that estimates an absolute shooting position in a non-reference image (image of the site) from a reference image in which an absolute shooting position in real scale is specified, and a non-reference image (image of the site) in which an absolute shooting position is not specified.

[0014] In this embodiment, the coordinate specifying device 100 is assumed to be configured as an inspection support device that supports the monitoring and inspection of various facilities (for example, equipment such as piping in a nuclear power plant). The inspection support device generates support information used for monitoring and inspecting the facility using image information acquired by the acquisition device 50 and various input information input via the input device 31.

[0015] The coordinate specifying device 100 is configured by, for example, a personal computer or a server. As shown in Fig. 1, the coordinate specifying device 100 is connected to an input device 31 such as a keyboard or a mouse, a display device 32 such as a display, and an acquisition device 50 such as a camera 51 or a laser scanner 52. The camera 51 captures images or videos of the plant site. The laser scanner 52 captures, for example, a 360° video around the plant site and measures the distance to an arbitrary point.

[0016] Moreover, the coordinate specifying device 100 internally has a control unit 10 and a storage unit 20. The control unit 10 has an acquisition unit 11, a reference image creation unit 12, a data input unit 13, a relative position estimation unit 14, an absolute position estimation unit 15, a position correction unit 16, an attribute assignment unit 17, and an image display unit 18.

[0017] The acquisition unit 11 is a component that acquires various types of data. The reference image creating unit 12 is a component that creates a reference image. The data input unit 13 is a component that accepts input of various types of information.

[0018] The relative position estimation unit 14 is a component that estimates a relative shooting position that is not on a real scale. In this embodiment, the relative position estimation unit 14 estimates a relative shooting position that is not on a real scale for an image group that includes a reference image in which an absolute shooting position on a real scale is specified and a non-reference image (image of the site) in which an absolute shooting position is not specified.

[0019] The absolute position estimation unit 15 is a component that estimates the absolute shooting position of a non-reference image (image of the site). In this embodiment, the absolute position estimation unit 15 estimates the absolute shooting position of the non-reference image by using the relative shooting position and the absolute shooting position of the reference image.

[0020] The position correction unit 16 is a component that corrects at least one of the relative position and absolute position information using an input from a user. The attribute assigning unit 17 is a component that assigns attribute information of a CAD model to either or both of a reference image and a non-reference image. In this embodiment, the attribute assigning unit 17 assigns attribute information of a CAD model to either or both of a reference image and a non-reference image based on the absolute shooting position and the CAD model. The image display unit 18 is a component that causes the display device 32 to display various images.

[0021] In addition, the relative position estimation unit 14 and the absolute position estimation unit 15 are preferably configured to generate information representing the shooting position so as to include at least one of information on two-dimensional coordinates parallel to the ground surface and three-dimensional coordinates including the vertical direction, and information specifying the direction within the image.

[0022] In addition, the attribute assignment unit 17 may be configured to identify equipment appearing in the non-reference images by associating the non-reference images, whose absolute shooting positions have been estimated by the absolute position estimation unit 15, with the CAD model.

[0023] Further, a reference image database DB1, a reference image photographing position database DB2, an image database DB3, an image relative position database DB4, an image absolute position database DB5, a CAD model database DB6, an image-CAD correspondence database DB7, and a control program Pr are stored in the storage unit 20. Regarding the information stored in the storage unit 20, the coordinate specifying device 100 only needs to be able to utilize the information stored in the storage unit 20, and each piece of information may exist in a device external to the coordinate specifying device 100.

[0024] Fig. 2A is an explanatory diagram of the reference image database DB1. The reference image database DB1 is a database that stores images that serve as references for estimating the absolute position of an image. In the example shown in Fig. 2A, the reference image database DB1 includes number information and a reference image corresponding to the number information.

[0025] Fig. 2B is an explanatory diagram of the reference image shooting position database DB2. The reference image shooting position database DB2 is a database that stores three-dimensional coordinates of the shooting positions of the reference images. In the example shown in Fig. 2B, the reference image shooting position database DB2 includes a reference image shooting position ID and absolute coordinates corresponding to the reference image shooting position ID.

[0026] Fig. 2C is an explanatory diagram of the image database DB3. The image database DB3 is a database that stores images of the site captured by the camera 51 and the laser scanner 52. Hereinafter, the "images of the site" will be described as images of objects to be monitored, inspected, etc. Hereinafter, the "images of the site" may be referred to as "inspection images." In the example shown in Fig. 2C, the image database DB3 includes number information and captured images (non-reference images) of the site corresponding to the number information.

[0027] FIG. 2D is an explanatory diagram of the image relative position database DB4. The image relative position database DB4 is a database that stores the relative positions between images (inspection images) of the site. In the example shown in FIG. 2D, the image relative position database DB4 includes an SfM_ID, an image ID, relative coordinates, and a relative orientation. Here, "SfM_ID" represents an ID of an SfM (Structure from Motion) process. Furthermore, "relative coordinates" represent coordinates of a virtually defined relative coordinate system. Furthermore, "relative orientation" represents a relative orientation with respect to a reference direction (see FIG. 6B).

[0028] Fig. 2E is an explanatory diagram of the image absolute position database DB5. The image absolute position database DB5 is a database that stores the positions of on-site images (inspection images) in the plant coordinate system. In the example shown in Fig. 2E, the image absolute position database DB5 includes an image ID, absolute coordinates, and a direction (tilt θ (Fig. 6B)).

[0029] FIG. 2F is an explanatory diagram of the CAD model database DB6. The CAD model database DB6 is a database that stores CAD model information. In the example shown in FIG. 2F, the CAD model database DB6 includes number information and CAD drawings (drawing images generated by CAD) corresponding to the number information. Note that FIG. 2F shows a floor plan (CAD drawing) of the site for inspection work inside a nuclear power plant building.

[0030] Fig. 2G is an explanatory diagram of the image-CAD correspondence database DB7. The image-CAD correspondence database DB7 is a database that stores image-CAD model correspondence information that indicates the correspondence between an image and a CAD model. In the example shown in Fig. 2G, the image-CAD correspondence database DB7 includes an image ID, a photographed image, and a CAD object ID. Here, the "CAD object ID" refers to an object (e.g., equipment such as piping) drawn on a CAD drawing.

[0031] The control program Pr is a program that causes the control unit 10 to function as an acquisition unit 11, a reference image creation unit 12, a data input unit 13, a relative position estimation unit 14, an absolute position estimation unit 15, a position correction unit 16, an attribute assignment unit 17, and an image display unit 18. The control program Pr is stored in a storage medium 9, and is installed in the system control unit 80 from the storage medium 9 directly or indirectly.

[0032] <Operation of Acquisition Unit 11> The operation of the acquisition unit 11 (FIG. 1) will be described below. The acquisition unit 11 acquires images and point clouds of a site that is a target of monitoring or inspection by the acquisition device 50. Here, the "images of the site" refers to image data (pixel photo image data) or video data captured by the camera 51 or the laser scanner 52. In this embodiment, the camera 51 is assumed to be a 360° camera. However, the camera 51 may be a camera built into a device such as a general digital camera, a smartphone, or a tablet device. In addition, the "point cloud" is a collection of three-dimensional point data measured by the laser scanner 52. The "point cloud" includes three-dimensional position information of each point. In this embodiment, the laser scanner 52 is assumed to be a stationary laser scanner that is installed at the site using a tripod. However, the laser scanner 52 may be a handheld laser scanner that is held in the hand to measure distances. Here, the camera 51 and the laser scanner 52 are assumed to capture video of the site.

[0033] FIG. 3 is a flowchart showing the operation of the acquisition unit 11. As shown in FIG. 3, first, the acquisition unit 11 specifies (sets) a measurement position (step S105). Here, the "measurement position" means a photographing position (reference image photographing position) of a reference image serving as a reference for absolute position estimation, and a photographing position (inspection image photographing position) of a non-reference image (image of the site (inspection image)) to be subjected to absolute position estimation. In step S105, the acquisition unit 11 acquires, for example, a CAD drawing of the site from the CAD model database DB6 (FIG. 2F), and specifies (sets) a measurement position in the CAD drawing. Note that the reference image photographing position is a position photographed in advance by the camera 51 or the laser scanner 52 so that absolute coordinates can be estimated when photographing the non-reference image (image of the site). Here, the acquisition unit 11 may attach a marker to the floor or wall of the plant instead of the photographing position of the camera 51 or the laser scanner 52 as the reference image photographing position, and use the position where the marker is attached as a substitute. The shooting positions of the non-reference images (images of the site) can be set to the shooting positions of the whole room including the shooting position of the reference image, and the shooting positions of the images or videos taken by the camera 51 or the laser scanner 52.

[0034] Here, the acquisition unit 11 may be configured to select the measurement points of the laser scanner 52 from among the four corners of the room or the corners of the walking path. For example, as shown in FIG. 4A, in step S105, the acquisition unit 11 defines (sets) four reference image shooting positions BP1 to BP4 as measurement positions for the CAD drawing of the site acquired from the CAD model database DB6 (FIG. 2F). FIG. 4A is an explanatory diagram of an image (inspection image) of the site in which the measurement positions are defined by the acquisition unit 11. FIG. 4A shows the configuration of a room in a certain facility as seen from above. In the example shown in FIG. 4A, the room is configured such that only the left part at one place on the front side is open, and the internal space is divided into the front side and the back side by a wall 61. In addition, in the example shown in FIG. 4A, the four reference image shooting positions BP1 to BP4 are indicated by the symbol "x". Each of the reference image shooting positions BP1 to BP4 is a measurement position in step S105 of FIG. 3.

[0035] In step S105 (measurement position definition) in FIG. 3, the acquisition unit 11 displays the CAD drawing of the site shown in FIG. 4A on the display device 32, for example, using CAD model information in the CAD model database DB6 (FIG. 2F). As a result, the coordinate specifying device 100 presents the image of the site (inspection image) to the user. The user operates the mouse of the input device 31 or the like on the CAD drawing of the site to input (specify) the reference image shooting positions (four reference image shooting positions BP1 to BP4 in the example shown in FIG. 4A). This input (specification) is performed, for example, by the user operating the mouse provided as the input device 31 on the image of the site (inspection image) to click on any position. In this embodiment, four reference image shooting positions BP1 to BP4 are specified, but any number of points may be specified as long as they are at least two or more. The reference image shooting positions BP1 to BP4 can be input to the coordinate specifying device 100 through the CAD drawing. In this embodiment, the three-dimensional coordinate values ​​(position information) of the reference image shooting positions BP1 to BP4 will be described as being stored in a reference image shooting position database DB2 (FIG. 2B).

[0036] Returning to FIG. 3, next, the acquisition unit 11 acquires an image (inspection image) of the site as a reference image, which is taken by the camera 51 or the laser scanner 52 while the user is moving as shown in FIG. 4B (step S110). In step S110, the acquisition unit 11 acquires an image of the site taken by the camera 51 or the laser scanner 52 at a position of the plant site corresponding to a position stored in the reference image taking position database DB2, and specifies (sets) a measurement position of the site. The measurement position of the site can be specified by measuring a distance from a wall using a tape measure or the like. In addition, the measurement position of the site may be determined by attaching an AR marker (i.e., an image or illustration that is a starting point for displaying AR (Augmented Reality) content by reading it) to the reference image taking position on the floor or wall of the plant, instead of the taking position of the camera 51 or the taking position of the laser scanner 52, and using the position where the AR marker is attached as a substitute.

[0037] FIG. 4B is an explanatory diagram of a moving route. FIG. 4B shows an example of a route taken by a user while moving, using the camera 51 to capture images (inspection images) of a site. In the example shown in FIG. 4B, a route is defined (set) that passes through the reference image capturing positions BP1 to BP4 in order. In this embodiment, the camera 51 is configured as a 360° camera, and a description is given assuming a case where the user captures a video of the site with the camera 51 while walking along the route shown in FIG. 4B. However, the camera 51 may be configured as a digital camera, and the user may intermittently capture a plurality of images (still images) of the site with the camera 51 while walking along the route. In addition, the user defines (sets) the route so that the route passes through at least two or more reference image capturing positions. The three-dimensional coordinates of the route are not used in this embodiment. Therefore, in step S110, the definition (setting) of the route may be defined (set) by mouse operation on the captured image of the CAD site, similar to the mouse operation performed on the CAD drawing, which is a two-dimensional image, in step S105 (measurement position definition) of FIG. 3. Alternatively, the user may separately define (set) it in any manner.

[0038] Returning to FIG. 3, next, similarly to step S110, the acquisition unit 11 acquires an image (inspection image) of the site, which is a non-reference image, captured by the camera 51 or the laser scanner 52 while the user is moving as shown in FIG. 4B (step S115). The image acquired in step S115 (acquisition of an image (inspection image) of the site) is a non-reference image whose absolute shooting position is not specified. In step S115, the acquisition unit 11 captures an image (inspection image) of the site with the camera 51 while the user is walking along the route shown in FIG. 4B. Here, the camera 51 is configured as a 360° camera, and is described as capturing a 360° video of the site. The acquisition unit 11 acquires the image (inspection image) of the site from the camera 51, and stores it in the image database DB3 (FIG. 2C) as a non-reference image.

[0039] <Operation of the reference image creation unit 12> The operation of the reference image creation unit 12 (FIG. 1) will be described below. FIG.

[0040] As shown in Fig. 5, first, the reference image creation unit 12 selects one reference image captured by the acquisition unit 11 (step S205). After step S205, it is determined whether the selected reference image was captured at a specified position (step S210). This determination is made by the user. Then, the user inputs the determination result to the coordinate specifying device 100 using the input device 31.

[0041] If it is determined in step S210 that the selected reference image is photographed at a specified position (if "Yes"), the process proceeds to step S215. In this case, the reference image creation unit 12 stores the selected reference image in the reference image database DB1 (FIG. 2A) (step S215). Then, the reference image creation unit 12 stores the three-dimensional coordinates of the photographing position of each image and the photographing direction (FIG. 6A) in the reference image photographing position database DB2 (FIG. 2B). Here, the three-dimensional coordinates of the photographing position are stored as is as the specified value stored in the reference image photographing position database DB2 (FIG. 2B) used at the time of photographing, and the direction is stored as the direction (angle) of the camera 51 at the time of photographing the reference image when a previously determined direction is set to 0°. The three-dimensional coordinates of the photographing position are input by the user using the input device 31 after measuring the angle at the time of photographing the reference image. After step S215, the process proceeds to step S280.

[0042] FIG. 6A is an explanatory diagram of a reference image acquired by the reference image creation unit 12. FIG. 6A shows the position and orientation of the reference image determined. FIG. 6A shows the photographing position and orientation of the photographer 71 at the reference image photographing positions BP1 to BP4 determined by the reference image creation unit 12 with triangular marks. The position and orientation data are represented in the image absolute position database DB5 (FIG. 2E). In the example shown in FIG. 6A, the reference image photographing positions BP1 to BP4 are indicated by the symbol "x". In FIG. 6A, the upward direction is set as a predetermined reference direction. The inclination θ of the image absolute position database DB5 shown in FIG. 2E is that shown in FIG. 6B. FIG. 6B is an explanatory diagram of the image inclination θ. As shown in FIG. 6B, the image inclination θ indicates the inclination of the image with respect to the reference direction.

[0043] Returning to Fig. 5, if the determination in step S210 above indicates that the selected reference image was not captured at the specified position (if "No"), the process proceeds to step S220. In this case, it is determined whether the selected reference image is an image measured by the laser scanner 52 (step S220). This determination is also made by the user. Then, the user inputs the determination result to the coordinate identifying device 100 using the input device 31.

[0044] If it is determined in step S220 that the selected reference image is an image measured by the laser scanner 52 (if "Yes"), the process proceeds to step S225. In this case, the reference image creation unit 12 selects all point clouds measured as the reference image and performs registration of the point clouds (step S225). Here, "registration" refers to a process of determining the photographing positions of multiple point clouds and arranging the point clouds in the same coordinate system. The reference image creation unit 12 can perform registration by using software attached to the laser scanner 52, for example.

[0045] After step S225, the reference image creation unit 12 aligns the CAD model with the point cloud (step S230). The alignment in step S230 can be performed by using a method such as the well-known ICP (Iterative Closest Point). The reason for performing the alignment is to calculate the photographing position and orientation of the reference image photographed by the laser scanner 52 at the same time as the point cloud by arranging the point cloud in the coordinate system of the CAD model.

[0046] After step S230, the reference image creation unit 12 acquires the coordinates of the measurement positions of each point cloud (step S235). The coordinates of the measurement positions of each point cloud are included in point cloud data in a known format such as the PTX file format or E57 file format, and therefore can be easily acquired.

[0047] After step S235, the reference image creation unit 12 acquires the orientation of the image captured during each point cloud measurement (step S240). When the laser scanner 52 is a stationary laser scanner, the laser scanner 52 captures a 360° video. The reference image creation unit 12 cuts out an image from each frame of the 360° video captured by the laser scanner 52. At this time, an image cut out directly from the 360° video is distorted by equirectangular projection. Therefore, the reference image creation unit 12 converts the cut out image into a cubic panoramic image to correct the image distortion. The conversion can be performed, for example, using a known software such as HDRShop or other various software. Then, the reference image creation unit 12 determines the angle difference between the direction indicated by the pixel (picture element) in the leftmost column on the cubic panoramic image and a reference direction determined in advance. At this time, the angle is determined by taking the reference direction as 0° and the clockwise direction as the positive direction. The angle can be determined by extracting a 3D point cloud corresponding to the pixels in the leftmost column from the PTX file format data or the E57 file format data, and determining how many degrees the straight line connecting the absolute coordinates of the points in the 3D point cloud and the absolute coordinates of the measurement position is tilted from the reference direction. The reference image creation unit 12 cuts the cubic panoramic image into six images with a viewing angle of 60°, and stores them in the image database DB3 (Fig. 2C).

[0048] After step S240, the reference image creation unit 12 sets all images acquired in step S225 as reference images, and registers the reference images in a reference image database DB1 (FIG. 2A) (step S245). Furthermore, the reference image creation unit 12 registers the absolute positions and orientations of the reference images acquired in steps S235 and S240 in a reference image shooting position database DB2 (FIG. 2B).

[0049] If it is determined in the above-mentioned step S220 that the selected reference image is not an image measured by the laser scanner 52 ("No"), the process proceeds to step S250. In this case, it is determined whether or not the AR marker has been affixed to a floor or a wall and a video has been captured (step S250). This determination is also made by the user. Then, the user inputs the determination result to the coordinate identifying device 100 using the input device 31.

[0050] In the determination of step S250, if an AR marker is affixed to a floor or wall without taking a reference image using the camera 51 or measuring a reference image using the laser scanner 52 in step S110, the result is determined as "Yes." In the determination of step S250, if it is determined that an AR marker is affixed to a floor or wall and a video is taken (if "Yes"), the process proceeds to step S255. In this case, the reference image creation unit 12 first cuts out images from each frame of the 360° video acquired in step S115 shown in FIG. 3 (step S255). At this time, since the image cut out directly from the 360° video is distorted due to equirectangular projection, the image is converted into a cubic panoramic image to correct the distortion.

[0051] After step S255, the reference image creation unit 12 recognizes the AR marker in the cubic panoramic image of each frame (step S260). At this time, the reference image creation unit 12 inputs each of six images corresponding to each face of the cubic panoramic image and performs the AR marker recognition process using, for example, the well-known open source software OpenCV and the AR marker ArUco. The coordinate identification device 100 can determine the pixel position on the image of the AR marker included in each image and the three-dimensional position in the coordinate system of the camera 51.

[0052] After step S260, the reference image creation unit 12 judges the suitability of the reference image (step S265). The judgment of the suitability of the reference image in step S265 is performed by using the position information of the AR marker judged in step S260 to judge the timing at which the AR marker came closest to the camera 51. When an AR marker is affixed to a floor surface and a 360° video is shot so as to pass over the AR marker, the reference image creation unit 12 creates a bottom image of a cubic panoramic image and detects a frame in which the AR marker is at the center of the bottom image, thereby making it possible to judge the suitability of the reference image in step S265.

[0053] However, in step S265, if the user holds the camera 51 (360° camera) above his head to capture a video, the user (photographer) directly below the camera 51 (360° camera) will be in the blind spot and the AR marker will not be captured. Therefore, in this case, the reference image creation unit 12 performs a process of detecting frames before and after a frame in which the AR marker is located at the center. This will be described with reference to Figures 7A to 7C. Figures 7A to 7C are explanatory diagrams of bottom images of a cubic panoramic image.

[0054] FIG. 7A shows an example of a case where an AR marker (marker 81) is attached to the floor surface, and a user holds a camera 51 (360° camera) above the marker 81 to shoot a video. In the example shown in FIG. 7A, the marker 81 is attached above a reference image shooting position BP1. Note that FIG. 7A omits the markers 81 corresponding to the reference image shooting positions BP2 to BP4 shown in FIG. 4A. The user moves in the order of a first position Pa, a second position Pb, a third position Pc, a fourth position Pd, and a fifth position Pe. The first position Pa and the second position Pb are located in front of the marker 81. The third position Pc is located directly above the marker 81. The fourth position Pd and the fifth position Pe are located behind the marker 81. In FIG. 7A, the marker 81 includes a QR code (registered trademark). The reference image creation unit 12 can read the QR code (registered trademark) of the marker 81 to obtain attribute information related to the reference image photographing position BP1.

[0055] Fig. 7B shows a blind spot formed when a user holds camera 51 (360° camera) above his head to shoot a video. As shown in Fig. 7B, a blind spot 89 is formed by the user (camera person) who is directly below camera 51 (360° camera).

[0056] Fig. 7C shows bottom images of a cubic panorama image captured by camera 51 (360° camera) as the user moves. Frames Fa to Fe in Fig. 7C are bottom images of a cubic panorama image captured when the user is in first position Pa, second position Pb, third position Pc, fourth position Pd, and fifth position Pe, respectively. The center of each of frames Fa to Fe shows how a blind spot 89 is formed by the user (photographer 71) and the floor surface located in the blind spot 89 cannot be captured.

[0057] As shown in frames Fa to Fe in FIG. 7C, the position of the blind spot 89 moves as the user moves. In frame Fa taken when the user is at a first position Pa (FIG. 7A), the marker 81 is away from the user and is not captured. Next, in frame Fb taken when the user is at a second position Pb (FIG. 7A), the blind spot 89 is close to the marker 81. Next, in frame Fc taken when the user is at a third position Pc (FIG. 7A), the blind spot 89 is located directly above the marker 81, and the marker 81 is hidden by the blind spot 89. Next, in frame Fd taken when the user is at a fourth position Pd (FIG. 7A), the blind spot 89 is away from the marker 81. Next, in frame Fe taken when the user is at a fifth position Pe (FIG. 7A), the marker 81 is away from the user and is not captured.

[0058] 7C shows a state in which the blind spot 89 approaches the marker 81 in frame Fb, is located directly above the marker 81 in frame Fc, and passes the marker 81 in frame Fd. At this time, the reference image creation unit 12 wants to determine frame Fc as the reference image, but since the marker 81 is not captured, it is not possible to directly determine the reference image by determining the position of the marker 81. Therefore, the reference image creation unit 12 checks each captured image (frames Fa to Fe) in chronological order. Then, the reference image creation unit 12 extracts each captured image (frames Fb to Fd) in the order of the captured image (frame Fb) in which the marker 81 can be determined (detected), the captured image (frame Fc) in which the marker 81 is hidden by the blind spot 89, and the image (frame Fd) in which the marker 81 can be determined (detected) again thereafter. Next, in the captured images (frames Fb and Fd) before and after the marker 81 is hidden by the blind spot 89, if the average value of the position of the marker 81 is within the circle of the blind spot 89, the reference image creation unit 12 determines one of the captured images (frame Fb or frame Fd) as the reference image. At this time, if there are multiple captured images, the reference image creation unit 12 determines the captured image with the median value in chronological order as the reference image. In this way, in step S265 of FIG. 5, the reference image creation unit 12 performs processing for determining a frame in which the AR marker is located at the center when the AR marker (marker 81) is not captured due to the blind spot 89.

[0059] Returning to FIG. 5, after step S265, the reference image creation unit 12 sets the image determined as the reference image in step S265 as the reference image, and registers the reference image in the reference image database DB1 (FIG. 2A) (step S270). Furthermore, the reference image creation unit 12 registers the absolute position and orientation of the reference image in the reference image shooting position database DB2 (FIG. 2B). Here, the absolute position of the reference image is determined in step S105 (measurement position determination), and uses the coordinate values ​​registered in the reference image shooting position database DB2 (FIG. 2B). In addition, the shooting orientation of the reference image can be determined by the user specifying the orientation in which the AR marker (marker 81) is attached via the input device 31. At this time, the reference image creation unit 12 can determine the inclination angle of the image with respect to the reference angle of view from the orientation of the AR marker (marker 81) on the image determined as the reference image, for example, by the well-known open source software OpenCV.

[0060] After any of steps S215, S245, and S270 described above, the reference image creation unit 12 determines whether or not all target images or videos have been selected (step S280). If the determination in step S280 indicates that all target images or videos have been selected (if "Yes"), the processing of the series of routines shown in Fig. 5 ends. On the other hand, if the determination in step S280 indicates that all target images or videos have not been selected (if "No"), the processing returns to step S205.

[0061] Furthermore, if it is determined in the above-mentioned step S250 that the AR marker is not attached to a floor or a wall while the video is being shot (if "No"), the process proceeds to step S275. In this case, the reference image creation unit 12 notifies the user of an error. This ends the series of routine processes shown in FIG. 5 (step S275). This ends the series of routine processes shown in FIG. 5.

[0062] <Operation of data input unit 13> The data input unit 13 (FIG. 1) receives user instructions via the input device 31 as necessary and acquires information from the database.

[0063] <Operation of the relative position estimation unit 14> The operation of the relative position estimation unit 14 (FIG. 1) will be described below. FIG. 8 is a flowchart showing the operation of the relative position estimation unit 14.

[0064] 3, the relative position estimation unit 14 reads in the video acquired in step S115 (acquiring an image (inspection image) of the site) in Fig. 3 (step S305). The video to be read is specified by the user via the input device 31.

[0065] After step S305, the relative position estimation unit 14 cuts out images (still images) from the video at specific time intervals (frame intervals) Ti, and stores them in the image database DB3 (FIG. 2C) as site images (inspection images) (step S310). The specific time intervals (frame intervals) Ti are specified by the user via the input device 31 as, for example, 30 frame intervals or 1 second intervals. The relative position estimation unit 14 also cuts out images from each frame of the 360° video captured by the laser scanner 52, converts the cut-out images into cubic panorama images, and corrects image distortion. The relative position estimation unit 14 then cuts out the cubic panorama image into six separate images with a viewing angle of 60°, and stores them in the image database DB3 (FIG. 2C).

[0066] After step S310, the relative position estimation unit 14 reads (acquires) a reference image registered in the reference image database DB1 (FIG. 2A) (step S315).

[0067] After step S315, the relative position estimation unit 14 estimates the relative position of the image (inspection image) of the site from the read (acquired) reference image and the image (inspection image) of the site cut out in step S310 (step S320). The relative position can be estimated by matching the images by feature point matching and estimating the relative position based on the triangulation principle. The relative position can be estimated by using, for example, commercially available software or open source software that implements known SfM (Structure from Motion), Photogrammetry, SLAM (Simultaneous Localization and Mapping), etc.

[0068] FIG. 9 is an explanatory diagram of the estimation result of the relative position. In FIG. 9, the relative position and orientation determined by the relative position estimation unit 14 for the inspection images P1 to P27 are represented by triangular marks. The relative position estimation unit 14 stores the estimation result of the relative position in the image relative position database DB4 (FIG. 2D). "SfM_ID" shown in FIG. 2D represents the ID of the SfM (Structure from Motion) processing. All image data having the same SfM_ID are the results of estimation by the same SfM processing. The relative position estimation unit 14 can align image data having the same SfM_ID by treating the values ​​of the relative position coordinate X, the relative position coordinate Y, and the relative position coordinate Z as the same coordinate system. Although not shown in FIG. 2D, there is a possibility that image data having different fM_IDs exist. Image data having different fM_IDs are the results of estimation by different SfM processing. Image data having different SfM_IDs have different coordinate systems and scales. Therefore, the relative position estimation unit 14 cannot align image data having different SfM_IDs by treating the values ​​of the relative position coordinate X, the relative position coordinate Y, and the relative position coordinate Z as being in the same coordinate system.

[0069] 8, after step S320, the process proceeds to step S325. In this case, the relative position estimation unit 14 determines whether or not the relative positions have been estimated for all images (still images) cut out in step S310 (step S325).

[0070] When it is determined in step S325 that the relative positions have not been estimated for all images (still images) ("No"), it means that the estimation of the relative positions has failed. For example, the cause of this may be a sudden movement of the camera 51 around the time of shooting of the captured image, which may cause a failure in matching feature points between the captured images. Therefore, in this case ("No" in step S325), the process returns to step S310. In this case, the relative position estimation unit 14 sets the time interval Ti for cutting out images from the video in step S310 to be shorter than the previous time, and cuts out more images. Then, the relative position estimation unit 14 performs the processes of steps S315 and S320 again. At this time, if the time interval Ti is set to be shorter only around the frame number of the image for which the estimation of the relative position has failed, it is possible to suppress an increase in the calculation time when performing step S320 again.

[0071] If it is determined in the above-mentioned step S325 that the relative positions of all the images (still images) have been estimated (if "Yes"), the process proceeds to step S330. In this case, the relative position estimation unit 14 registers the relative position information estimated in step S320 in the image relative position database DB4 (FIG. 2D) (step S330). This ends the series of routine processes shown in FIG. 8.

[0072] <Operation of absolute position estimation unit 15> The operation of the absolute position estimation unit 15 (FIG. 1) will be described below with reference to Fig. 10. Fig. 10 is a flowchart showing the operation of the absolute position estimation unit 15.

[0073] 10, the absolute position estimation unit 15 extracts only data having a specific SfM_ID from the data registered in the image relative position database DB4 (FIG. 2D) (step S405). The SfM_ID may be specified by the user via the input device 31, or may be automatically specified in an appropriate order from among the registered SfM_IDs.

[0074] After step S405, the absolute position estimation unit 15 selects one image (for example, image i) from the extracted images (step S410). This selection is made in chronological order, for example.

[0075] After step S410, the absolute position estimation unit 15 extracts all reference images included in the extracted images and images of the site whose absolute positions have been estimated (inspection images) (step S415). Hereinafter, all reference images included in the extracted images and images of the site whose absolute positions have been estimated (inspection images) are collectively referred to as "images whose absolute positions have been estimated."

[0076] After step S415, the absolute position estimation unit 15 compares the relative distance between image i and each image for which the absolute position has been estimated, and extracts two images for which the absolute position has been estimated in order of the shortest relative distance to image i (step S420).

[0077] After step S420, the absolute position estimation unit 15 judges whether or not two images for which absolute positions have been estimated have been extracted (step S425). If it is judged in step S425 that two images for which absolute positions have been estimated have not been extracted (in the case of "No"), the process returns to step S410. In this case, the absolute position estimation unit 15 selects the next image in step S410 and repeats the processes from step S410 onward.

[0078] On the other hand, if it is determined in step S425 that two images for which absolute positions have been estimated have been extracted (if "Yes"), the process proceeds to step S430. In this case, the absolute position estimation unit 15 estimates the absolute coordinates of the image i (step S430).

[0079] Here, the process of estimating the absolute coordinates of image i in step S430 in Fig. 10 will be described with reference to Fig. 11 and Fig. 12. Fig. 11 is an explanatory diagram of the relationship between the relative position and the absolute position. Fig. 12 is an explanatory diagram of the absolute position estimation result.

[0080] Figure 11 shows that, of the five inspection images PT1 to PT5, inspection image PT1 is selected in step S410 of Figure 10, and reference image shooting position BPA11 and reference image shooting position BPA12 are selected as the nearest images with absolute positions estimated in step S420 of Figure 10.

[0081] FIG. 11 shows a vector V11, relative distances Lr11, Lr12, Lr13, and absolute distance La13. The vector V11 is a vector directed from the reference image photographing position BPA11 to the inspection image PT1. The relative distance Lr11 is a relative distance between the reference image photographing position BPA11 and the inspection image PT1 that is not a real scale. The relative distance Lr12 is a relative distance between the reference image photographing position BPA12 and the inspection image PT1 that is not a real scale. The relative distance Lr13 is a relative distance between the reference image photographing position BPA11 and the reference image photographing position BPA12 that is not a real scale. The absolute distance La13 is an absolute distance between the reference image photographing position BPA11 and the reference image photographing position BPA12 that is a real scale.

[0082] Here, the positions (relative positions) of the inspection image PT1, the reference image shooting position BPA11, and the reference image shooting position BPA12 in the relative coordinate system are known. Therefore, the coordinate specifying device 100 can calculate the vector V11 from the reference image shooting position BPA11 toward the inspection image PT1, and the relative distance Lr13 between the reference image shooting position BPA11 and the reference image shooting position BPA12.

[0083] Furthermore, the positions (absolute positions) of the reference image photographing position BPA11 and the reference image photographing position BPA12 in the absolute coordinate system are known, so the coordinate specifying device 100 can calculate the absolute distance La13 between the reference image photographing position BPA11 and the reference image photographing position BPA12.

[0084] The coordinate identifying device 100 obtains a correction value for converting a relative position to an absolute position by multiplying the vector V11 by the absolute distance La13 between the reference image photographing position BPA11 and the reference image photographing position BPA12 divided by the relative distance Lr13 between the reference image photographing position BPA11 and the reference image photographing position BPA12 (i.e., calculating "V11 x (La13 / Lr13)").The coordinate identifying device 100 can then obtain the absolute position (X1, Y1, Z1) of the inspection image PT1 by calculating the following formula (1) using the absolute position (X11, Y11, Z11) of the reference image photographing position BPA11 and the correction value. (X1,Y1,Z1)=(X11,Y11,Z11)+V11×(La13 / Lr13) ……(1)

[0085] Fig. 12 shows the results of estimating the absolute position and orientation of each inspection image using the absolute coordinates estimated in step S430 in Fig. 10, and plotting the estimation results on the coordinate system of the CAD model. In the example shown in Fig. 12, inspection images P2, P5, P15, and P19 are the images closest to the reference image shooting positions BP1 to BP4. In this way, by estimating the absolute position, the shooting position of each inspection image on the CAD model can be identified.

[0086] Returning to Fig. 10, after step S430, the absolute position estimation unit 15 judges whether or not all images have been selected (step S435). If it is judged in step S430 that all images have been selected (if "Yes"), the series of routine processes shown in Fig. 10 ends. On the other hand, if it is judged in step S430 that all images have not been selected (if "No"), the process returns to step S410.

[0087] <Operation of position correction unit 16> The operation of the position correction unit 16 (FIG. 1) will be described below. The position correction unit 16 corrects the absolute position of the image estimated by the absolute position estimation unit 15. FIGS. 13A and 13B are diagrams illustrating the operation of the position correction unit 16.

[0088] Fig. 13A shows an example in which the position and orientation of an image (inspection image) with an estimated absolute position estimated by the absolute position estimation unit 15 is superimposed on the coordinate system of a CAD model and displayed on the display device 32. Fig. 13B shows an example in which, when the position and orientation of some of the images with an estimated absolute position are incorrect, the user corrects the incorrect position and orientation of the inspection image by operating the mouse of the input device 31 or the like. Fig. 13B shows an inspection image (inspection image Pxb in the example shown in Fig. 13B) after correction of an inspection image with an incorrect position and orientation (inspection image Pxa in the example shown in Fig. 13A). The position correction unit 16 can accept a correction operation performed by the user and correct the absolute position and orientation of the image.

[0089] <Operation of the attribute assignment unit 17> The operation of the attribute assigning unit 17 (FIG. 1) will be described below. The attribute assigning unit 17 assigns the ID of the equipment shown in the image to the inspection image whose absolute position has been estimated. FIG. 14 is a flowchart showing the operation of the attribute assigning unit 17.

[0090] As shown in FIG. 14, the attribute assigning unit 17 arranges the equirectangular panoramic image of the inspection image and a CAD model in a VR (Virtual Reality) three-dimensional space (step S505). Then, the attribute assigning unit 17 arranges the equirectangular panoramic image of the inspection image at the estimated absolute position of the inspection image to generate a half-line ray R11 (FIG. 15A) (step S510). For example, the attribute assigning unit 17 arranges each pixel of the equirectangular panoramic image of the inspection image at a predetermined angle from the absolute position of the inspection image. In this way, the attribute assigning unit 17 generates a half-line ray R11 (FIG. 15A). The angle of each pixel is represented by a circumferential angle φ and a radial angle θ. This image is shown in FIG. 15A and FIG. 15B. Note that a half-line is a straight line that has a starting point and extends infinitely from the starting point.

[0091] 15A and 15B are explanatory diagrams of rays generated by the attribute assignment unit 17. FIG. 15A shows a half-line ray R11 in which the angle of each pixel is expressed by the circumferential angle φ and the radial angle θ. FIG. 15B shows an example in which the ray R11 (FIG. 15A) is converted into a cubic panoramic image (equidistant projection panoramic image). In the example shown in FIG. 15B, the angle of the top pixel is θ=0°, the angle of the bottom pixel is θ=180°, the angle of the leftmost pixel is φ=0, and the angle of the rightmost pixel is φ=360°. The difference Δθ in θ between each pixel can be calculated by dividing 180° by the number of pixels in the vertical direction of the image. Similarly, the difference Δφ in φ between each pixel can be calculated by dividing 360° by the number of pixels in the horizontal direction of the image. Therefore, for example, in an equirectangular panoramic image with M rows and N columns, the angle of each pixel can be calculated as θ=M×Δφ and φ=N×Δφ. The attribute assignment unit 17 generates a half-line ray R11 ( FIG. 15A ) in the direction of (θ, φ) by arranging each pixel of the equirectangular panoramic image of the inspection image, for which the angle of each pixel has been calculated in this manner, at the angle calculated from the absolute position of the inspection image.

[0092] Returning to Fig. 14, after step S510, the attribute assignment unit 17 extracts (detects) a point of contact between the ray R11 (Fig. 15A) in the (θ, φ) direction generated in step S510 and the CAD model (step S515). The extraction (detection) of the point of contact can be performed in a similar manner to the occlusion determination process provided in general CAD. By extracting (detecting) the point of contact, the attribute assignment unit 17 can obtain the object ID of the CAD model having the point of contact.

[0093] After step S515, the attribute assignment unit 17 associates the acquired object ID with each pixel and registers it in the image-CAD correspondence database DB7 (FIG. 2G) (step S520). In the example shown in FIG. 2G, the image-CAD correspondence database DB7 includes the image ID, the X coordinate value, the Y coordinate value of the image coordinate system, and the ID of the CAD object detected by the contact point determination with the ray R11 (FIG. 15A).

[0094] <Operation of image display unit 18> The operation of the image display unit 18 (FIG. 1) will be described below. As shown in FIG. 16, the image display unit 18 can display a panoramic image, attribute information of the image, and the like in a VR three-dimensional space. FIG. 16 is a flowchart showing the operation of the image display unit 18.

[0095] As shown in Fig. 16, the image display unit 18 refers to the CAD model database DB6 (Fig. 2F), and when the CAD object to be pasted on the inspection image is registered in the CAD model database DB6, creates a CAD object for pasting the image at the image absolute position in the VR three-dimensional space (step S605a). The shape of the CAD object may be a sphere or cube of a certain size. The size of the CAD object may be any value (for example, 1m, etc.).

[0096] After step S605a, the image display unit 18 pastes the panoramic image onto the inner surface of the CAD object for pasting the image (step S610a). At this time, if the CAD object is a sphere, an equirectangular panoramic image is pasted, and if the CAD object is a cube, a cubic panoramic image is pasted.

[0097] After step S610a, the image display unit 18 places a VR camera object virtually representing the camera 51 at the absolute position of the panoramic image (step S615a). This allows the coordinate specifying device 100 to view the panoramic image in the VR three-dimensional space.

[0098] After step S615a, the image display unit 18 superimposes the attribute information of the image on the camera view (step S620a). The superimposed display of the attribute information can be performed by the image display unit 18 acquiring a CAD object ID for each pixel of the panoramic image using information in the image-CAD correspondence database DB7 (FIG. 2G). For example, the image display unit 18 can change the color of the pixel of the panoramic image for each acquired CAD object ID to display it as a highlight, or select one representative pixel position and create an appropriate marker object at a position on the inner surface of the object that corresponds to the pixel position.

[0099] 17, the image display unit 18 can search for an arbitrary image and adjust the orientation of the camera 51 to the direction of the searched device. FIG 17 is a flowchart showing the operation of the image display unit 18.

[0100] As shown in FIG. 17, when the user operates the input device 31 to input a device ID (CAD object ID) as a text character string or the like, the image display unit 18 accepts the input ID (step S605b).

[0101] After step S605b, the image display unit 18 refers to the image-CAD correspondence database DB7 (FIG. 2G) to search for images having the input device ID as an attribute (i.e., images showing the input device ID) (step S610b), and selects one corresponding image.

[0102] After step S610b, the image display unit 18 determines whether or not a corresponding image is found (step S615b). If the determination in step S615b indicates that a corresponding image is found ("Yes"), the process proceeds to step S620b. On the other hand, if the corresponding image is not found ("No"), the process returns to step S605b.

[0103] If the corresponding image is found in the determination of step S615b (if "Yes"), the image display unit 18 acquires absolute position information from the image absolute position database DB5 (FIG. 2E) using the image ID assigned to the corresponding image as a key. Then, the image display unit 18 places a VR camera object virtually representing the camera 51 at the absolute position of the panoramic image. As a result, the image display unit 18 displays a panoramic image showing the device corresponding to the searched device ID (hereinafter, may be referred to as the "searched device") (step S620b).

[0104] After step S620a, the image display unit 18 adjusts (points) the orientation of the VR camera object to the direction in which the searched device appears. This can be done by selecting one representative pixel position from the panoramic image in which the searched device appears, expressing the pixel angle with the circumferential angle φ and the radial angle θ, and setting the orientation of the VR camera object to the direction (θ, φ).

[0105] <Main features of the coordinate determination device 100> (1) As shown in Fig. 1, the coordinate specifying device 100 according to this embodiment includes a relative position estimation unit 14 and an absolute position estimation unit 15. The relative position estimation unit 14 estimates a relative shooting position that is not on a real scale for an image group including a reference image whose absolute shooting position on a real scale has been specified and a non-reference image (image of the site) whose absolute shooting position has not been specified. The absolute position estimation unit 15 estimates the absolute shooting position of the non-reference image using the relative shooting position and the absolute shooting position of the reference image.

[0106] The coordinate specifying device 100 according to this embodiment estimates a relative shooting position not in real scale for an image group including a reference image in which an absolute shooting position in real scale is specified and a non-reference image (image of the site) in which an absolute shooting position is not specified, by the relative position estimation unit 14. Then, the coordinate specifying device 100 can easily estimate the absolute shooting position of the image of the site (inspection image) which is a non-reference image, by using the relative shooting position and the absolute shooting position of the reference image by the absolute position estimation unit 15. The coordinate specifying device 100 according to this embodiment can estimate the absolute shooting position of all the non-reference images (inspection images) without attaching multiple markers 81 to the floor or wall of the site or measuring the distance to any point with the laser scanner 52. This allows the coordinate specifying device 100 to quickly display the image of the site (inspection image) in the VR three-dimensional space. For example, in a case where the preparation man-hour required for estimating the absolute shooting position of the image of the site (inspection image) in the conventional technology was one month, the coordinate specifying device 100 can reduce it to about one week.

[0107] (1a) As shown in FIG. 10, in the coordinate identifying device 100 according to this embodiment, the image group may include at least two reference images (step S420) and one or more non-reference images (step S410).

[0108] The coordinate specifying device 100 according to this embodiment can obtain the absolute distance between the reference images and the absolute position of any of the reference images using at least two or more reference images. The coordinate specifying device 100 can also estimate the relative distance between each reference image and the non-reference image and the vector between any of the reference images and the non-reference image from each reference image and the non-reference image. The coordinate specifying device 100 can then estimate the absolute position of the non-reference image using the estimated absolute distance, relative distance, and vector.

[0109] (2) In the coordinate identification device 100 according to this embodiment, the relative position estimation unit 14 and the absolute position estimation unit 15 may be configured to generate information representing the shooting position so as to include at least one of information on two-dimensional coordinates parallel to the ground surface and three-dimensional coordinates including the vertical direction, and information identifying the direction within the image.

[0110] The coordinate specifying device 100 according to this embodiment can facilitate estimation of the relative photographing position of an image.

[0111] (2a) In the coordinate specifying device 100 according to the present embodiment, the relative position estimation unit 14 may be configured to calculate the relative shooting position of an image using SfM (Structure from Motion), photogrammetry, or SLAM (Simultaneous Localization and Mapping).

[0112] The coordinate specifying device 100 according to this embodiment can use commercially available software or open source software that implements SfM (Structure from Motion), Photogrammetry, or SLAM (Simultaneous Localization and Mapping), which are widely available. Therefore, the coordinate specifying device 100 can realize a configuration that calculates the relative shooting position of an image at low cost.

[0113] (3) In the coordinate identification device 100 of this embodiment, the absolute position estimation unit 15 may be configured to calculate the absolute shooting position of the non-reference image using the distance between the relative shooting positions and absolute shooting positions of the two reference images, and the distance between the relative shooting position of the non-reference image and the relative shooting position of the reference image.

[0114] The coordinate specifying device 100 according to this embodiment can easily calculate the absolute shooting position of the non-reference image.

[0115] (4) As shown in FIG. 1, the coordinate specifying device 100 according to this embodiment may further include a position correction unit 16 that corrects at least one of the relative position and absolute position information using an input from a user.

[0116] The coordinate specifying device 100 according to this embodiment can easily correct at least one of the information on the relative position and the absolute position.

[0117] (5) As shown in FIG. 1, the coordinate identification device 100 according to this embodiment may further include an attribute assignment unit 17 that assigns attribute information of the CAD model to either or both of the reference image and the non-reference image based on the absolute shooting position and the CAD model.

[0118] The coordinate specifying device 100 according to this embodiment can add attribute information of a CAD model to a reference image and a non-reference image. Therefore, the coordinate specifying device 100 can easily search for an arbitrary image.

[0119] (6) In the coordinate identification device 100 of this embodiment, the attribute assignment unit 17 may be configured to assign, in addition to the attribute information of the CAD model, distance information between the absolute shooting position of the image and the absolute position of the CAD model to either or both of the reference image and the non-reference image.

[0120] The coordinate specifying device 100 according to this embodiment can assign distance information between the absolute photographing position of the image and the absolute position of the CAD model to the reference image and the non-reference image. The coordinate specifying device 100 can easily perform the relative position estimation process and the absolute position estimation process by using the distance information assigned to the reference image and the non-reference image in the relative position estimation process and the absolute position estimation process.

[0121] (6a) In the coordinate identification device 100 of this embodiment, the attribute assignment unit 17 may be configured to identify equipment appearing in a non-reference image by matching the non-reference image, whose absolute shooting position has been estimated by the absolute position estimation unit 15, with a CAD model.

[0122] The coordinate specifying device 100 according to this embodiment can specify the facility that appears in the non-reference image.

[0123] (7) As shown in Fig. 1, the coordinate identifying device 100 according to this embodiment may further include an acquisition unit 11 that acquires various data, and a reference image creation unit 12 that creates a reference image. The acquisition unit 11 may acquire an image or video captured by a camera 51, and the reference image creation unit 12 may create a reference image using the captured image or video.

[0124] The coordinate specifying device 100 according to this embodiment can create a reference image using an image or video captured by the camera 51.

[0125] (8) In the coordinate identification device 100 of this embodiment, the acquisition unit 11 acquires an image or video of a site to which a marker 81 indicating a specified absolute position is affixed, and the reference image creation unit 12 is configured to calculate the absolute shooting position of the reference image based on the position of the marker 81 appearing in the captured image or video.

[0126] The coordinate specifying device 100 according to this embodiment can calculate the absolute shooting position of the reference image based on the position of the marker 81 appearing in an image or video.

[0127] (9) In the coordinate determination device 100 of this embodiment, the acquisition unit 11 further acquires positional information of a three-dimensional point cloud at the site measured by the laser scanner 52, and the reference image creation unit 12 is configured to calculate the absolute shooting position of the reference image using the positional information of the three-dimensional point cloud.

[0128] The coordinate specifying device 100 according to this embodiment can calculate the absolute shooting position of the reference image by using the position information of the three-dimensional point cloud.

[0129] (10) As shown in FIG. 4A, in the coordinate specifying device 100 according to this embodiment, the acquisition unit 11 may be configured to select a measurement point for the laser scanner 52 from among the four corners of a room or a corner of a walking route.

[0130] The coordinate determination device 100 of this embodiment can easily perform relative position estimation processing and absolute position estimation processing by setting a location where the direction of a non-reference image (image of the site) is likely to change as the measurement point of the laser scanner 52.

[0131] (11) As shown in FIG. 12, in the coordinate identification device 100 according to this embodiment, it is preferable to further display a shape (a triangular mark in FIG. 12) representing the absolute shooting position calculated by the absolute position estimation unit 15 in a two-dimensional or three-dimensional space, and display the shape in correspondence with either the reference image or the non-reference image.

[0132] The coordinate specifying device 100 according to this embodiment can make it easier for the user to grasp the absolute photographing position. Therefore, when the user performs an operation (for example, a correction operation of the position and orientation of the inspection image shown in FIG. 13B, etc.), the coordinate specifying device 100 can make it easier for the user to make a decision.

[0133] (12) In the coordinate specifying device 100 according to the present embodiment, it is preferable to search for and display an image having attribute information of a CAD model that partially matches a character string input by the user.

[0134] The coordinate specifying device 100 according to this embodiment can facilitate searching for any image.

[0135] (13) As shown in FIG. 12, in the coordinate identification device 100 of this embodiment, it is preferable to display, among the searched images, the images (inspection images P2, P5, P15, P19) in which the distance between the absolute shooting positions of the images (reference image shooting positions BP1 to BP4) and the absolute position of the CAD model is the smallest.

[0136] The coordinate determination device 100 of this embodiment can make it easier for the user to understand the image shooting position at which the distance between the absolute image shooting position (reference image shooting positions BP1 to BP4) and the absolute position of the CAD model is smallest by displaying images (inspection images P2, P5, P15, P19).

[0137] (14) As shown in Fig. 8 and Fig. 10, the coordinate identification method according to this embodiment may include a relative position estimation step (step S320 in Fig. 8) and an absolute position estimation step (step S430 in Fig. 10). The relative position estimation step estimates a relative shooting position that is not on a real scale for an image group including a reference image whose absolute shooting position on a real scale has been identified and a non-reference image whose absolute shooting position has not been identified. The absolute position estimation step estimates the absolute shooting position of the non-reference image using the relative shooting position and the absolute shooting position of the reference image.

[0138] The coordinate specifying method according to this embodiment makes it possible to easily estimate the absolute shooting position of an on-site image, which is a non-reference image.

[0139] (15) The control program Pr according to this embodiment may be a program for causing a computer to function as a relative position estimation unit 14 and an absolute position estimation unit 15. The relative position estimation unit 14 is a component that estimates a relative shooting position that is not on a real scale for an image group including a reference image whose absolute shooting position on a real scale has been specified and a non-reference image whose absolute shooting position has not been specified. The absolute position estimation unit 15 is a component that estimates the absolute shooting position of the non-reference image using the relative shooting position and the absolute shooting position of the reference image.

[0140] The control program Pr according to this embodiment can realize a coordinate specifying device 100 that easily estimates the absolute photographing position of an on-site image, which is a non-reference image.

[0141] As described above, according to the coordinate specifying device 100 of this embodiment, it is possible to easily estimate the absolute photographing position of an on-site image, which is a non-reference image.

[0142] The present invention is not limited to the above-described embodiment, and includes various modified examples. For example, the above-described embodiment has been described in detail to easily explain the present invention, and is not necessarily limited to having all of the configurations described. In addition, it is possible to replace a part of the configuration of the embodiment with another configuration, and it is also possible to add another configuration to the configuration of the embodiment. In addition, it is possible to add, delete, or replace a part of each configuration with another configuration.

[0143] In the above embodiment, the coordinate specifying device 100 is configured as an inspection support device. However, the present invention can also be applied to devices other than inspection support devices. [Explanation of symbols]

[0144] 9 Storage medium 10 Control section 11 Acquisition Department 12 Reference image creation section 13 Data Entry Section 14 Relative position estimation unit 15 Absolute position estimation unit 16 Position correction section 17 Attribute assignment section 18 Image display section 20 Memory section 31 Input Devices 32 Display device 50 Acquisition device 51 Camera 52 Laser Scanner 61 Wall 71 Photographer 81 Marker 89 Blind spot 100 Coordinate identification device (inspection support device) BP1, BP2, BP3, BP4, BPA1, BPA2 Reference image capture position DB1 Reference Image Database DB2 Reference Image Location Database DB3 Image Database DB4 Image Relative Position Database DB5 Image Absolute Position Database DB6 CAD model database DB7 Image-CAD Correspondence Relational Database Fa, Fb, Fc, Fd, Fe Frame La13 Absolute distance Lr11, Lr12, Lr13 relative distance P1, P2, P5, P15, P19, P27 Inspection images (non-reference images) PT1, PT2, PT3, PT4, PT5 inspection images Pr control program R11 Ray V11 Vector X,Y,Z relative position coordinates X11,Y11,Z11 Absolute position θ,φ angle (tilt)

Claims

1. a relative position estimation unit that estimates a relative shooting position that is not real scale for an image group including a reference image for which an absolute shooting position in real scale is specified and a non-reference image for which the absolute shooting position is not specified; an absolute position estimation unit that estimates an absolute shooting position of the non-reference image by using the relative shooting position and the absolute shooting position of the reference image. A coordinate specifying device characterized by:

2. 2. The coordinate determining device according to claim 1, The relative position estimation unit and the absolute position estimation unit generate information representing the photographing position so as to include at least one of information on two-dimensional coordinates parallel to the ground and three-dimensional coordinates including a vertical direction, and information specifying a direction within the image. A coordinate specifying device characterized by:

3. 2. The coordinate determining device according to claim 1, The absolute position estimation unit calculates the absolute shooting position of the non-reference image by using a distance between the relative shooting positions and the absolute shooting positions of the two reference images and a distance between the relative shooting position of the non-reference image and the relative shooting position of the reference image. A coordinate specifying device characterized by:

4. 2. The coordinate determining device according to claim 1, Further, a position correction unit is provided that corrects at least one of the relative position and the absolute position information using an input from a user. A coordinate specifying device characterized by:

5. 2. The coordinate determining device according to claim 1, The image capturing apparatus further includes an attribute assigning unit that assigns attribute information of the CAD model to either one or both of the reference image and the non-reference image based on the absolute photographing position and the CAD model. A coordinate specifying device characterized by:

6. 6. The coordinate determining device according to claim 5, The attribute assigning unit assigns distance information between an absolute photographing position of an image and an absolute position of a CAD model to either one or both of the reference image and the non-reference image in addition to the attribute information of the CAD model. A coordinate specifying device characterized by:

7. 2. The coordinate determining device according to claim 1, Furthermore, an acquisition unit that acquires various data; A reference image creating unit that creates a reference image, The acquisition unit acquires an image or a video captured by a camera, The reference image creation unit creates the reference image using a captured image or video. A coordinate specifying device characterized by:

8. 8. The coordinate specifying device according to claim 7, The acquisition unit acquires an image or video of a site to which a marker indicating a defined absolute position is attached, The reference image creation unit calculates an absolute shooting position of the reference image based on a position of the marker appearing in a captured image or video. A coordinate specifying device characterized by:

9. 8. The coordinate specifying device according to claim 7, The acquisition unit further acquires position information of a three-dimensional point cloud of the site measured by a laser scanner, The reference image creation unit calculates the absolute shooting position of the reference image using the position information of the three-dimensional point cloud. A coordinate specifying device characterized by:

10. 10. The coordinate specifying device according to claim 9, The acquisition unit selects a measurement point for the laser scanner from among four corners of a room or a corner of a walking path. A coordinate specifying device characterized by:

11. 2. The coordinate determining device according to claim 1, Furthermore, a shape representing the absolute photographing position calculated by the absolute position estimation unit is displayed in a two-dimensional or three-dimensional space, and either one of the reference image and the non-reference image is displayed in association with the shape. A coordinate specifying device characterized by:

12. 12. The coordinate specifying device according to claim 11, Searches for and displays images having CAD model attribute information that partially matches a character string entered by a user A coordinate specifying device characterized by:

13. 13. The coordinate determining device according to claim 12, Among the retrieved images, an image having a minimum distance between the absolute photographing position of the image and the absolute position of the CAD model is displayed. A coordinate specifying device characterized by:

14. a relative position estimation step of estimating a relative shooting position that is not real scale for an image group including a reference image for which an absolute shooting position in real scale has been specified and a non-reference image for which the absolute shooting position has not been specified; and an absolute position estimating step of estimating an absolute shooting position of the non-reference image by using the relative shooting position and the absolute shooting position of the reference image. A coordinate determination method comprising:

15. Computer, a relative position estimation unit that estimates a relative shooting position that is not real scale for an image group including a reference image for which an absolute shooting position in real scale is specified and a non-reference image for which the absolute shooting position is not specified; A program for causing the program to function as an absolute position estimation unit that estimates the absolute shooting position of the non-reference image by using the relative shooting position and the absolute shooting position of the reference image.

Images