Management support system for buildings or civil engineering structures, and mobile communication terminals.

The management support system uses marker-based image recognition and alignment techniques to ensure accurate self-positioning of mobile terminals within buildings or civil engineering structures, enhancing the precision of image and drawing space comparison.

JP7953854B2Active Publication Date: 2026-11-19IXS CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2023000458
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2023-01-05
Publication Date
2026-11-19
Estimated Expiration
2043-01-05

AI Technical Summary

Technical Problem

Existing self-position estimation methods for imaging means in construction management systems lack accuracy and efficiency, particularly in determining the position of mobile communication terminals within buildings or civil engineering structures.

Method used

A management support system utilizing markers installed at multiple locations, where a mobile communication terminal with a camera and acceleration sensor detects markers through image recognition, determines its position based on marker location information and motion data, and aligns captured video space with drawing data using triangulation and rotational alignment techniques.

Benefits of technology

Maintains accurate self-position determination of the mobile communication terminal while navigating within buildings or civil engineering structures, allowing for precise comparison and management of image and drawing spaces.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007953854000001
    Figure 0007953854000001
  • Figure 0007953854000002
    Figure 0007953854000002
  • Figure 0007953854000003
    Figure 0007953854000003
Patent Text Reader

Abstract

To provide a management support system for a building or a civil engineering structure that prevents a reduction in the accuracy of a mobile communication terminal, which photographs the inside of a building or a civil engineering structure to be managed, in determining its own position, which is caused by its movement, or a mobile communication terminal used for the system.SOLUTION: A mobile communication terminal detects a marker through image recognition processing on a video photographed by a camera while moving the inside of a building or a civil engineering structure, extracts a drawing marker corresponding to the detected marker from drawing data, determines its own position based on information on the position of the extracted drawing marker and information on its own motion detected by an acceleration sensor, and collates a video space photographed by the camera and a drawing space represented by the drawing data based on the determined its own position and outputs a result of the collation.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a management support system for buildings or civil engineering structures, and also relates to a mobile communication terminal used in the system.

Background Art

[0002] In recent years, the Ministry of Land, Infrastructure, Transport and Tourism has published guidelines for BIM and CIM, and the introduction of BIM / CIM into operations in the architectural and civil engineering fields is progressing. Here, BIM is an abbreviation for Building Information Modeling, which is a three-dimensional model of a building created on a computer with various attribute data added thereto. CIM is an abbreviation for Construction Information Modeling, which applies the concept of BIM to the civil engineering field.

[0003] By introducing BIM / CIM from the planning, survey, and design stages, information sharing in the subsequent construction, maintenance and management stages is facilitated, and the efficiency and sophistication of a series of operations and tasks can be achieved. In addition, as secondary effects of the introduction of BIM / CIM, improvements in national life through better infrastructure development and maintenance, and increased motivation of employees engaged in the construction and civil engineering fields are also expected.

[0004] In view of the background described above, various construction management support systems have been proposed one after another (for example, Patent Document 1). The system according to Patent Document 1 attaches an imaging means to workers or work vehicles that patrol construction sites for architecture or civil engineering, analyzes video data captured by the imaging means, and manages the progress of work at the construction site. Furthermore, the document discloses that the above-mentioned imaging means is equipped with an estimation means for estimating its own position, and that the self-position estimated by the estimation means is used to analyze the video data. In addition, the document explains that SLAM, satellite positioning systems, position estimation methods using feature point extraction from multiple images, autonomous navigation, and positioning systems using communication devices can be utilized as the above-mentioned estimation means. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2019-148946 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] The self-position estimation method for the imaging means disclosed in Patent Document 1 is merely a list and combination of existing technologies, and there is still room for improvement.

[0007] The present invention has been made in view of the above problems, and provides a building or civil engineering structure management support system, or a mobile communication terminal used in said system, that prevents the accuracy of determining the self-position of a mobile communication terminal used to photograph the premises of a building or civil engineering structure from decreasing as it moves. [Means for solving the problem]

[0008] According to the present invention, a building or civil engineering structure management support system is provided to support the management of a building or civil engineering structure in which markers are installed at multiple locations, comprising: a storage device that stores drawing data of the building or civil engineering structure; and a mobile communication terminal that is communicatively connected to the storage device, wherein the drawing data includes location information of drawing markers corresponding to at least some of the markers placed on the building or civil engineering structure, before the mobile communication terminal moves within the premises of the building or civil engineering structure, and the mobile communication terminal has a camera and an acceleration sensor, and detects markers by image recognition processing of images captured by the camera when moving within the premises of the building or civil engineering structure. At a first timing in which at least two or more markers are detected from the captured video, a plurality of drawing markers corresponding to some or all of the detected markers are extracted from the drawing data, and the self-position at the first timing is determined based on the position information of the extracted plurality of drawing markers and the self-motion information detected by the acceleration sensor. At a second timing in which a new marker is detected from the video captured by the camera that has moved to a different location after the first timing, a plurality of drawing markers corresponding to the newly detected marker and two or more markers included in the markers detected up to the detection of the said marker are again extracted from the drawing data, and the self-position at the second timing is determined based on the position information of the extracted plurality of drawing markers and the self-motion information detected by the acceleration sensor. Based on the determined self-position of the mobile communication terminal, the video space captured by the camera and the drawing space represented by the drawing data are compared and output. Furthermore, the mobile communication terminal extracts a first drawing marker and a second drawing marker from the drawing data at the first timing or the second timing, searches the drawing data for a third drawing marker that satisfies predetermined conditions from among the drawing markers that are different from both the extracted first drawing marker and the second drawing marker, and if it is not possible to find a third drawing marker that satisfies the predetermined conditions, it uses the position information relating to the two points of the first drawing marker and the second drawing marker, and its own motion information detected by the acceleration sensor, to determine the third drawing marker. The system determines its own position, and when it finds the third drawing marker that satisfies the predetermined conditions, it determines its own position based on the positional information of the three points, the first drawing marker, the second drawing marker, and the third drawing marker, and the motion information of the system detected by the acceleration sensor. The predetermined conditions are that, for the triangle formed by the three points, the first drawing marker, the second drawing marker, and the third drawing marker, the angle of the interior angle with the first drawing marker as the vertex and the angle of the interior angle with the second drawing marker as the vertex are both within a predetermined range. A management support system for buildings or civil engineering structures is provided, characterized by the following features. Furthermore, according to the present invention, a building or civil engineering structure management support system for a building or civil engineering structure in which markers are installed at multiple locations, comprising: a storage device that stores drawing data of the building or civil engineering structure; and a mobile communication terminal that is communicably connected to the storage device, wherein the drawing data includes location information of drawing markers corresponding to at least some of the markers placed on the building or civil engineering structure, before the mobile communication terminal moves within the premises of the building or civil engineering structure; the mobile communication terminal has a camera and an acceleration sensor, detects markers by image recognition processing of images captured by the camera when moving within the premises of the building or civil engineering structure, and at a first timing in which at least two or more markers are detected from the captured images, extracts a plurality of drawing markers corresponding to some or all of the detected markers from the drawing data, and calculates its own position at the first timing based on the location information of the extracted plurality of drawing markers and its own motion information detected by the acceleration sensor. At a second timing in which a new marker is detected from the image captured by the camera, which has moved to a different location after the first timing, the mobile communication terminal determines its own position at the second timing based on the positional information of the newly detected marker and the motion information of the mobile communication terminal detected by the acceleration sensor, compares the image space captured by the camera with the image space represented by the image data based on the determined position of the mobile communication terminal, and outputs the result. Furthermore, the mobile communication terminal designates the marker corresponding to the first image marker as the first marker, the marker corresponding to the second image marker as the second marker, and the marker corresponding to the third image marker as the third marker, the vector from the first image marker to the second image marker in the image space as the image reference vector, and the vector from the first image marker to the second image marker in the image space as the image reference vector, and at the first timing or the second timing,The first drawing marker and the second drawing marker are extracted from the drawing data, and a third drawing marker that satisfies predetermined conditions is searched from the drawing data among the drawing markers that are different from both the extracted first and second drawing markers. If a third drawing marker that satisfies the predetermined conditions cannot be found, a first process is performed in which the position of the first marker and the position of the first drawing marker are translated parallel to each other, and a second process is performed in which the image reference vector and the drawing reference vector are made to match by rotational movement around the position where the first marker and the first drawing marker overlap as a result of the first process, and the camera is used to... A management support system for buildings or civil engineering structures is provided, characterized by comparing the captured video space with the drawing space represented by the drawing data, and when the third drawing marker that satisfies the predetermined conditions is found, executing the first and second processes, and further executing a third process to match a vector that passes through the third marker and intersects the video reference vector perpendicularly with the drawing reference vector, by rotational movement around the video reference vector and the drawing reference vector that were matched in the second process, thereby matching the video space captured by the camera with the drawing space represented by the drawing data.

[0009] According to the present invention, a building or civil engineering structure management support system is provided to support the management of a building or civil engineering structure in which markers are installed at multiple locations, comprising: a storage device that stores drawing data of the building or civil engineering structure; and a mobile communication terminal that is communicatively connected to the storage device, wherein the drawing data does not contain location information of drawing markers corresponding to markers placed on the building or civil engineering structure before the mobile communication terminal moves within the premises of the building or civil engineering structure, and the mobile communication terminal has a camera and an acceleration sensor, and when moving within the premises of the building or civil engineering structure, the camera captures images. When the marker detected by image recognition processing on the captured video is the first marker, the first marker is registered at the origin of the coordinate space in the drawing data. When the marker detected by image recognition processing on the video captured by the camera while the building or civil engineering structure is moving is not the first marker and a drawing marker corresponding to that marker is not registered in the drawing data, the mobile communication terminal's own position and the position information of the marker are determined based on the position information of other markers detected before that marker and its own motion information detected by the acceleration sensor, and the drawing marker corresponding to that marker is registered in the drawing data. At a first timing in which at least two or more markers are detected from the video footage captured by the camera while moving within the premises of the building or civil engineering structure, a plurality of drawing markers corresponding to some or all of the detected markers are extracted from the drawing data, and the self-position at the first timing is determined based on the position information of the extracted plurality of drawing markers and the self-motion information detected by the acceleration sensor. At a second timing in which a new marker is detected from the video footage captured by the camera after it has moved to a different location, a plurality of drawing markers corresponding to the newly detected marker and two or more markers included in the markers detected up to the time of detection are again extracted from the drawing data, and the self-position at the second timing is determined based on the position information of the extracted plurality of drawing markers and the self-motion information detected by the acceleration sensor. Based on the determined self-position of the mobile communication terminal, the video space captured by the camera and the drawing space represented by the drawing data are compared and output. Furthermore, the mobile communication terminal extracts a first drawing marker and a second drawing marker from the drawing data at the first timing or the second timing, searches the drawing data for a third drawing marker that satisfies predetermined conditions from among the drawing markers that are different from both the extracted first drawing marker and the second drawing marker, and if it is not possible to find a third drawing marker that satisfies the predetermined conditions, it uses the position information relating to the two points of the first drawing marker and the second drawing marker, and its own motion information detected by the acceleration sensor, to determine the third drawing marker. The system determines its own position, and when it finds the third drawing marker that satisfies the predetermined conditions, it determines its own position based on the positional information of the three points, the first drawing marker, the second drawing marker, and the third drawing marker, and the motion information of the system detected by the acceleration sensor. The predetermined conditions are that, for the triangle formed by the three points, the first drawing marker, the second drawing marker, and the third drawing marker, the angle of the interior angle with the first drawing marker as the vertex and the angle of the interior angle with the second drawing marker as the vertex are both within a predetermined range. A management support system for buildings or civil engineering structures is provided, characterized by the following features. Furthermore, according to the present invention, a building or civil engineering structure management support system for supporting the management of a building or civil engineering structure in which markers are installed at multiple locations comprises a storage device that stores drawing data of the building or civil engineering structure, and a mobile communication terminal that is communicably connected to the storage device, wherein the drawing data does not contain location information of drawing markers corresponding to markers placed on the building or civil engineering structure before the mobile communication terminal moves within the premises of the building or civil engineering structure, the mobile communication terminal has a camera and an acceleration sensor, and when the marker detected by image recognition processing on the image captured by the camera while moving within the premises of the building or civil engineering structure is the first marker, the first marker is registered at the origin of the coordinate space in the drawing data, and when the marker detected by image recognition processing on the image captured by the camera while the building or civil engineering structure is moving is not the first marker and the drawing marker corresponding to that marker is not registered in the drawing data, the location information of other markers detected before that marker is used, Based on the motion information of the mobile communication terminal detected by the acceleration sensor, the mobile communication terminal's own position and the position information of the marker are determined, the drawing marker corresponding to the marker is registered in the drawing data, at a first timing when at least two or more markers are detected from the video footage taken by the camera while moving within the premises of the building or civil engineering structure, a plurality of drawing markers corresponding to some or all of the detected markers are extracted from the drawing data, the self-position at the first timing is determined based on the position information of the extracted plurality of drawing markers and the motion information of the mobile communication terminal detected by the acceleration sensor, at a second timing when a new marker is detected from the video footage taken by the camera that has moved to a different location after the first timing, a plurality of drawing markers corresponding to two or more markers included in the newly detected marker and the markers detected up to the detection of the new marker are again extracted from the drawing data, and the self-position at the second timing is determined based on the position information of the extracted plurality of drawing markers and the motion information of the mobile communication terminal detected by the acceleration sensor.Based on the determined self-position of the mobile communication terminal, the mobile communication terminal compares the video space captured by the camera with the drawing space represented by the drawing data and outputs the result. Furthermore, the mobile communication terminal designates the marker corresponding to the first drawing marker as the first marker, the marker corresponding to the second drawing marker as the second marker, and the marker corresponding to the third drawing marker as the third marker, and the vector from the first drawing marker to the second drawing marker in the drawing space as the drawing reference vector, and the vector from the first marker to the second marker in the video space as the video reference vector. At the first timing or the second timing, the first drawing marker and the second drawing marker are extracted from the drawing data, and the third drawing marker that satisfies predetermined conditions is searched from the drawing data among the drawing markers that are different from both the extracted first drawing marker and the second drawing marker. If the third drawing marker that satisfies the predetermined conditions cannot be found, the position of the first marker and the first drawing marker A management support system for buildings or civil engineering structures is provided, characterized by performing a first process of translating the video space and the drawing space so that the positions of the first marker and the first drawing marker overlap, and a second process of aligning the video reference vector and the drawing reference vector by rotating around the position where the first marker and the first drawing marker overlap as a result of the first process, thereby comparing the video space captured by the camera with the drawing space represented by the drawing data, and when the third drawing marker that satisfies the predetermined conditions can be found, the first and second processes are executed, and further, a third process is executed of aligning a vector that passes through the third marker and intersects the video reference vector perpendicularly with the video reference vector and the drawing reference vector perpendicularly with the third drawing marker by rotating around the axis of the video reference vector and the drawing reference vector aligned as a result of the second process, thereby comparing the video space captured by the camera with the drawing space represented by the drawing data.

[0010] According to the present invention, a mobile communication terminal moves within the premises of a building or civil engineering structure, each of which has markers installed at multiple locations, wherein the mobile communication terminal can refer to drawing data of the building or civil engineering structure stored in a storage device that is integrated with or externally attached to the device, and the drawing data includes location information of drawing markers corresponding to at least some of the markers placed on the building or civil engineering structure, before the mobile communication terminal moves within the premises of the building or civil engineering structure, and the mobile communication terminal has a camera and an acceleration sensor, and detects markers by image recognition processing of images captured by the camera when moving within the premises of the building or civil engineering structure. At a first timing in which at least two or more markers are detected from the captured video, a plurality of drawing markers corresponding to some or all of the detected markers are extracted from the drawing data, and the self-position at the first timing is determined based on the position information of the extracted plurality of drawing markers and the self-motion information detected by the acceleration sensor. At a second timing in which a new marker is detected from the video captured by the camera that has moved to a different location after the first timing, a plurality of drawing markers corresponding to the newly detected marker and two or more markers included in the markers detected up to the detection of the said marker are again extracted from the drawing data, and the self-position at the second timing is determined based on the position information of the extracted plurality of drawing markers and the self-motion information detected by the acceleration sensor. Based on the determined self-position of the mobile communication terminal, the video space captured by the camera and the drawing space represented by the drawing data are compared and output. Furthermore, the mobile communication terminal extracts a first drawing marker and a second drawing marker from the drawing data at the first timing or the second timing, searches the drawing data for a third drawing marker that satisfies predetermined conditions from among the drawing markers that are different from both the extracted first drawing marker and the second drawing marker, and if it is not possible to find a third drawing marker that satisfies the predetermined conditions, it uses the position information relating to the two points of the first drawing marker and the second drawing marker, and its own motion information detected by the acceleration sensor, to determine the third drawing marker. The system determines its own position, and when it finds the third drawing marker that satisfies the predetermined conditions, it determines its own position based on the positional information of the three points, the first drawing marker, the second drawing marker, and the third drawing marker, and the motion information of the system detected by the acceleration sensor. The predetermined conditions are that, for the triangle formed by the three points, the first drawing marker, the second drawing marker, and the third drawing marker, the angle of the interior angle with the first drawing marker as the vertex and the angle of the interior angle with the second drawing marker as the vertex are both within a predetermined range. A mobile communication terminal characterized by the above is provided. Furthermore, according to the present invention, a mobile communication terminal moves within the premises of a building or civil engineering structure in which markers are installed at multiple locations, wherein the mobile communication terminal can refer to drawing data of the building or civil engineering structure stored in a storage device that is integrally provided with the device or attached externally, and the drawing data contains location information of drawing markers corresponding to at least some of the markers placed on the building or civil engineering structure before the mobile communication terminal moves within the premises of the building or civil engineering structure, the mobile communication terminal has a camera and an acceleration sensor, detects markers by image recognition processing of video footage captured by the camera when moving within the premises of the building or civil engineering structure, and at a first timing in which at least two or more markers are detected from the captured video, extracts a plurality of drawing markers corresponding to some or all of the detected markers from the drawing data, and based on the location information of the extracted plurality of drawing markers and its own motion information detected by the acceleration sensor, it performs self At a second timing in which a new marker is detected from the image captured by the camera, which has moved to a different location after the first timing, the mobile communication terminal determines its position and extracts from the drawing data a plurality of drawing markers corresponding to two or more markers included in the newly detected marker and the markers detected up to the time of detection of the said marker. Based on the position information of the plurality of drawing markers extracted again and the motion information of the mobile communication terminal detected by the acceleration sensor, the mobile communication terminal determines its own position at the second timing, and outputs the image space captured by the camera and the drawing space represented by the drawing data based on the determined self-position of the mobile communication terminal. Furthermore, the mobile communication terminal designates the marker corresponding to the first drawing marker as the first marker, the marker corresponding to the second drawing marker as the second marker, and the marker corresponding to the third drawing marker as the third marker, and the vector from the first drawing marker to the second drawing marker in the drawing space as the drawing reference vector, and the vector from the first marker to the second marker in the image space as the image reference vector, at the first timing or the second timing,The first drawing marker and the second drawing marker are extracted from the drawing data, and a third drawing marker that satisfies predetermined conditions is searched from the drawing data among the drawing markers that are different from both the extracted first drawing marker and the second drawing marker. If a third drawing marker that satisfies the predetermined conditions cannot be found, the following are performed: a first process in which the position of the first marker and the position of the first drawing marker are translated parallel to each other, and a second process in which the image reference vector and the drawing reference vector are made to match by rotational movement around the position in which the first marker and the first drawing marker overlap as a result of the first process. A mobile communication terminal is provided, characterized by comparing the video space captured by the camera with the drawing space represented by the drawing data, and when the third drawing marker that satisfies the predetermined conditions is found, executing the first and second processes, and further executing a third process to match a vector that passes through the third marker and intersects the video reference vector perpendicularly with the drawing reference vector, which were matched in the second process, by rotational movement around the drawing reference vector, thereby matching the video space captured by the camera with the drawing space represented by the drawing data.

[0011] According to the present invention, a mobile communication terminal moves within the premises of a building or civil engineering structure, each of which has markers installed at multiple locations, wherein the mobile communication terminal can refer to drawing data of the building or civil engineering structure stored in a storage device that is integrated with the device or attached externally, and the drawing data does not contain location information of drawing markers corresponding to markers placed on the building or civil engineering structure before the mobile communication terminal moves within the premises of the building or civil engineering structure, and the mobile communication terminal has a camera and an acceleration sensor, and when moving within the premises of the building or civil engineering structure, the camera captures images. When the marker detected by image recognition processing on the captured video is the first marker, the first marker is registered at the origin of the coordinate space in the drawing data. When the marker detected by image recognition processing on the video captured by the camera while the building or civil engineering structure is moving is not the first marker and a drawing marker corresponding to that marker is not registered in the drawing data, the mobile communication terminal's own position and the position information of the marker are determined based on the position information of other markers detected before that marker and its own motion information detected by the acceleration sensor, and the drawing marker corresponding to that marker is registered in the drawing data. At a first timing in which at least two or more markers are detected from the video footage captured by the camera while moving within the premises of the building or civil engineering structure, a plurality of drawing markers corresponding to some or all of the detected markers are extracted from the drawing data, and the self-position at the first timing is determined based on the position information of the extracted plurality of drawing markers and the self-motion information detected by the acceleration sensor. At a second timing in which a new marker is detected from the video footage captured by the camera after it has moved to a different location, a plurality of drawing markers corresponding to the newly detected marker and two or more markers included in the markers detected up to the time of detection are again extracted from the drawing data, and the self-position at the second timing is determined based on the position information of the extracted plurality of drawing markers and the self-motion information detected by the acceleration sensor. Based on the determined self-position of the mobile communication terminal, the video space captured by the camera and the drawing space represented by the drawing data are compared and output. Furthermore, the mobile communication terminal extracts a first drawing marker and a second drawing marker from the drawing data at the first timing or the second timing, searches the drawing data for a third drawing marker that satisfies predetermined conditions from among the drawing markers that are different from both the extracted first drawing marker and the second drawing marker, and if it is not possible to find a third drawing marker that satisfies the predetermined conditions, it uses the position information relating to the two points of the first drawing marker and the second drawing marker, and its own motion information detected by the acceleration sensor, to determine the third drawing marker. The system determines its own position, and when it finds the third drawing marker that satisfies the predetermined conditions, it determines its own position based on the positional information of the three points, the first drawing marker, the second drawing marker, and the third drawing marker, and the motion information of the system detected by the acceleration sensor. The predetermined conditions are that, for the triangle formed by the three points, the first drawing marker, the second drawing marker, and the third drawing marker, the angle of the interior angle with the first drawing marker as the vertex and the angle of the interior angle with the second drawing marker as the vertex are both within a predetermined range. A mobile communication terminal characterized by the above is provided. Furthermore, according to the present invention, a mobile communication terminal moves within the premises of a building or civil engineering structure in which markers are installed at multiple locations, wherein the mobile communication terminal can refer to drawing data of the building or civil engineering structure stored in a storage device that is integrally provided with the device or attached externally, the drawing data does not contain location information of drawing markers corresponding to markers placed on the building or civil engineering structure before the mobile communication terminal moves within the premises of the building or civil engineering structure, the mobile communication terminal has a camera and an acceleration sensor, when the marker detected by image recognition processing on the image captured by the camera while moving within the premises of the building or civil engineering structure is the first marker, the first marker is registered at the origin of the coordinate space in the drawing data, and when the marker detected by image recognition processing on the image captured by the camera while the building or civil engineering structure is moving is not the first marker and the drawing marker corresponding to that marker is not registered in the drawing data, the location information of other markers detected before that marker and Based on the motion information of the mobile communication terminal detected by the acceleration sensor, the mobile communication terminal's own position and the position information of the marker are determined, the drawing marker corresponding to the marker is registered in the drawing data, at a first timing when at least two or more markers are detected from the video footage taken by the camera while moving within the premises of the building or civil engineering structure, a plurality of drawing markers corresponding to some or all of the detected markers are extracted from the drawing data, the self-position at the first timing is determined based on the position information of the extracted plurality of drawing markers and the motion information of the mobile communication terminal detected by the acceleration sensor, at a second timing when a new marker is detected from the video footage taken by the camera that has moved to a different location after the first timing, a plurality of drawing markers corresponding to two or more markers included in the newly detected marker and the markers detected up to the time of detection are again extracted from the drawing data, and the self-position at the second timing is determined based on the position information of the extracted plurality of drawing markers and the motion information of the mobile communication terminal detected by the acceleration sensor.Based on the determined self-position of the mobile communication terminal, the mobile communication terminal compares the video space captured by the camera with the drawing space represented by the drawing data and outputs the result. Furthermore, the mobile communication terminal designates a marker corresponding to the first drawing marker as the first marker, a marker corresponding to the second drawing marker as the second marker, and a marker corresponding to the third drawing marker as the third marker, and uses the vector from the first drawing marker to the second drawing marker in the drawing space as the drawing reference vector, and the vector from the first marker to the second marker in the video space as the video reference vector. At the first timing or the second timing, the mobile communication terminal extracts the first drawing marker and the second drawing marker from the drawing data, searches the drawing data for a third drawing marker that satisfies predetermined conditions from among the drawing markers that are different from both the extracted first drawing marker and the second drawing marker, and if it is not possible to find a third drawing marker that satisfies the predetermined conditions, the position of the first marker and the A mobile communication terminal is provided, characterized by performing a first process of translating the video space and the drawing space so that the position of a drawing marker coincides with that of a drawing marker, and a second process of aligning the video reference vector and the drawing reference vector by rotating around the position where the first marker and the first drawing marker coincide as a result of the first process, thereby comparing the video space captured by the camera with the drawing space represented by the drawing data, and when the third drawing marker that satisfies the predetermined conditions can be found, the first and second processes are executed, and further, a third process is performed to align a vector that passes through the third marker and intersects the video reference vector perpendicularly with the drawing reference vector by rotating around the video reference vector and the drawing reference vector aligned as a axis in the second process, thereby comparing the video space captured by the camera with the drawing space represented by the drawing data.

[0012] The mobile communication terminal according to the above invention can move while comparing a marker in the shooting space with a drawing marker in the drawing space, so it can patrol the premises of a building or civil engineering structure that is to be managed while maintaining a certain level of accuracy in comparing the image space and the drawing space. [Effects of the Invention]

[0013] According to the present invention, there is provided a building or civil engineering structure management support system that prevents the accuracy of self-position determination from decreasing as a mobile communication terminal that captures images of the premises of a building or civil engineering structure to be managed moves, or a mobile communication terminal used in the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] [Figure 1] It is a configuration diagram of the management support system in the embodiment of the present invention. [Figure 2] It is a perspective view of the self-propelled robot illustrated in FIG. 1. [Figure 3] It is a diagram showing a specific example of an image displayed on a touch panel of a mobile communication terminal. [Figure 4] It is a flowchart in the case where the mobile communication terminal starts patrolling from a state where no marker corresponding to a drawing marker included in drawing data is registered. [Figure 5] It is a flowchart in the case where the mobile communication terminal starts patrolling from a state where one marker corresponding to a drawing marker included in drawing data is registered. [Figure 6] It is a flowchart in the case where the mobile communication terminal starts patrolling from a state where two markers corresponding to drawing markers included in drawing data are registered. [Figure 7] It is a flowchart in the case where the mobile communication terminal starts patrolling from a state where three or more markers corresponding to drawing markers included in drawing data are registered. [Figure 8] It is a schematic diagram showing the concept of self-position estimation based on one marker. [Figure 9] It is a schematic diagram showing the concept of self-position estimation based on two markers. [Figure 10] It is a schematic diagram showing the concept of self-position estimation based on three markers. [Figure 11] It is a schematic diagram showing criteria for selectively using self-position estimation based on two markers and self-position estimation based on three markers. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0015] Embodiments of the present invention will be described below with reference to the drawings. In all drawings, the same reference numerals are used for similar components, and their descriptions are omitted where appropriate.

[0016] <System Configuration of Management Support System 100> First, I will explain the system configuration of the management support system 100. Figure 1 is a diagram showing the configuration of the management support system 100 in an embodiment of the present invention. Figure 2 is a perspective view of the self-propelled robot 110 shown in Figure 1.

[0017] The building or civil engineering structure management support system according to the present invention (hereinafter simply referred to as the management support system 100) supports the management of a building or civil engineering structure in which markers are installed at multiple locations. As shown in Figure 1, the management support system 100 includes a self-propelled robot 110 and a management device 120.

[0018] The self-propelled robot 110 is equipped with a sensor 111, a laser scanner 112, and a mobile communication terminal 113, and is configured to be able to move autonomously over buildings or civil engineering structures that are to be managed. Sensor 111 is a distance measuring sensor that detects the distance to objects around the self-propelled robot 110, and is implemented using an infrared sensor or the like. The laser scanner 112 measures the shape of objects in the surrounding area by irradiating a laser around the self-propelled robot 110 and acquires three-dimensional point cloud data. The mobile communication terminal 113 is connected to the management device 120 in a communication manner. The communication terminal 113 has a touch panel and functions as an operation reception means for receiving operations from worker WK, and as a display means for displaying the contents of various data in a way that is visible to worker WK. For example, the communication terminal 113 can transmit control commands generated based on the worker WK's operations to the self-propelled robot 110, or read and display data files stored in the storage device 122 specified by the worker WK's operations. Furthermore, the mobile communication terminal 113 has a camera (not shown) and an accelerometer, and can capture images within the field of view of the camera, and can detect its own motion information (such as speed and posture) using the accelerometer.

[0019] The self-propelled robot 110 may autonomously control its own movement. Here, "the self-propelled robot 110 autonomously controls its own movement" means that the self-propelled robot 110 processes data acquired by its onboard components (including photographic data taken by the mobile communication terminal 113 and drawing data received from the storage device 122, etc.) to determine a travel path and then travels accordingly. Alternatively, the movement of the self-propelled robot 110 may be remotely controlled from the control device 120, or controlled based on the operations of the accompanying operator WK, or it may be realized by combining these controls with autonomous control of the self-propelled robot 110.

[0020] In Figure 2, the self-propelled robot 110 is shown as moving on wheels, but the present invention may also be implemented by mounting a mobile communication terminal 113 on a robot that moves through the air like a drone. Alternatively, the present invention may be implemented by having worker WK carry a mobile communication terminal 113 without using a robot.

[0021] The management device 120 comprises a management server 121, a storage device 122, and a display device 123. The storage device 122 stores at least drawing data of the building or civil engineering structure that is being managed. Here, the drawing data stored in the storage device 122 is preferably three-dimensional drawing data, but two-dimensional drawing data is also acceptable. Furthermore, the memory device 122 stores various data acquired by the self-propelled robot 110 (such as video data captured by the camera). The management server 121 processes various data stored in the storage device 122. The management server 121 can also accept operations from an operator (not shown). The display device 123 functions as a user interface for the management server 121 and can display information about the processes performed by the management server 121. For example, the display device 123 may also display video data received by the management server 121 from the self-propelled robot 110.

[0022] The drawing data stored in the memory device 122 includes location information of drawing markers corresponding to markers placed on the building or civil engineering structure. Here, a marker is a flat plate placed in real space, on which a two-dimensional code detectable by image recognition processing is printed. As a result, when a marker appears in the image captured by the onboard camera, the mobile communication terminal 113 can detect the marker and obtain the identification information represented by the two-dimensional code of that marker. In the following explanation, the identification information assigned to a marker refers to the identification information represented by the two-dimensional code printed on that marker. In this embodiment, when the mobile communication terminal 113 searches for a drawing marker registered in the drawing data, it extracts the drawing marker that corresponds to the identification information assigned to the marker detected from the captured video as the drawing marker corresponding to that marker. Furthermore, a drawing marker is placed in the drawing space (a virtual space in the data) represented by the drawing data of the building or civil engineering structure being managed, and has location information in that drawing space. When the mobile communication terminal 113 searches for a drawing marker registered in the drawing data, and there is no drawing marker in the drawing data that corresponds to the identification information assigned to the marker detected from the captured video, it refers to the location information of the drawing marker and extracts a drawing marker located near the detection position of the marker and that is not associated with identification information as the drawing marker corresponding to the marker. At this time, the mobile communication terminal 113 requests the management device 120 to associate the identification information assigned to the extracted drawing marker, and updates the drawing data in the storage device 122. Furthermore, the mobile communication terminal 113 refers to the location information of the drawing marker and determines that if there is no drawing marker located near the detection position of the detected marker, or if a different identification information is associated with and registered for a drawing marker located near the detection position of the detected marker, then a drawing marker corresponding to that marker is not registered in the drawing data. At this time, the mobile communication terminal 113 requests the management device 120 to register the drawing marker associated with the identification information assigned to that marker as the drawing marker corresponding to that marker in the drawing data, and updates the drawing data in the storage device 122. In this embodiment, the state in which identification information assigned to a specific marker in the drawing data stored in the storage device 122 is associated with a specific drawing marker may be expressed as "a drawing marker corresponding to a marker is registered in the drawing data."

[0023] The self-propelled robot 110 (mobile communication terminal 113) may patrol the premises of the building or civil engineering structure to be managed, starting from a state in which drawing markers corresponding to at least some of the markers placed within the premises of the building or civil engineering structure to be managed are registered. In other words, the drawing data stored in the storage device 122 may have location information of drawing markers corresponding to at least some of the markers placed within the building or civil engineering structure registered in the drawing data before the self-propelled robot 110 (mobile communication terminal 113) moves within the premises of the building or civil engineering structure. When the mobile communication terminal 113 begins patrolling the premises in the state described above, if it detects a marker from the video captured by the camera, it extracts a drawing marker corresponding to that marker from the drawing data stored in the storage device 122, determines the position of the mobile communication terminal 113 based on the position information of the extracted drawing marker and its own motion information detected by the acceleration sensor, and then compares the video space captured by the camera with the drawing space represented by the drawing data based on the determined position of the mobile communication terminal 113 and outputs the result.

[0024] Figure 3 shows a specific example of an image displayed on the touch panel of the mobile communication terminal 113. Figure 3(a) shows the display before the image space and drawing space are compared, and Figure 3(b) shows the display after the image space and drawing space are compared. As shown in Figure 3(a), when the mobile communication terminal 113 detects a marker MK from the video footage captured by the camera while patrolling the premises, it performs the above-described process to compare the video space with the drawing space represented by the drawing data stored in the storage device 122. Then, as shown in Figure 3(b), the mobile communication terminal 113 aligns the drawing marker DM, which corresponds to the marker MK shown in the video, with the video marker MK, so that it is superimposed on the marker MK, and displays an augmented reality image of the structure OB (which exists in the drawing space but not in the real space) located near the drawing marker DM on the touch panel. By displaying this type of output, it is possible to allow viewers (such as workers WK) to experience the completed state of the designed structure in real space.

[0025] In implementing the present invention, the output mode of the mobile communication terminal 113 is not limited to the mode shown in Figure 3, as long as it is possible for a person to visually compare the video space captured by the camera with the drawing space represented by the drawing data. Furthermore, the output from the mobile communication terminal 113 is not limited to the display output (display on the touch panel) as described above, but may also be a transmission output to a communication-capable external device (for example, the management device 120).

[0026] Here, if the mobile communication terminal 113 detects a new marker from the image captured by the camera, and a drawing marker corresponding to that new marker is registered in the drawing data, the mobile communication terminal 113 can determine its own position by processing as described above. On the other hand, if the mobile communication terminal 113 detects a new marker from the image captured by the camera, and a drawing marker corresponding to the new marker is not registered in the drawing data, it cannot process the data as described above. Therefore, the mobile communication terminal 113 determines its own position and the position information of the new marker based on the position information of other markers detected before the new marker and its own motion information detected by the acceleration sensor, and registers a drawing marker corresponding to the new marker in the drawing data (it assigns the identification information of the new marker to the drawing marker and stores it in the drawing data).

[0027] The self-propelled robot 110 (mobile communication terminal 113) may patrol the premises of a building or civil engineering structure that is to be managed, even if no drawing markers corresponding to markers placed within the premises of the building or civil engineering structure are registered. In other words, the drawing data stored in the storage device 122 does not need to contain location information of drawing markers corresponding to markers placed within the building or civil engineering structure before the self-propelled robot 110 (mobile communication terminal 113) moves within the premises of the building or civil engineering structure. If the mobile communication terminal 113 begins patrolling the premises under the conditions described above, and the marker detected from the video captured by the camera is the first marker, the mobile communication terminal 113 registers the first marker at the origin of the coordinate space in the drawing data. If the marker detected from the video captured by the camera is not the first marker and the corresponding drawing marker is not registered in the drawing data, the mobile communication terminal 113 determines its own position and the position information of the marker based on the position information of other markers detected before the marker and its own motion information detected by the acceleration sensor, and registers the corresponding drawing marker in the drawing data. Then, the mobile communication terminal 113 compares the video space captured by the camera with the drawing space represented by the drawing data based on the determined position of the mobile communication terminal 113, and outputs the mobile communication terminal 113's own position.

[0028] As described above, the mobile communication terminal 113 can move while comparing markers in the video space with drawing markers in the drawing space, so it can patrol the premises of the building or civil engineering structure under management while maintaining a certain level of accuracy in comparing the video space and the drawing space. Furthermore, when the mobile communication terminal 113 detects a marker that is not registered in the drawing data, it can register that marker as it tours the area. Therefore, it is not necessary to register all the corresponding drawing markers in the drawing data before the tour begins. Consequently, the work costs associated with prior preparation can be reduced.

[0029] <Regarding the handling of markers discovered through patrols by mobile communication terminal 113> Next, the handling of markers discovered by the patrol of the mobile communication terminal 113 will be explained using the flowcharts in Figures 4 to 7.

[0030] Figure 4 is a flowchart showing the case when the mobile communication terminal 113 starts its patrol from a state in which no markers corresponding to the drawing markers included in the drawing data are registered (a state in which there are zero registered markers). As described above, the patrol begins, and the first marker is detected from the video being captured by the mobile communication terminal 113. In this embodiment, the detection of a marker from the video being captured by the mobile communication terminal 113 may simply be described as the mobile communication terminal 113 discovering a marker. At this time, the mobile communication terminal 113 considers the location of the discovered marker to be the origin in the drawing space and updates the drawing data stored in the storage device 122 to register the drawing marker to which the identification information of the marker is assigned at the origin in the drawing space (step S101). Then, the mobile communication terminal 113 determines its own position by calculating its relative positional relationship with the origin of the drawing space based on distance information to the discovered marker (measurement results from sensor 111 and laser scanner 112) and motion information of the mobile communication terminal 113 (its own attitude and speed detected by the acceleration sensor of the mobile communication terminal 113) (step S102). Here, the self-position estimation process in step S102 corresponds to the "self-position estimation based on one marker" described later.

[0031] Next, it is assumed that the mobile communication terminal 113 has moved from the position where it discovered the marker in step S101, and has since discovered a new marker. At this time, if the newly discovered marker is a marker already registered in the drawing data, that is, if it is the same as the marker registered in step S101 (step S103), the mobile communication terminal 113 estimates its own position based on the discovered marker without updating the drawing data stored in the storage device 122 (step S104). Here, the self-position estimation process in step S104 corresponds to the "self-position estimation based on a single marker" described later. Alternatively, if a newly discovered marker is not registered in the drawing data, that is, if a marker different from the one registered in step S101 is discovered, the mobile communication terminal 113 updates the drawing data stored in the storage device 122 to register the drawing marker to which the identification information of the discovered marker is assigned at the position in the drawing space corresponding to the position of the discovered marker (step S105). The position of the discovered marker in step S105 is calculated using the direction and distance traveled by the mobile communication terminal 113 from its own position determined in step S102. Furthermore, the mobile communication terminal 113 estimates its own position based on the marker discovered in step S101 and the marker discovered in step S105 (step S106). Here, the self-position estimation process in step S106 corresponds to the "self-position estimation based on two markers" described later.

[0032] If the mobile communication terminal 113 discovers a new marker after the process in step S104 described above, the process returns to step S103 or step S105.

[0033] Next, it is assumed that the mobile communication terminal 113 has moved from the position where it discovered the marker in step S105, and has since discovered a new marker. At this time, if the newly discovered marker is a marker already registered in the drawing data, that is, if it is the same as the marker registered in step S101 or step S105 (step S107), the mobile communication terminal 113 estimates its own position based on the two markers registered up to that point without updating the drawing data stored in the storage device 122 (step S108). Here, the self-position estimation process in step S108 corresponds to the "self-position estimation based on two markers" described later. Alternatively, if a newly discovered marker is not registered in the drawing data, that is, if a marker different from the markers registered in steps S101 and S105 is discovered, the mobile communication terminal 113 updates the drawing data stored in the storage device 122 to register the drawing marker to which the identification information of the discovered marker is assigned at the position in the drawing space corresponding to the position of the discovered marker (step S109). The method for calculating the position of the discovered marker in step S109 is the same as the calculation method described in step S105. Furthermore, the mobile communication terminal 113 selects at least two markers from those already registered (markers discovered in steps S101, S105, and S109) and estimates its own position based on the selected markers (step S110). Here, the self-position estimation process in step S110 corresponds to "self-position estimation based on two markers" described later if two markers are selected, and to "self-position estimation based on three markers" described later if three markers are selected.

[0034] If the mobile communication terminal 113 discovers a new marker after the process in step S108 described above, the process returns to step S107 or step S109.

[0035] Next, it is assumed that the mobile communication terminal 113 has moved from the position where it discovered the marker in step S109, and has since discovered a new marker. At this time, if the newly discovered marker is a marker already registered in the drawing data (step S111), the mobile communication terminal 113 selects at least two markers from those previously registered without updating the drawing data stored in the storage device 122, and estimates its own position based on the selected markers (step S113). Alternatively, if a newly discovered marker is a marker not yet registered in the drawing data, the mobile communication terminal 113 updates the drawing data stored in the storage device 122 to register a drawing marker with the identification information of the discovered marker assigned to the position in the drawing space corresponding to the position of the discovered marker (step S112). The method for calculating the position of the discovered marker in step S112 is the same as the calculation method described in steps S105 and S109. Furthermore, the mobile communication terminal 113 selects at least two markers from those already registered and estimates its own position based on the selected markers (step S113). Here, the self-position estimation process in step S113 corresponds to "self-position estimation based on two markers" described later if two markers are selected, and to "self-position estimation based on three markers" described later if three markers are selected.

[0036] If the mobile communication terminal 113 discovers a new marker after the process in step S113 described above, the process returns to step S111 or step S112.

[0037] Figure 5 is a flowchart showing the case when the mobile communication terminal 113 starts its patrol with one marker corresponding to a drawing marker included in the drawing data already registered. Let's assume that the patrol begins under the conditions described above, and that the mobile communication terminal 113 discovers the first marker. At this time, if the newly discovered marker is a marker not registered in the drawing data, that is, if a marker different from the markers registered in advance of the patrol is discovered, the mobile communication terminal 113 only records the real-world position of the discovered marker and does not update the drawing data in the storage device 122 (step S201). This is because the mobile communication terminal 113 does not have enough information at this point to estimate its own position in the drawing space, and therefore cannot register the marker discovered in step S201 in the drawing data. Alternatively, if a newly discovered marker is a marker already registered in the drawing data, that is, the same as a marker registered in advance of the patrol (step S202), the mobile communication terminal 113 estimates its own position based on the discovered marker without updating the drawing data stored in the storage device 122 (step S203). Here, the self-position estimation process in step S203 corresponds to the "self-position estimation based on a single marker" described later.

[0038] Next, it is assumed that the mobile communication terminal 113 has moved from the position where it discovered the marker in step S202, and has since discovered a new marker. At this time, if the newly discovered marker is a marker already registered in the drawing data, that is, the same as a marker registered in advance of the patrol (step S204), the mobile communication terminal 113 estimates its own position based on the discovered marker without updating the drawing data stored in the storage device 122 (step S205). Here, the self-position estimation process in step S205 corresponds to the "self-position estimation based on a single marker" described later. Alternatively, if a newly discovered marker is not registered in the drawing data, that is, if a marker different from those registered in advance of the patrol is discovered, the mobile communication terminal 113 updates the drawing data stored in the storage device 122 to register the drawing marker to which the identification information of the discovered marker is assigned at the position in the drawing space corresponding to the position of the discovered marker (step S206). The position of the discovered marker in step S206 is calculated using the direction and distance traveled by the mobile communication terminal 113 from its own position determined in step S203. Furthermore, the mobile communication terminal 113 estimates its own position based on the markers registered in advance for the patrol and the marker discovered in step S206 (step S207). Here, the self-position estimation process in step S207 corresponds to the "self-position estimation based on two markers" described later.

[0039] If the mobile communication terminal 113 discovers a new marker after the process in step S205 described above, the process returns to step S204 or step S206.

[0040] Next, it is assumed that the mobile communication terminal 113 has moved from the position where it discovered the marker in step S206, and has since discovered a new marker. At this time, if the newly discovered marker is a marker already registered in the drawing data, that is, if it is the same as a marker registered in advance of the tour or a marker discovered in step S206 (step S208), the mobile communication terminal 113 estimates its own position based on the two markers registered up to that point without updating the drawing data stored in the storage device 122 (step S209). Here, the self-position estimation process in step S209 corresponds to the "self-position estimation based on two markers" described later. Alternatively, if a newly discovered marker is not registered in the drawing data, that is, if a marker different from the markers registered in advance of the patrol and the marker discovered in step S206 is discovered, the mobile communication terminal 113 updates the drawing data stored in the storage device 122 to register a drawing marker to which the identification information of the discovered marker is assigned at the position in the drawing space corresponding to the position of the discovered marker (step S210). The method for calculating the position of the discovered marker in step S210 is the same as the calculation method described in step S206. Furthermore, the mobile communication terminal 113 selects at least two markers from among those already registered (markers registered in advance, as well as markers discovered in steps S206 and S210), and estimates its own position based on the selected markers (step S211). Here, the self-position estimation process in step S110 corresponds to "self-position estimation based on two markers" described later when two markers are selected, and to "self-position estimation based on three markers" described later when three markers are selected.

[0041] If the mobile communication terminal 113 discovers a new marker after the process in step S209 described above, the process returns to step S208 or step S210.

[0042] Next, it is assumed that the mobile communication terminal 113 has moved from the position where it discovered the marker in step S210, and has since discovered a new marker. At this time, if the newly discovered marker is a marker already registered in the drawing data (step S212), the mobile communication terminal 113 selects at least two markers from those previously registered without updating the drawing data stored in the storage device 122, and estimates its own position based on the selected markers (step S214). Alternatively, if a newly discovered marker is a marker not yet registered in the drawing data, the mobile communication terminal 113 updates the drawing data stored in the storage device 122 to register a drawing marker with the identification information of the discovered marker assigned to the position in the drawing space corresponding to the position of the discovered marker (step S213). The method for calculating the position of the discovered marker in step S213 is the same as the calculation method described in steps S206 and S210. Furthermore, the mobile communication terminal 113 selects at least two markers from those already registered and estimates its own position based on the selected markers (step S214). Here, the self-position estimation process in step S214 corresponds to "self-position estimation based on two markers" described later if two markers are selected, and to "self-position estimation based on three markers" described later if three markers are selected.

[0043] If the mobile communication terminal 113 discovers a new marker after the processing in step S214 described above, the process returns to step S212 or step S213.

[0044] Figure 6 is a flowchart showing the case when the mobile communication terminal 113 starts its patrol with two markers corresponding to drawing markers included in the drawing data already registered. Let's assume that the patrol begins under the conditions described above, and that the mobile communication terminal 113 discovers the first marker. At this time, if the newly discovered marker is a marker not registered in the drawing data, that is, if a marker different from the markers registered in advance of the patrol is discovered, the mobile communication terminal 113 only records the real-world location of the discovered marker and does not update the drawing data in the storage device 122 (step S301). This is for the same reason as explained in step S201 in Figure 5. Furthermore, if a newly discovered marker is a marker already registered in the drawing data, that is, if it is the same as a marker registered in advance of the patrol, the mobile communication terminal 113 does not update the drawing data stored in the storage device 122, nor does it estimate its own position based on the discovered marker (step S302). Although it is possible to perform "self-position estimation based on a single marker" as described later in step S302, this is not performed in this embodiment.

[0045] Next, it is assumed that the mobile communication terminal 113 has moved from the position where it discovered the marker in step S302, and has since discovered a new marker. At this time, if the newly discovered marker is a marker not registered in the drawing data, that is, a marker different from the markers registered in advance of the patrol, and also different from the marker discovered in step S302, the mobile communication terminal 113 only records the real-world location of the discovered marker and does not update the drawing data in the storage device 122 (step S303). This is for the same reason as explained in step S201 in Figure 5. Alternatively, if the newly discovered marker is a marker already registered in the drawing data, that is, the same as a marker registered in advance of the tour, and is a different marker from the one discovered in step S302 (step S304), the mobile communication terminal 113 estimates its own position based on the discovered marker without updating the drawing data stored in the storage device 122 (step S305). Here, the self-position estimation process in step S305 corresponds to the "self-position estimation based on two markers" described later.

[0046] Next, it is assumed that the mobile communication terminal 113 has moved from the position where it discovered the marker in step S304, and has since discovered a new marker. At this time, if the newly discovered marker is a marker already registered in the drawing data, that is, if it is the same as the marker discovered in step S302 or step S304 (step S306), the mobile communication terminal 113 estimates its own position based on the two markers registered up to that point without updating the drawing data stored in the storage device 122 (step S307). Here, the self-position estimation process in step S307 corresponds to the "self-position estimation based on two markers" described later. Alternatively, if a newly discovered marker is not registered in the drawing data, that is, if a different marker is discovered from the markers discovered in steps S302 and S304, the mobile communication terminal 113 updates the drawing data stored in the storage device 122 to register the drawing marker to which the identification information of the discovered marker is assigned at the position in the drawing space corresponding to the position of the discovered marker (step S308). The position of the discovered marker in step S308 is calculated using the direction and distance traveled by the mobile communication terminal 113 from its own position determined in step S305. Furthermore, the mobile communication terminal 113 selects at least two markers from among those already registered and estimates its own position based on the selected markers (step S309). Here, the self-position estimation process in step S309 corresponds to "self-position estimation based on two markers" described later when two markers are selected, and to "self-position estimation based on three markers" described later when three markers are selected.

[0047] If the mobile communication terminal 113 discovers a new marker after the process in step S307 described above, the process returns to step S306 or step S308.

[0048] Next, it is assumed that the mobile communication terminal 113 has moved from the position where it discovered the marker in step S308, and has since discovered a new marker. At this time, if the newly discovered marker is a marker already registered in the drawing data (step S310), the mobile communication terminal 113 selects at least two markers from those previously registered without updating the drawing data stored in the storage device 122, and estimates its own position based on the selected markers (step S312). Alternatively, if a newly discovered marker is a marker not yet registered in the drawing data, the mobile communication terminal 113 updates the drawing data stored in the storage device 122 to register a drawing marker to which the identification information of the discovered marker is assigned at a position in the drawing space corresponding to the position of the discovered marker (step S311). The method for calculating the position of the discovered marker in step S311 is the same as the calculation method described in step S308. Furthermore, the mobile communication terminal 113 selects at least two markers from those already registered and estimates its own position based on the selected markers (step S312). Here, the self-position estimation process in step S312 corresponds to "self-position estimation based on two markers" described later if two markers are selected, and to "self-position estimation based on three markers" described later if three markers are selected.

[0049] If the mobile communication terminal 113 discovers a new marker after the processing in step S312 described above, the process returns to step S310 or step S311.

[0050] Figure 7 is a flowchart showing the case when the mobile communication terminal 113 starts its patrol when three or more markers corresponding to drawing markers included in the drawing data are registered. Let's assume that the patrol begins under the conditions described above, and that the mobile communication terminal 113 discovers the first marker. At this time, if the newly discovered marker is a marker not registered in the drawing data, that is, if a marker different from the markers registered in advance of the patrol is discovered, the mobile communication terminal 113 only records the real-world location of the discovered marker and does not update the drawing data in the storage device 122 (step S401). This is for the same reason as explained in step S201 in Figure 5. Furthermore, if a newly discovered marker is a marker already registered in the drawing data, that is, if it is the same as a marker registered in advance of the patrol, the mobile communication terminal 113 does not update the drawing data stored in the storage device 122, nor does it estimate its own position based on the discovered marker (step S402). Although it is possible to perform "self-position estimation based on a single marker" as described later in step S402, this is not performed in this embodiment.

[0051] Next, it is assumed that the mobile communication terminal 113 has moved from the position where it discovered the marker in step S402, and has since discovered a new marker. At this time, if the newly discovered marker is a marker not registered in the drawing data, that is, a marker different from the markers registered in advance of the patrol, and also different from the marker discovered in step S402, the mobile communication terminal 113 only records the real-world location of the discovered marker and does not update the drawing data in the storage device 122 (step S403). This is for the same reason as explained in step S201 in Figure 5. Alternatively, if a newly discovered marker is a marker already registered in the drawing data, that is, the same as a marker registered in advance of the tour, and is a different marker from the one discovered in step S402, the mobile communication terminal 113 does not update the drawing data stored in the storage device 122, nor does it estimate its own position based on the discovered marker (step S404). Although it is possible to perform "self-position estimation based on two markers" as described later in step S404, this is not performed in this embodiment.

[0052] Next, it is assumed that the mobile communication terminal 113 has moved from the position where it discovered the marker in step S404, and has since discovered a new marker. At this time, if the newly discovered marker is a marker not registered in the drawing data, that is, a marker different from the markers registered in advance of the tour, and also different from the markers discovered in steps S402 and S404, the mobile communication terminal 113 only records the real-world location of the discovered marker and does not update the drawing data in the storage device 122 (step S405). This is for the same reason as explained in step S201 in Figure 5. Alternatively, if a newly discovered marker is a marker already registered in the drawing data, that is, the same as a marker registered in advance of the tour, and is a different marker from the marker discovered in steps S402 and S404 (step S406), the mobile communication terminal 113 selects at least two markers from those registered up to that point without updating the drawing data stored in the storage device 122, and estimates its own position based on the selected markers (step S407). Here, the self-position estimation process in step S407 corresponds to "self-position estimation based on two markers" described later if two markers are selected, and to "self-position estimation based on three markers" described later if three markers are selected.

[0053] Next, it is assumed that the mobile communication terminal 113 has moved from the position where it discovered the marker in step S406, and has since discovered a new marker. At this time, if the newly discovered marker is a marker already registered in the drawing data (step S408), the mobile communication terminal 113 selects at least two markers from those previously registered without updating the drawing data stored in the storage device 122, and estimates its own position based on the selected markers (step S410). Alternatively, if a newly discovered marker is not registered in the drawing data, the mobile communication terminal 113 updates the drawing data stored in the storage device 122 to register a drawing marker with the identification information of the discovered marker assigned to it at a position in the drawing space corresponding to the position of the discovered marker (step S409). The position of the discovered marker in step S409 is calculated using the direction and distance traveled by the mobile communication terminal 113 from its own position determined in step S407. Furthermore, the mobile communication terminal 113 selects at least two markers from those already registered and estimates its own position based on the selected markers (step S410). Here, the self-position estimation process in step S410 corresponds to "self-position estimation based on two markers" described later if two markers are selected, and to "self-position estimation based on three markers" described later if three markers are selected.

[0054] If the mobile communication terminal 113 discovers a new marker after the process in step S410 described above, the process returns to step S408 or step S409.

[0055] As explained above, when the mobile communication terminal 113 patrols the premises of a building or civil engineering structure that is under its management, it processes the markers it discovers during the patrol using different procedures depending on the number of markers registered in the drawing data in advance of the patrol. If the discovered marker is already registered, the mobile communication terminal 113 estimates its own position using at least the discovered marker (and also taking into account the discovered marker if a registered marker has been discovered). If the discovered marker is not registered, the mobile communication terminal 113 updates the drawing data stored in the storage device 122 to register the marker, and treats it as a registered marker in subsequent processing.

[0056] Furthermore, the management support system 100 can start its patrol from a state where no drawing markers are registered in the storage device 122, and can increase the number of registered markers during the patrol. This allows the system to step up from a state where "self-position estimation based on one marker" is being performed (for example, step S102 in Figure 4) to a state where "self-position estimation based on two markers" is being performed (for example, step S106 in Figure 4), thereby improving the accuracy of matching the drawing space with the real space.

[0057] Then, once the management support system 100 is in a state where it can perform "self-localization based on two markers," it performs self-localization based on multiple markers. In other words, at a first timing (for example, step S106 in Figure 4) when the mobile communication terminal 113 detects at least two markers from the video footage captured by the camera while moving within the premises of a building or civil engineering structure, it extracts multiple drawing markers from the drawing data that correspond to some or all of the detected markers, and determines its own position at the first timing based on the position information of the extracted multiple drawing markers and its own motion information detected by the acceleration sensor. Then, at a second timing (for example, step S110 in Figure 4) when the mobile communication terminal 113 detects a new marker from the video footage captured by the camera that has moved to a different location after the first timing, it again extracts multiple drawing markers from the drawing data that correspond to the newly detected marker and two or more markers included in the markers detected up to the time of detection, and determines its own position at the second timing based on the position information of the extracted multiple drawing markers and its own motion information detected by the acceleration sensor. Because the above processing is performed, the accuracy of estimating the self-position of the mobile communication terminal 113 is maintained to a certain extent, so the management support system 100 can prevent the self-position of the mobile communication terminal 113 from drifting in the drawing space.

[0058] Because it has the above-mentioned features, the management support system 100 can appropriately update the drawing data stored in the storage device 122 (register markers) according to the situation. Furthermore, since the management support system 100 automatically performs the process of registering markers placed in real space into the drawing data in conjunction with the patrol of the self-propelled robot 110 (mobile communication terminal 113), it can reduce the work costs of the worker WK.

[0059] <Regarding the self-position estimation process of the mobile communication terminal 113> Next, the process of estimating the self-position of the mobile communication terminal 113 will be explained using Figures 8 to 11.

[0060] Figure 8 is a schematic diagram illustrating the concept of self-localization based on a single marker. The marker MK1 shown in Figure 8(a) represents a marker found in the video space captured by the camera. The two arrows superimposed on marker MK1 represent the vertical and horizontal directions (two axes intersecting at right angles) detected by image recognition processing based on the shape of marker MK1. The drawing marker DM1 and structure OB shown in Figure 8(a) represent objects (objects in the drawing space) that exist in the drawing data stored in the storage device 122. Figure 8(a) also shows the drawing area DS as the planar region (region within the drawing space) where the drawing marker DM1 and structure OB exist. The numbers depicted within marker MK1 and drawing marker DM1 in Figure 8 represent their respective identification information, with marker MK1 in the video space corresponding to drawing marker DM1 in the drawing space.

[0061] First, the mobile communication terminal 113 overlaps the marker MK1 and the drawing marker DM1 by translating the video space captured by the camera and the drawing space represented by the drawing data (see Figure 8(b)). Next, the mobile communication terminal 113 rotates the drawing space around the Y axis so that the vertical direction of marker MK1 aligns with the Z axis in the drawing space, and the horizontal direction of marker MK1 aligns with the X axis in the drawing space (see Figure 8(c)). By following the above procedure, the mobile communication terminal 113 can determine the location of marker MK1 in the drawing space by comparing it with drawing marker DM1. Then, the mobile communication terminal 113 can estimate its own position based on the position information of marker MK1 in the drawing space and its own attitude information (camera shooting direction) detected by the acceleration sensor.

[0062] Figure 9 is a schematic diagram illustrating the concept of self-localization based on two markers. The markers MK1 and MK2 shown in Figure 9(a) represent markers found within the video space captured by the camera. The vector RB12 shown in Figure 9(a) is a vector pointing from marker MK1 to marker MK2. The drawing markers DM1 and DM2 shown in Figure 9(a) represent objects (objects in the drawing space) that exist in the drawing data stored in the storage device 122. The vector VB12 shown in Figure 9(a) is a vector that points from drawing marker DM1 to drawing marker DM2. The numbers depicted within markers MK1, marker MK2, drawing marker DM1, and drawing marker DM2 shown in Figure 9 are their respective identification information. Marker MK1 in the video space corresponds to drawing marker DM1 in the drawing space, and marker MK2 in the video space corresponds to drawing marker DM2 in the drawing space. Note that the structure OB and drawing area DS shown in Figure 9 are the same as those shown in Figure 8.

[0063] First, the mobile communication terminal 113 overlaps the marker MK1 and the drawing marker DM1 by translating the video space captured by the camera and the drawing space represented by the drawing data (see Figure 9(b)). Next, the mobile communication terminal 113 rotates around the position where marker MK1 and drawing marker DM1 overlap to make vector RB12 and vector VB12 coincide (see Figure 9(c)). By following the above procedure, the mobile communication terminal 113 can compare the video space and the drawing space using the two markers MK1 and MK2. Then, through the above comparison, the mobile communication terminal 113 can determine the positions in the drawing space of markers MK1 and MK2 found in the video space. Furthermore, the mobile communication terminal 113 can estimate its own position in the drawing space based on the position information of markers MK1 and MK2 in the drawing space and its own motion information detected by the acceleration sensor (camera shooting direction and its own movement information).

[0064] Figure 10 is a schematic diagram illustrating the concept of self-localization based on three markers. The markers MK1, MK2, and MK3 shown in Figure 10(a) represent markers found within the video space captured by the camera. The vector RB12 shown in Figure 10(a) is a vector pointing from marker MK1 to marker MK2. Furthermore, a perpendicular line from marker MK3 to vector RB12 is drawn in Figure 10(a). The drawing markers DM1, DM2, and DM3 shown in Figure 10(a) represent objects (objects in the drawing space) that exist in the drawing data stored in the storage device 122. Furthermore, the vector VB12 shown in Figure 10(a) is a vector pointing from drawing marker DM1 to drawing marker DM2. Additionally, a perpendicular line from drawing marker DM3 to vector VB12 is drawn in Figure 10(a). The numbers depicted within markers MK1, MK2, MK3, drawing markers DM1, DM2, and DM3 shown in Figure 10 are their respective identification information. Marker MK1 in the video space corresponds to drawing marker DM1 in the drawing space, marker MK2 in the video space corresponds to drawing marker DM2 in the drawing space, and marker MK3 corresponds to drawing marker DM3. Note that the structure OB and drawing area DS shown in Figure 10 are the same as those shown in Figures 8 and 9.

[0065] First, the mobile communication terminal 113 overlaps the marker MK1 and the drawing marker DM1 by translating the video space captured by the camera and the drawing space represented by the drawing data (see Figure 10(b)). Next, the mobile communication terminal 113 rotates around the point where marker MK1 and drawing marker DM1 overlap to make vector RB12 and vector VB12 coincide (see Figure 10(c)). Then, the mobile communication terminal 113 rotates around the coincident vectors RB12 and VB12 to make the perpendicular line (a perpendicularly intersecting vector) passing through marker MK3 and tolerating vector RB12 coincide with the perpendicular line (a perpendicularly intersecting vector) passing through drawing marker DM3 and tolerating vector VB12 (see Figure 10(d)). Through the above procedure, the mobile communication terminal 113 can compare the video space and the drawing space using the three markers MK1, MK2, and MK3. Based on this comparison, the mobile communication terminal 113 can determine the locations in the drawing space of markers MK1, MK2, and MK3 found in the video space. Furthermore, the mobile communication terminal 113 can estimate its own position in the drawing space based on the positional information of markers MK1, MK2, and MK3 in the drawing space, as well as its own motion information detected by the acceleration sensor (camera shooting direction and its own movement information).

[0066] Figure 11 is a schematic diagram illustrating the criteria for choosing between self-localization based on two markers and self-localization based on three markers. As shown in Figure 11(a), when the process described using Figure 10 is performed based on three markers that are roughly aligned in a straight line, the errors that occur in the position recognition of each marker greatly affect the matching process and hinder accurate self-localization. To avoid the phenomena described above, if three markers discovered by the mobile communication terminal 113 during its patrol are registered, and these three markers are arranged as shown in Figure 11(a), the mobile communication terminal 113 will ignore the third marker it discovers (marker MK3 in Figure 11(a)) and estimate its own position using the "self-position estimation based on two markers" method explained in Figure 9. Subsequently, the mobile communication terminal 113 registers the fourth marker MK4 it discovers during its patrol. If a proper triangle is formed by combining the previously registered markers MK1, MK2, and MK4, as shown in Figure 11(b), the mobile communication terminal 113 estimates its own position using the markers that make up that triangle as a reference, based on the "self-position estimation based on three markers" method described in Figure 10.

[0067] The conditions for distinguishing between the linear shape shown in Figure 11(a) and the triangular shape shown in Figure 11(b) (the "predetermined conditions" according to the present invention) are as follows. When two or more registered markers (marker MK1 and marker MK2 in Figure 11) have already been discovered, the mobile communication terminal 113 determines that the newly discovered and registered marker satisfies the predetermined conditions when, for a triangle formed by the three points of the newly discovered and registered marker (marker MK3 in Figure 11(a), marker MK4 in Figure 11(b)) and the previously discovered markers, both the interior angle with marker MK1 as the vertex and the interior angle with marker MK2 as the vertex fall within a predetermined range (for example, 30 degrees or more and less than 150 degrees). The mobile communication terminal 113 then determines that the newly discovered and registered marker does not satisfy the predetermined conditions when, for the triangle, at least one of the interior angles with marker MK1 as the vertex or marker MK2 as the vertex falls outside the predetermined range (for example, less than 30 degrees or 150 degrees or more). Furthermore, when the mobile communication terminal 113 determines that the above predetermined conditions are met, that is, when two or more registered markers have already been discovered and a newly discovered marker by the mobile communication terminal 113 is registered, steps S109, S111, and S112 of the flowchart in Figure 4 may apply. Also, steps S210, S212, and S213 of the flowchart in Figure 5 may apply. Also, steps S308, S311, and S312 of the flowchart in Figure 6 may apply. Also, steps S406, S408, and S409 of the flowchart in Figure 7 may apply.

[0068] <Variation> The embodiments of the present invention described above can be modified in various ways, within the scope of achieving the objectives of the present invention. The following describes variations of the present invention that have not yet been explained.

[0069] Some of the components included in the system configuration shown in Figure 1 may be omitted, and components not shown may be added. Furthermore, in the system configuration shown in Figure 1, a single component may be implemented using multiple components. For example, the present invention may be implemented by having a device corresponding to the self-propelled robot 110 shown in Figure 1 patrol the premises of a building or civil engineering structure that is to be managed. In this modified example, multiple self-propelled robots 110 (mobile communication terminals 113) may communicate with the management device 120 and refer to the same drawing data stored in the storage device 122. This allows one self-propelled robot to refer to information related to markers that are updated as one self-propelled robot patrols the area.

[0070] In the system configuration shown in Figure 1, one of the configurations shown for the self-propelled robot 110 and the management device 120 may be used for the other configuration. For example, the configuration corresponding to the storage device 122 for storing drawing data may be replaced with a storage means such as ROM built into the mobile communication terminal 113. In other words, the storage device and the mobile communication terminal according to the present invention may be integrated.

[0071] Some of the steps included in the flowcharts in Figures 4 to 7 may be omitted, and steps not shown may be added.

[0072] In implementing the present invention, an embodiment was described in which, even when there are three or more registered markers, self-localization based on two markers and self-localization based on three markers are used interchangeably. The implementation of the present invention is not limited to this embodiment. Furthermore, an embodiment was described in which self-localization based on two markers and self-localization based on three markers are used, based on the criteria (predetermined conditions according to the present invention) explained with reference to Figure 11. The implementation of the present invention is not limited to this embodiment. For example, the mobile communication terminal 113 may estimate its own position based on a fixed number of registered markers, or based on a number of registered markers specified by the operator WK.

[0073] This embodiment encompasses the following technical concepts. (1) A building or civil engineering structure management support system that supports the management of a building or civil engineering structure in which markers are installed at multiple locations, comprising: a storage device that stores drawing data of the building or civil engineering structure; and a mobile communication terminal that is communicatively connected to the storage device, wherein the drawing data includes location information of drawing markers corresponding to at least some of the markers placed on the building or civil engineering structure before the mobile communication terminal moves within the premises of the building or civil engineering structure, and the mobile communication terminal includes a camera and an acceleration sensor. A management support system for buildings or civil engineering structures, characterized by having the following functions: detecting markers by image recognition processing of video footage captured by the camera when moving within the premises of the building or civil engineering structure; extracting the corresponding drawing markers from the drawing data; determining the position of the mobile communication terminal based on the position information of the extracted drawing markers and its own motion information detected by the acceleration sensor; and matching the video space captured by the camera with the drawing space represented by the drawing data based on the determined position of the mobile communication terminal and outputting the result. (2) The building or civil engineering structure management support system according to (1), wherein when the mobile communication terminal detects a new marker by image recognition processing of video footage captured by the camera while the building or civil engineering structure is moving, and a drawing marker corresponding to the new marker is registered in the drawing data, the mobile communication terminal determines its own position based on the position information of the drawing marker and its own motion information detected by the acceleration sensor; and when a drawing marker corresponding to the new marker is not registered in the drawing data, the mobile communication terminal determines its own position and the position information of the new marker based on the position information of other markers detected before the new marker and its own motion information detected by the acceleration sensor, and registers the drawing marker corresponding to the new marker in the drawing data. (3) A building or civil engineering structure management support system that supports the management of a building or civil engineering structure in which markers are installed at multiple locations, comprising: a storage device that stores drawing data of the building or civil engineering structure; and a mobile communication terminal that is communicatively connected to the storage device, wherein the drawing data does not contain location information of drawing markers corresponding to markers placed on the building or civil engineering structure before the mobile communication terminal moves within the premises of the building or civil engineering structure, the mobile communication terminal has a camera and an acceleration sensor, and when the marker detected by image recognition processing on the video captured by the camera while moving within the premises of the building or civil engineering structure is the first marker, in the drawing data A management support system for buildings or civil engineering structures, characterized by registering the first marker at the origin of the coordinate space, and when a marker detected by image recognition processing on video footage captured by the camera while the building or civil engineering structure is moving is not the first marker and a corresponding drawing marker is not registered in the drawing data, the system determines the position of the mobile communication terminal and the position information of the marker based on the position information of other markers detected before the marker and the motion information of the mobile communication terminal detected by the acceleration sensor, registers the corresponding drawing marker in the drawing data, and compares the video space captured by the camera with the drawing space represented by the drawing data based on the determined position of the mobile communication terminal and outputs the result. (4) The building or civil engineering structure management support system according to any one of (1) to (3), wherein the mobile communication terminal, at a first timing in which it detects at least two or more markers from the video footage captured by the camera while moving within the premises of the building or civil engineering structure, extracts a plurality of drawing markers corresponding to some or all of the detected markers from the drawing data, determines its own position at the first timing based on the position information of the extracted plurality of drawing markers and its own motion information detected by the acceleration sensor, and at a second timing in which it detects a new marker from the video footage captured by the camera that has moved to a different location after the first timing, it again extracts a plurality of drawing markers corresponding to the newly detected marker and two or more markers included in the markers detected up to the time of detection from the drawing data, and determines its own position at the second timing based on the position information of the extracted plurality of drawing markers and its own motion information detected by the acceleration sensor. (5) The building or civil engineering structure management support system as described in (4), wherein the mobile communication terminal extracts a first drawing marker and a second drawing marker from the drawing data at the first timing or the second timing, searches the drawing data for a third drawing marker that satisfies predetermined conditions from among the drawing markers that are different from both the extracted first drawing marker and the second drawing marker, and if it is not possible to find a third drawing marker that satisfies the predetermined conditions, it determines its own position based on the position information relating to the two points of the first drawing marker and the second drawing marker and its own motion information detected by the acceleration sensor, and if it is possible to find a third drawing marker that satisfies the predetermined conditions, it determines its own position based on the position information relating to the three points of the first drawing marker, the second drawing marker and the third drawing marker and its own motion information detected by the acceleration sensor. (6) The building or civil engineering structure management support system as described in (5), wherein the mobile communication terminal determines that the third drawing marker satisfies the predetermined conditions when the angle of the interior angle with the first drawing marker as the vertex and the angle of the interior angle with the second drawing marker as the vertex both fall within a predetermined range for a triangle formed by the three points of the first drawing marker, the second drawing marker and the third drawing marker. (7) The mobile communication terminal has a marker corresponding to the first drawing marker as the first marker, a marker corresponding to the second drawing marker as the second marker, and a marker corresponding to the third drawing marker as the third marker, and a vector from the first drawing marker to the second drawing marker in the drawing space as the drawing reference vector, and a vector from the first marker to the marker corresponding to the second marker in the video space as the video reference vector, and when it is not possible to find the third drawing marker that satisfies the predetermined conditions, it performs a first process of translating the video space and the drawing space so that the position of the first marker and the position of the first drawing marker overlap, and rotates the video by the position where the first marker and the first drawing marker overlap as a result of the first process. A building or civil engineering structure management support system according to (5), which performs a second process of matching a reference vector with the drawing reference vector, compares the image space captured by the camera with the drawing space represented by the drawing data, and when it is possible to find the third drawing marker that satisfies the predetermined conditions, performs the first process and the second process, and further performs a third process of matching a vector that passes through the third marker and intersects the image reference vector perpendicularly with the image reference vector by rotational movement around the image reference vector and the drawing reference vector that were matched in the second process, and compares the image space captured by the camera with the drawing space represented by the drawing data. (8) A building or civil engineering structure management support system according to any one of (1) to (3), wherein each marker installed on the building or civil engineering structure is assigned identification information detectable by image recognition processing, and the mobile communication terminal extracts the drawing marker associated with the identification information assigned to the detected marker as the drawing marker corresponding to the detected marker, and when there is no drawing marker in the drawing data that is associated with the identification information assigned to the detected marker, the mobile communication terminal extracts a drawing marker located near the detection position of the marker and that is not associated with the identification information as the drawing marker corresponding to the marker, and associates the identification information assigned to the extracted drawing marker with the marker. (9) The building or civil engineering structure management support system according to (8), wherein when there is no drawing marker located near the detection position of the detected marker, or when another identification information is associated with and registered for a drawing marker located near the detection position of the detected marker, the mobile communication terminal determines that a drawing marker corresponding to the marker is not registered in the drawing data, and registers the drawing marker to which the identification information assigned to the marker is associated as the drawing marker corresponding to the marker in the drawing data. (10) A mobile communication terminal that moves within the premises of a building or civil engineering structure in which markers are installed at multiple locations, wherein the mobile communication terminal is able to refer to drawing data of the building or civil engineering structure stored in a storage device that is integrated with the device or attached externally, and the drawing data includes location information of drawing markers corresponding to at least some of the markers placed on the building or civil engineering structure before the mobile communication terminal moves within the premises of the building or civil engineering structure, and the mobile communication terminal is equipped with a camera and an acceleration sensor. A mobile communication terminal having the following characteristics: detecting markers by image recognition processing on video footage captured by the camera when moving within the premises of the building or civil engineering structure; extracting a drawing marker corresponding to the detected marker from the drawing data by referring to the drawing data; determining the position of the mobile communication terminal based on the position information of the extracted drawing marker and its own motion information detected by the acceleration sensor; and matching the video space captured by the camera with the drawing space represented by the drawing data based on the determined position of the mobile communication terminal and outputting the result. (11) A mobile communication terminal that moves within the premises of a building or civil engineering structure in which markers are installed at multiple locations, wherein the mobile communication terminal is able to refer to drawing data of the building or civil engineering structure stored in a storage device that is integrated with the device or attached externally, and the drawing data does not contain location information of drawing markers corresponding to markers placed on the building or civil engineering structure before the mobile communication terminal moves within the premises of the building or civil engineering structure, and the mobile communication terminal has a camera and an acceleration sensor, and when the marker detected by image recognition processing on the image captured by the camera while moving within the premises of the building or civil engineering structure is the first marker, the drawing A mobile communication terminal characterized by registering the first marker at the origin of the coordinate space in the data, and when a marker detected by image recognition processing on video footage taken by the camera while the building or civil engineering structure is moving is not the first marker and a drawing marker corresponding to that marker is not registered in the drawing data, the mobile communication terminal determines its own position and the position information of the marker based on the position information of other markers detected before the marker and its own motion information detected by the acceleration sensor, registers the drawing marker corresponding to the marker in the drawing data, and compares the video space taken by the camera with the drawing space represented by the drawing data based on the determined self-position of the mobile communication terminal and outputs the result. [Explanation of symbols]

[0074] 100 Management Support Systems 110 Self-propelled robots 111 Sensor 112 Laser Scanners 113 Mobile communication terminals 120 Management device 121 Management Server 122 Storage device 123 Display device WK Worker MK, MK1~MK4 Marker DM, DM1~DM3 drawing markers OB structure DS drawing area

Claims

1. A building or civil engineering structure management support system that assists in the management of buildings or civil engineering structures in which markers are installed at multiple locations, A storage device that stores drawing data of the aforementioned building or civil engineering structure, A mobile communication terminal that is communicatively connected to the aforementioned storage device, Equipped with, The aforementioned drawing data includes location information of drawing markers corresponding to at least some of the markers placed on the building or civil engineering structure, before the mobile communication terminal moves within the premises of the building or civil engineering structure. The aforementioned mobile communication terminal is It has a camera and an accelerometer. When moving within the premises of the aforementioned building or civil engineering structure, markers are detected by image recognition processing of the video captured by the camera. At a first timing in which at least two or more markers are detected from the captured video, a plurality of drawing markers corresponding to some or all of the detected markers are extracted from the drawing data, and the self-position at the first timing is determined based on the positional information of the extracted plurality of drawing markers and the self-motion information detected by the acceleration sensor. At a second timing in which a new marker is detected in the image captured by the camera that has moved to a different location after the first timing, the drawing markers corresponding to two or more markers included in the newly detected marker and the markers detected up to that point are again extracted from the drawing data, and the self's position at the second timing is determined based on the position information of the multiple drawing markers that were extracted again and the self's motion information detected by the acceleration sensor. Based on the determined self-position of the mobile communication terminal, the video space captured by the camera and the drawing space represented by the drawing data are compared and output. Furthermore, the mobile communication terminal, At the first timing or the second timing, a first drawing marker and a second drawing marker are extracted from the drawing data, and a third drawing marker that satisfies predetermined conditions is searched from the drawing data among the drawing markers that are different from both the extracted first drawing marker and the second drawing marker. If the third drawing marker that satisfies the predetermined conditions cannot be found, the self-position is determined based on the positional information of the two points of the first drawing marker and the second drawing marker, and the self-motion information detected by the acceleration sensor. When the third drawing marker that satisfies the predetermined conditions is found, the self-position is determined based on the positional information of the three points of the first drawing marker, the second drawing marker, and the third drawing marker, and the self-motion information detected by the acceleration sensor. The predetermined condition is that, for a triangle formed by the three points of the first drawing marker, the second drawing marker, and the third drawing marker, the interior angle with the first drawing marker as the vertex and the interior angle with the second drawing marker as the vertex are both within a predetermined range. A management support system for buildings or civil engineering structures characterized by the following features.

2. A building or civil engineering structure management support system that assists in the management of buildings or civil engineering structures in which markers are installed at multiple locations, A storage device that stores drawing data of the aforementioned building or civil engineering structure, A mobile communication terminal that is communicatively connected to the aforementioned storage device, Equipped with, The aforementioned drawing data includes location information of drawing markers corresponding to at least some of the markers placed on the building or civil engineering structure, before the mobile communication terminal moves within the premises of the building or civil engineering structure. The aforementioned mobile communication terminal is It has a camera and an accelerometer. When moving within the premises of the aforementioned building or civil engineering structure, markers are detected by image recognition processing of the video captured by the camera. At a first timing in which at least two or more markers are detected from the captured video, a plurality of drawing markers corresponding to some or all of the detected markers are extracted from the drawing data, and the self-position at the first timing is determined based on the positional information of the extracted plurality of drawing markers and the self-motion information detected by the acceleration sensor. At a second timing in which a new marker is detected in the image captured by the camera that has moved to a different location after the first timing, the drawing markers corresponding to two or more markers included in the newly detected marker and the markers detected up to that point are again extracted from the drawing data, and the self's position at the second timing is determined based on the position information of the multiple drawing markers that were extracted again and the self's motion information detected by the acceleration sensor. Based on the determined self-position of the mobile communication terminal, the video space captured by the camera and the drawing space represented by the drawing data are compared and output. Furthermore, the mobile communication terminal, The marker corresponding to the first drawing marker will be called the first marker, the marker corresponding to the second drawing marker will be called the second marker, and the marker corresponding to the third drawing marker will be called the third marker. The vector from the first drawing marker to the second drawing marker in the drawing space is defined as the drawing reference vector, and the vector from the first marker to the second marker in the video space is defined as the video reference vector. At the first timing or the second timing, the first drawing marker and the second drawing marker are extracted from the drawing data, and the third drawing marker that satisfies predetermined conditions is searched from the drawing data among the drawing markers that are different from both the extracted first drawing marker and the second drawing marker. If the third drawing marker that satisfies the predetermined conditions cannot be found, the following are performed: a first process of translating the video space and the drawing space so that the position of the first marker and the position of the first drawing marker overlap; and a second process of aligning the video reference vector and the drawing reference vector by rotational movement around the position where the first marker and the first drawing marker overlap as a result of the first process, thereby comparing the video space captured by the camera with the drawing space represented by the drawing data. When the third drawing marker that satisfies the predetermined conditions is found, the first and second processes are executed, and further, a third process is executed to match a vector that passes through the third marker and intersects the image reference vector perpendicularly with the drawing reference vector, by rotational movement around the image reference vector and the drawing reference vector that were matched in the second process, with a vector that passes through the third drawing marker and intersects the drawing reference vector perpendicularly, thereby comparing the image space captured by the camera with the drawing space represented by the drawing data. A management support system for buildings or civil engineering structures characterized by the following features.

3. A building or civil engineering structure management support system that assists in the management of buildings or civil engineering structures in which markers are installed at multiple locations, A storage device that stores drawing data of the aforementioned building or civil engineering structure, A mobile communication terminal that is communicatively connected to the aforementioned storage device, Equipped with, The aforementioned drawing data does not contain location information of drawing markers corresponding to markers placed on the building or civil engineering structure before the mobile communication terminal moves within the premises of the building or civil engineering structure. The aforementioned mobile communication terminal is It has a camera and an accelerometer. When the marker detected by image recognition processing on the video captured by the camera while moving within the premises of the building or civil engineering structure is the first marker, the first marker is registered at the origin of the coordinate space in the drawing data. When a marker detected by image recognition processing on video footage captured by the camera while the building or civil engineering structure is moving is not the first marker, and a corresponding drawing marker is not registered in the drawing data, the mobile communication terminal's own position and the position information of the marker are determined based on the position information of other markers detected before the marker and its own motion information detected by the acceleration sensor, and the corresponding drawing marker is registered in the drawing data. At a first timing in which at least two or more markers are detected from the video footage captured by the camera while moving within the premises of the building or civil engineering structure, a plurality of drawing markers corresponding to some or all of the detected markers are extracted from the drawing data, and the position of the extracted plurality of drawing markers and the motion information of the self detected by the acceleration sensor are used to determine the self's position at the first timing. At a second timing in which a new marker is detected in the image captured by the camera that has moved to a different location after the first timing, the drawing markers corresponding to two or more markers included in the newly detected marker and the markers detected up to that point are again extracted from the drawing data, and the self's position at the second timing is determined based on the position information of the multiple drawing markers that were extracted again and the self's motion information detected by the acceleration sensor. Based on the determined self-position of the mobile communication terminal, the video space captured by the camera and the drawing space represented by the drawing data are compared and output. Furthermore, the mobile communication terminal, At the first timing or the second timing, a first drawing marker and a second drawing marker are extracted from the drawing data, and a third drawing marker that satisfies predetermined conditions is searched from the drawing data among the drawing markers that are different from both the extracted first drawing marker and the second drawing marker. If the third drawing marker that satisfies the predetermined conditions cannot be found, the self-position is determined based on the positional information of the two points of the first drawing marker and the second drawing marker, and the self-motion information detected by the acceleration sensor. When the third drawing marker that satisfies the predetermined conditions is found, the self-position is determined based on the positional information of the three points of the first drawing marker, the second drawing marker, and the third drawing marker, and the self-motion information detected by the acceleration sensor. The predetermined condition is that, for a triangle formed by the three points of the first drawing marker, the second drawing marker, and the third drawing marker, the interior angle with the first drawing marker as the vertex and the interior angle with the second drawing marker as the vertex are both within a predetermined range. A management support system for buildings or civil engineering structures characterized by the following features.

4. A building or civil engineering structure management support system that assists in the management of buildings or civil engineering structures in which markers are installed at multiple locations, A storage device that stores drawing data of the aforementioned building or civil engineering structure, A mobile communication terminal that is communicatively connected to the aforementioned storage device, Equipped with, The aforementioned drawing data does not contain location information of drawing markers corresponding to markers placed on the building or civil engineering structure before the mobile communication terminal moves within the premises of the building or civil engineering structure. The aforementioned mobile communication terminal is It has a camera and an accelerometer. When the marker detected by image recognition processing on the video captured by the camera while moving within the premises of the building or civil engineering structure is the first marker, the first marker is registered at the origin of the coordinate space in the drawing data. When a marker detected by image recognition processing on video footage captured by the camera while the building or civil engineering structure is moving is not the first marker, and a corresponding drawing marker is not registered in the drawing data, the mobile communication terminal's own position and the position information of the marker are determined based on the position information of other markers detected before the marker and its own motion information detected by the acceleration sensor, and the corresponding drawing marker is registered in the drawing data. At a first timing in which at least two or more markers are detected from the video footage captured by the camera while moving within the premises of the building or civil engineering structure, a plurality of drawing markers corresponding to some or all of the detected markers are extracted from the drawing data, and the position of the extracted plurality of drawing markers and the motion information of the self detected by the acceleration sensor are used to determine the self's position at the first timing. At a second timing in which a new marker is detected in the image captured by the camera that has moved to a different location after the first timing, the drawing markers corresponding to two or more markers included in the newly detected marker and the markers detected up to that point are again extracted from the drawing data, and the self's position at the second timing is determined based on the position information of the multiple drawing markers that were extracted again and the self's motion information detected by the acceleration sensor. Based on the determined self-position of the mobile communication terminal, the video space captured by the camera and the drawing space represented by the drawing data are compared and output. Furthermore, the mobile communication terminal, The marker corresponding to the first drawing marker will be called the first marker, the marker corresponding to the second drawing marker will be called the second marker, and the marker corresponding to the third drawing marker will be called the third marker. The vector from the first drawing marker to the second drawing marker in the drawing space is defined as the drawing reference vector, and the vector from the first marker to the second marker in the video space is defined as the video reference vector. At the first timing or the second timing, the first drawing marker and the second drawing marker are extracted from the drawing data, and the third drawing marker that satisfies predetermined conditions is searched from the drawing data among the drawing markers that are different from both the extracted first drawing marker and the second drawing marker. If the third drawing marker that satisfies the predetermined conditions cannot be found, the following are performed: a first process of translating the video space and the drawing space so that the position of the first marker and the position of the first drawing marker overlap; and a second process of aligning the video reference vector and the drawing reference vector by rotational movement around the position where the first marker and the first drawing marker overlap as a result of the first process, thereby comparing the video space captured by the camera with the drawing space represented by the drawing data. When the third drawing marker that satisfies the predetermined conditions is found, the first and second processes are executed, and further, a third process is executed to match a vector that passes through the third marker and intersects the image reference vector perpendicularly with the drawing reference vector, by rotational movement around the image reference vector and the drawing reference vector that were matched in the second process, with a vector that passes through the third drawing marker and intersects the drawing reference vector perpendicularly, thereby comparing the image space captured by the camera with the drawing space represented by the drawing data. A management support system for buildings or civil engineering structures characterized by the following features.

5. A mobile communication terminal that moves within the premises of a building or civil engineering structure, where markers are installed at multiple locations, The mobile communication terminal is capable of accessing drawing data of the building or civil engineering structure stored in a storage device that is integrated with or externally attached to the device. The aforementioned drawing data includes location information of drawing markers corresponding to at least some of the markers placed on the building or civil engineering structure, before the mobile communication terminal moves within the premises of the building or civil engineering structure. The aforementioned mobile communication terminal is It has a camera and an accelerometer. When moving within the premises of the aforementioned building or civil engineering structure, markers are detected by image recognition processing of the video captured by the camera. At a first timing in which at least two or more markers are detected from the captured video, a plurality of drawing markers corresponding to some or all of the detected markers are extracted from the drawing data, and the self-position at the first timing is determined based on the positional information of the extracted plurality of drawing markers and the self-motion information detected by the acceleration sensor. At a second timing in which a new marker is detected in the image captured by the camera that has moved to a different location after the first timing, the drawing markers corresponding to two or more markers included in the newly detected marker and the markers detected up to that point are again extracted from the drawing data, and the self's position at the second timing is determined based on the position information of the multiple drawing markers that were extracted again and the self's motion information detected by the acceleration sensor. Based on the determined self-position of the mobile communication terminal, the video space captured by the camera and the drawing space represented by the drawing data are compared and output. Furthermore, the mobile communication terminal, At the first timing or the second timing, a first drawing marker and a second drawing marker are extracted from the drawing data, and a third drawing marker that satisfies predetermined conditions is searched from the drawing data among the drawing markers that are different from both the extracted first drawing marker and the second drawing marker. If the third drawing marker that satisfies the predetermined conditions cannot be found, the self-position is determined based on the positional information of the two points of the first drawing marker and the second drawing marker, and the self-motion information detected by the acceleration sensor. When the third drawing marker that satisfies the predetermined conditions is found, the self-position is determined based on the positional information of the three points of the first drawing marker, the second drawing marker, and the third drawing marker, and the self-motion information detected by the acceleration sensor. The predetermined condition is that, for a triangle formed by the three points of the first drawing marker, the second drawing marker, and the third drawing marker, the interior angle with the first drawing marker as the vertex and the interior angle with the second drawing marker as the vertex are both within a predetermined range. A mobile communication terminal characterized by the following features.

6. A mobile communication terminal that moves within the premises of a building or civil engineering structure, where markers are installed at multiple locations, The mobile communication terminal is capable of accessing drawing data of the building or civil engineering structure stored in a storage device that is integrated with or externally attached to the device. The aforementioned drawing data includes location information of drawing markers corresponding to at least some of the markers placed on the building or civil engineering structure, before the mobile communication terminal moves within the premises of the building or civil engineering structure. The aforementioned mobile communication terminal is It has a camera and an accelerometer. When moving within the premises of the aforementioned building or civil engineering structure, markers are detected by image recognition processing of the video captured by the camera. At a first timing in which at least two or more markers are detected from the captured video, a plurality of drawing markers corresponding to some or all of the detected markers are extracted from the drawing data, and the self-position at the first timing is determined based on the positional information of the extracted plurality of drawing markers and the self-motion information detected by the acceleration sensor. At a second timing in which a new marker is detected in the image captured by the camera that has moved to a different location after the first timing, the drawing markers corresponding to two or more markers included in the newly detected marker and the markers detected up to that point are again extracted from the drawing data, and the self's position at the second timing is determined based on the position information of the multiple drawing markers that were extracted again and the self's motion information detected by the acceleration sensor. Based on the determined self-position of the mobile communication terminal, the video space captured by the camera and the drawing space represented by the drawing data are compared and output. Furthermore, the mobile communication terminal, The marker corresponding to the first drawing marker will be called the first marker, the marker corresponding to the second drawing marker will be called the second marker, and the marker corresponding to the third drawing marker will be called the third marker. The vector from the first drawing marker to the second drawing marker in the drawing space is defined as the drawing reference vector, and the vector from the first marker to the second marker in the video space is defined as the video reference vector. At the first timing or the second timing, the first drawing marker and the second drawing marker are extracted from the drawing data, and the third drawing marker that satisfies predetermined conditions is searched from the drawing data among the drawing markers that are different from both the extracted first drawing marker and the second drawing marker. If the third drawing marker that satisfies the predetermined conditions cannot be found, the following are performed: a first process of translating the video space and the drawing space so that the position of the first marker and the position of the first drawing marker overlap; and a second process of aligning the video reference vector and the drawing reference vector by rotational movement around the position where the first marker and the first drawing marker overlap as a result of the first process, thereby comparing the video space captured by the camera with the drawing space represented by the drawing data. When the third drawing marker that satisfies the predetermined conditions is found, the first and second processes are executed, and further, a third process is executed to match a vector that passes through the third marker and intersects the image reference vector perpendicularly with the drawing reference vector, by rotational movement around the image reference vector and the drawing reference vector that were matched in the second process, with a vector that passes through the third drawing marker and intersects the drawing reference vector perpendicularly, thereby comparing the image space captured by the camera with the drawing space represented by the drawing data. A mobile communication terminal characterized by the following features.

7. A mobile communication terminal that moves within the premises of a building or civil engineering structure, where markers are installed at multiple locations, The mobile communication terminal is capable of accessing drawing data of the building or civil engineering structure stored in a storage device that is integrated with or externally attached to the device. The aforementioned drawing data does not contain location information of drawing markers corresponding to markers placed on the building or civil engineering structure before the mobile communication terminal moves within the premises of the building or civil engineering structure. The aforementioned mobile communication terminal is It has a camera and an accelerometer. When the marker detected by image recognition processing on the video captured by the camera while moving within the premises of the building or civil engineering structure is the first marker, the first marker is registered at the origin of the coordinate space in the drawing data. When a marker detected by image recognition processing on video footage captured by the camera while the building or civil engineering structure is moving is not the first marker, and a corresponding drawing marker is not registered in the drawing data, the mobile communication terminal's own position and the position information of the marker are determined based on the position information of other markers detected before the marker and its own motion information detected by the acceleration sensor, and the corresponding drawing marker is registered in the drawing data. At a first timing in which at least two or more markers are detected from the video footage captured by the camera while moving within the premises of the building or civil engineering structure, a plurality of drawing markers corresponding to some or all of the detected markers are extracted from the drawing data, and the position of the extracted plurality of drawing markers and the motion information of the self detected by the acceleration sensor are used to determine the self's position at the first timing. At a second timing in which a new marker is detected in the image captured by the camera that has moved to a different location after the first timing, the drawing markers corresponding to two or more markers included in the newly detected marker and the markers detected up to that point are again extracted from the drawing data, and the self's position at the second timing is determined based on the position information of the multiple drawing markers that were extracted again and the self's motion information detected by the acceleration sensor. Based on the determined self-position of the mobile communication terminal, the video space captured by the camera and the drawing space represented by the drawing data are compared and output. Furthermore, the mobile communication terminal, At the first timing or the second timing, a first drawing marker and a second drawing marker are extracted from the drawing data, and a third drawing marker that satisfies predetermined conditions is searched from the drawing data among the drawing markers that are different from both the extracted first drawing marker and the second drawing marker. If the third drawing marker that satisfies the predetermined conditions cannot be found, the self-position is determined based on the positional information of the two points of the first drawing marker and the second drawing marker, and the self-motion information detected by the acceleration sensor. When the third drawing marker that satisfies the predetermined conditions is found, the self-position is determined based on the positional information of the three points of the first drawing marker, the second drawing marker, and the third drawing marker, and the self-motion information detected by the acceleration sensor. The predetermined condition is that, for a triangle formed by the three points of the first drawing marker, the second drawing marker, and the third drawing marker, the interior angle with the first drawing marker as the vertex and the interior angle with the second drawing marker as the vertex are both within a predetermined range. A mobile communication terminal characterized by the following features.

8. A mobile communication terminal that moves within the premises of a building or civil engineering structure, where markers are installed at multiple locations, The mobile communication terminal is capable of accessing drawing data of the building or civil engineering structure stored in a storage device that is integrated with or externally attached to the device. The aforementioned drawing data does not contain location information of drawing markers corresponding to markers placed on the building or civil engineering structure before the mobile communication terminal moves within the premises of the building or civil engineering structure. The aforementioned mobile communication terminal is It has a camera and an accelerometer. When the marker detected by image recognition processing on the video captured by the camera while moving within the premises of the building or civil engineering structure is the first marker, the first marker is registered at the origin of the coordinate space in the drawing data. When a marker detected by image recognition processing on video footage captured by the camera while the building or civil engineering structure is moving is not the first marker, and a corresponding drawing marker is not registered in the drawing data, the mobile communication terminal's own position and the position information of the marker are determined based on the position information of other markers detected before the marker and its own motion information detected by the acceleration sensor, and the corresponding drawing marker is registered in the drawing data. At a first timing in which at least two or more markers are detected from the video footage captured by the camera while moving within the premises of the building or civil engineering structure, a plurality of drawing markers corresponding to some or all of the detected markers are extracted from the drawing data, and the position of the extracted plurality of drawing markers and the motion information of the self detected by the acceleration sensor are used to determine the self's position at the first timing. At a second timing in which a new marker is detected in the image captured by the camera that has moved to a different location after the first timing, the drawing markers corresponding to two or more markers included in the newly detected marker and the markers detected up to that point are again extracted from the drawing data, and the self's position at the second timing is determined based on the position information of the multiple drawing markers that were extracted again and the self's motion information detected by the acceleration sensor. Based on the determined self-position of the mobile communication terminal, the video space captured by the camera and the drawing space represented by the drawing data are compared and output. Furthermore, the mobile communication terminal, The marker corresponding to the first drawing marker will be called the first marker, the marker corresponding to the second drawing marker will be called the second marker, and the marker corresponding to the third drawing marker will be called the third marker. The vector from the first drawing marker to the second drawing marker in the drawing space is defined as the drawing reference vector, and the vector from the first marker to the second marker in the video space is defined as the video reference vector. At the first timing or the second timing, the first drawing marker and the second drawing marker are extracted from the drawing data, and the third drawing marker that satisfies predetermined conditions is searched from the drawing data among the drawing markers that are different from both the extracted first drawing marker and the second drawing marker. If the third drawing marker that satisfies the predetermined conditions cannot be found, the following are performed: a first process of translating the video space and the drawing space so that the position of the first marker and the position of the first drawing marker overlap; and a second process of aligning the video reference vector and the drawing reference vector by rotational movement around the position where the first marker and the first drawing marker overlap as a result of the first process, thereby comparing the video space captured by the camera with the drawing space represented by the drawing data. When the third drawing marker that satisfies the predetermined conditions is found, the first and second processes are executed, and further, a third process is executed to match a vector that passes through the third marker and intersects the image reference vector perpendicularly with the drawing reference vector, by rotational movement around the image reference vector and the drawing reference vector that were matched in the second process, with a vector that passes through the third drawing marker and intersects the drawing reference vector perpendicularly, thereby comparing the image space captured by the camera with the drawing space represented by the drawing data. A mobile communication terminal characterized by the following features.

Citation Information

Patent Citations

  • Display method, display program and display device

    JP2015204079A

  • Construction process management system and construction process management method

    JP2019148946A

  • Movable body

    JP2021056764A

  • Work support system and work support method

    JP2022161770A