Guide system, guide method and guide program for assisting work of operator in work site

Abstract

To secure position determination with high accuracy in a work site, and intuitively grasp a work object part at the site in a view point of an operator.SOLUTION: A guide system for assisting work of an operator in a work site comprises: an imaging position calculation part which has a guide terminal being possessed by an operator and having an imaging part for imaging a new image, and which uses positional information of an associated common feature point and the new image for calculating positional information of an imaging position and an imaging posture of the new image; a selection part for selecting a work object for which an installation scheduled position should be guided, from multiple objects or points stored in a storage part; and a guide calculation part for outputting to the operator, guide information of a work object or a work object point to the installation scheduled position from the imaging position of the new image, on the basis of the positional information of the installation scheduled position of the selected work object or work object point, the positional information of the imaging position of the new image and the imaging posture.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a guide that assists workers in performing tasks at a work site. [Background technology]

[0002] 2. Description of the Related Art Conventionally, systems have been known that use three-dimensional models to support on-site construction work by workers. For example, Patent Document 1 describes a construction support system that supports the installation of components that make up building facilities on a building. Even indoors, where radio waves from GPS satellites cannot reach, it describes how the system uses images taken by cameras worn by workers to generate three-dimensional data of the work space as the work progresses, calculates the worker's position using three-dimensional synthesis processing based on photogrammetry technology, and reads tags such as two-dimensional codes attached to the components to determine whether there are any newly matching components that have been installed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-149546 A Summary of the Invention [Problem to be solved by the invention]

[0004] In environments where it is difficult for GPS and other satellite signals to reach, such as inside buildings under construction, it can be difficult for workers to confirm their current location or intuitively grasp the location of materials and signs that they need to install.

[0005] Therefore, the present disclosure aims to provide a guide system, guide method, and guide program that assists workers in their work at work sites, which enables the workers to intuitively understand the areas to be worked on from their perspective on the spot while ensuring highly accurate position measurement at the work site. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, the guide system according to the present disclosure is a guide system that assists a worker in the work of a work site, and includes a memory unit that stores design information including position information of planned installation positions of multiple objects or points that are elements constituting a planned construction at the work site, and multiple existing feature points and position information of site reference objects to be used as a reference for positions at the work site obtained as a result of surveying by a surveying device, and a guide terminal that is held by the worker and has an imaging unit that captures a new image and an attitude detection unit that detects imaging direction information of the new image, and the guide terminal includes a feature extraction unit that extracts multiple feature points from the new image, and a guide terminal that uses the multiple feature points included in the new image to extract a feature from the multiple existing feature points stored in the memory unit. the image capturing apparatus includes a feature matching unit that extracts a predetermined number or more of common feature points that are common to the feature points from the image capturing apparatus and matches them; an imaging position calculation unit that calculates position information of the imaging position of the new image and the imaging attitude using position information of the matched common feature points and the new image; a selection unit that selects a work target object or work target point, the planned installation position of which should be guided, from the multiple objects or points stored in the memory unit; and a guide calculation unit that outputs guide information from the imaging position of the new image to the planned installation position of the work target object or work target point, to the worker, based on the position information of the planned installation position of the selected work target object or work target point and the position information and imaging attitude of the imaging position of the new image.

[0007] In order to achieve the above-mentioned object, the guiding method according to the present disclosure is a guiding method for assisting a worker in work at a work site, comprising: a storage step for storing in a storage unit design information including position information of a plurality of objects or points that are elements constituting a planned construction at the work site and their planned installation positions, and a plurality of existing feature points and position information of site reference objects to be used as a reference for positions at the work site obtained as a result of surveying by a surveying device; an imaging step in which the worker captures a new image using a guide terminal having an imaging unit; a new image feature point extraction step for extracting a plurality of feature points from the new image by a feature extraction unit; and a feature matching step for selecting, by a feature matching unit, feature points that are common to the feature points included in the new image from the plurality of existing feature points stored in the storage unit. the image capturing position calculation step of calculating position information of the image capturing position and the image capturing attitude of the new image by an image capturing position calculation unit using position information of the associated common feature points and the new image; a selection step of causing a selection unit to select a work target object or work target point whose planned installation position should be guided from the multiple objects or points stored in the storage unit; and a guide step of outputting guide information from the image capturing position of the new image to the planned installation position of the work target object or work target point, to the worker, based on absolute position information of the planned installation position of the selected work target object or work target point and the position information and image capturing attitude of the image capturing position of the new image.

[0008] In order to achieve the above-mentioned object, a guide program according to the present disclosure is a guide program installed in a guide terminal held by a worker in a guide system that assists a worker in work at a work site, the guide program including a storage step of storing in a storage unit a plurality of existing feature points and position information of a site reference object to be used as a reference for a position at the work site obtained as a result of surveying by a surveying device; a new image acquisition step of acquiring a new image captured by the worker using a guide terminal having an imaging unit; a new image feature point extraction step of extracting a plurality of feature points from the new image by a feature extraction unit; and a feature matching step of using the plurality of feature points included in the new image to select from the plurality of existing feature points stored in the storage unit a predetermined number or more of common features that are common to the feature points included in the new image. The present invention is intended to have a computer execute the following steps: a feature matching step of extracting and matching points; an imaging position calculation step of calculating absolute position information and imaging orientation of the imaging position of the new image by an imaging position calculation unit using position information of the matched common feature points and the new image; a selection step of causing a selection unit to select a work target object, the planned installation position of which should be guided, from a plurality of objects or points stored in a memory unit; and a guide step of outputting guide information from the imaging position of the new image to the planned installation position of the work target object or work target point, to a worker, based on the absolute position information of the planned installation position of the selected work target object or work target point and the position information and imaging orientation of the shooting position of the new image. Effect of the Invention

[0009] By using the guide system of the present disclosure that employs the above-mentioned means, highly accurate position measurement can be ensured at the work site, while allowing the worker to intuitively understand the area to be worked on from his / her perspective on the spot. [Brief description of the drawings]

[0010] [Figure 1] 1 is an overall configuration diagram of a guide system according to an embodiment; [Diagram 2] FIG. 2 is a configuration diagram of a guide system. [Diagram 3]4 is a flowchart of each process of a guide method. [Figure 4] FIG. 13 is an overview for explaining a method for acquiring a photographing position. [Diagram 5] 13 is an example of an output of guide information. [Figure 6] 13 is an example of an output of guide information. [Figure 7] 13 is an example of an output of guide information. [Figure 8] 4 is a flowchart of each process of a guide method. [Figure 9] 13 is an example of an output of guide information. [Figure 10] FIG. 13 is an overall configuration diagram of a guide system according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. Fig. 1 is an overall configuration diagram of a guide system 1 according to an embodiment of the present disclosure. In the description of this embodiment, each device and its arrangement are shown in a schematic manner, and may differ from the actual scale for the sake of convenience.

[0012] As shown in Fig. 1, the guide system 1 is a system in which a worker A moves around holding a guide terminal 2, takes pictures, calculates the position of the guide terminal 2 based on the taken images, and displays the work target position on the guide terminal 2 to assist the worker in his / her work. This is particularly effective indoors, where it is difficult to obtain the self-position using GNSS such as GPS.

[0013] The guide terminal 2 includes a main body 21. The main body 21 may be held by a rod-shaped part 22, which is a pinpole having a pin at its tip. That is, the tip of the pinpole is configured to be able to contact one point and indicate that one point, and the positional relationship between the tip of the pinpole and the main body 21 is fixed and the distance is known. Then, if the tip of the pinpole is brought into contact with a certain position X and the rod-shaped part 22 is held vertically, the position of the main body 21 is automatically determined. The guide terminal 2 may also include a prism 23, which is a retroreflective prism for tracking that position.

[0014] 2, the main body 21 has an imaging unit 30, an operation unit 32, a display unit 33, a communication unit 34, a storage unit 35, an audio unit 38, a timing unit 39, and a control unit 36. The main body 21 may also have a posture detection unit 31.

[0015] The imaging unit 30 is an image sensor (such as a CCD sensor or a CMOS sensor) that captures an image. The image that the imaging unit 30 can capture may be a still image or a moving image. In this embodiment, a still image is described as an example to facilitate understanding of the invention, but this embodiment can also be applied to a moving image. The camera that constitutes the imaging unit may be a stereo camera, a depth camera, a ToF camera, a LIDAR integrated camera, or the like, which will be described later. These cameras have a function that enables them to measure the distance to the imaging target.

[0016] The attitude detection unit 31 is, for example, an IMU (Inertial Measurement Unit) equipped with an accelerometer and a gyroscope, and can measure acceleration, angular velocity, and the like, and acquire attitude information of the guide terminal 2. For example, the accelerometer can detect acceleration along three axes, and the gyroscope can detect angular velocity along three axes.

[0017] The operation unit 32 is an operating means capable of inputting various operation instructions and settings. Operation instructions can include, for example, switching the power on or off, instructions for shooting by the imaging unit 30, settings related to imaging, and selection of a work target object, which will be described later. The operation unit 32 may also include other operation devices or input devices, such as switches, buttons, and dials. The operation unit 32 may also perform voice input using an audio unit 38 equipped with a microphone.

[0018] The display unit 33 is, for example, a liquid crystal display or an organic EL display. When the display unit 33 is a touch panel display, the display unit 33 and the operation unit 32 may be integrally formed. The display unit 33 may also be smart glasses or a head-mounted display connected to the guide terminal 2 and worn by the worker.

[0019] The communication unit 34 has a function of communicating with an external device so as to share information. For example, it is a wireless communication means. The communication unit 34 may function as a wired communication means via a connection terminal. Examples of the external device include a portable personal digital assistant (PDA), a smartphone, a tablet terminal, a personal computer, and the like.

[0020] The storage unit 35 is an auxiliary storage device such as a hard disk or SSD in which various information is stored. The storage unit 35 stores design information including a plurality of three-dimensional objects, a three-dimensional model of a planned construction object composed of a plurality of three-dimensional objects, position information of the planned installation position in a coordinate system specific to the work site, and position information of points on the work site, as well as feature points and feature quantities of the building T as a site reference object that enable identification of the position of the building T as a site reference object for use as a reference for the coordinate system specific to the work site. These feature points and feature quantities are stored in association with points included in the three-dimensional model.

[0021] Coordinate systems include a global coordinate system, a world coordinate system, and an absolute coordinate system, but the work site-specific coordinate system in this disclosure does not necessarily have to be the same as these coordinate systems, and may be a coordinate system that is specific to each work site and is managed based on a reference object at the site.The design information may be BIM (Building Information Modeling), and may include position information of the points where work is to be performed, the objects that will be building materials, or points where piles and the like will be installed, and for objects, dimensional information such as their width, depth, height, etc., as well as information such as ID and product number.

[0022] The object may include a component having a size such as a building material. The point may include a location (point) for driving a stake or a location (point) for driving a pillar, or a marker used to install them as a sign. In the present disclosure, the work target object or the work target point may include information indicating a component such as a building material and a location (point) for installing a stake or a pillar. The work target object or the work target point has position information in a coordinate system specific to the work site where the installation is to be performed.

[0023] In the present disclosure, an image including the feature points and feature amounts stored in the storage unit 35 is referred to as an existing image. A feature point is a characteristic part (position) that appears in an image, and a value that quantifies the uniqueness of the feature point by expressing it as a vector or the like is referred to as a feature amount. The feature amount is calculated by a specific algorithm, for example, SIFT, SURF, ORB, HOG, etc. are used. At least the feature points and various information (data) associated with them are stored in the storage unit 35, and a so-called feature point database (hereinafter also referred to as DB) is constructed. In addition, the storage unit 35 may store an existing image from which the feature points are extracted. For example, the feature point ID and the image ID may be linked and stored together with the position on the image. As a result, the feature points may be linked and stored so that it is possible to identify which feature points were extracted from which position of which image. In addition, when referring to the extraction of feature points, it may be expressed as a process that also includes the calculation of the feature points.

[0024] For example, the feature points include at least position information in an absolute coordinate system or a coordinate system specific to the work site, and are linked to position information (three-dimensional coordinate data) in a global coordinate system (absolute coordinate system) or a coordinate system specific to the work site in accordance with the position in the existing image. This position information of the feature points is basically position information based on the results of a survey conducted in advance using a surveying device such as a surveying instrument (total station or scanner), or may be position information calculated by the control unit 36 ​​described later, which is obtained by performing position analysis or the like based on the results of the survey conducted by the surveying device.

[0025] In addition, the position information of the feature points includes the type of surveying (total station, scanner, photogrammetry, etc.) that calculated the position information, the installation coordinates of the surveying device, and the time data when the surveying device was used. For example, a point cloud obtained by a LiDAR scanner or the like has position information between the scanner and the target object, but this is a relative position. Therefore, using surveying information of an absolute coordinate system previously obtained by a total station (TS) or a coordinate system specific to the work site, the point cloud and feature points stored in the storage unit 35 are linked to the above coordinates as position information.

[0026] Further, other information related to imaging is linked to the feature points. The information related to imaging includes, for example, information including the specifications of the imaging unit (resolution, angle of view, etc.), imaging position information, imaging time information, imaging attitude information, imaging direction information, and imaging time information. The imaging position information is calculated by an imaging position calculation unit 42 of the control unit 36, which will be described later. The attitude information is detected by the attitude detection unit 31 when imaging is performed, and the imaging direction (imaging direction information) is calculated from the attitude information. The imaging time information is, for example, the date and time at the time of imaging, as measured by the clock unit 39 of the guide terminal 2.

[0027] The control unit 36 ​​executes functions and / or methods realized by codes or instructions included in the programs stored in the storage unit 35. The control unit 36 ​​may include, for example, a central processing unit (CPU), an MPU, a GPU, a microprocessor, a processor core, a multiprocessor, an ASIC, an FPGA, etc., and may realize each process disclosed in each embodiment by a logic circuit or a dedicated circuit formed in an integrated circuit, etc. Furthermore, these circuits may be realized by one or more integrated circuits, and multiple processes shown in each embodiment may be realized by one integrated circuit.

[0028] The control unit 36 ​​of this embodiment has a function of calculating the imaging position and imaging orientation of a newly captured image (new image) and outputting guide information for a position where a work target object selected by a worker should be placed based on the imaging position and imaging orientation. The control unit 36 ​​has the following functions: a feature extraction unit 40, a feature matching unit 41, an imaging position calculation unit 42, a difference feature identification unit 43, a DB update unit 44, a display control unit 45, a selection unit 46, a guide calculation unit 47, and an imaging range calculation unit 48.

[0029] The feature extraction unit 40 has a function of extracting feature points from an image. Specifically, the feature points are extracted from the new image captured by the imaging unit 30 using a predetermined algorithm for extracting feature points. The feature extraction unit 40 may also add imaging situation information such as imaging time information to the extracted feature points. The feature points and existing feature point information acquired from the new image may also be used to sequentially update information in the feature point database stored in the storage unit 35. As described above, feature points can be extracted not only from still images but also from videos. Since videos are still images arranged in chronological order, it is possible to extract feature points by analyzing a pre-recorded video file, etc. Furthermore, within the range of the processing capacity of the hardware computational resources constituting the control unit 36, feature points can also be continuously extracted from the new image in real time at the highest possible processing speed.

[0030] The feature matching unit 41 has a function of extracting, by matching, feature points common to the feature points extracted from the new image from the storage unit 35. The storage unit 35 stores a plurality of feature points (existing feature points) extracted from existing images, that is, images that were captured in the past and whose imaging positions were simultaneously tracked by a surveying device and to which imaging position and imaging attitude information are linked. The feature matching unit 41 has a function of extracting feature points common to the feature points extracted from the new image from among these plurality of existing feature points.

[0031] More specifically, the feature matching unit 41 has a function of using a plurality of feature points included in the new image to extract a predetermined number or more of feature points (common feature points) that are common to feature points included in the new image from all existing feature points stored in the storage unit 35, and associating the positional relationship between the new image and the stored existing feature points based on these common feature points. This means that the feature matching unit 41 has a search function of searching the storage unit 35 for feature points similar to feature points included in the new image, using an index indicating commonality regarding feature points such as feature amounts. This makes it possible to acquire feature points that serve as clues for estimating the imaging position of the new image.

[0032] The feature matching unit 41 performs matching by extracting feature points whose feature amounts match or are similar to those of existing feature points stored in the storage unit 35 from the feature points of the new image acquired as described above, setting them as common feature points, and associating them one-to-one. This specifies the relative positional relationship between the existing feature points and the new image. For the feature points in the new image that are associated with the relative positional relationship with the existing feature points, new positional information (three-dimensional coordinates in the global coordinate system) in the absolute coordinate system of the imaging position or in a coordinate system specific to the work site and imaging attitude information are assigned using the positional information of the existing feature points.

[0033] The imaging position calculation unit 42 has a function of calculating the imaging position and imaging attitude of the new image from the feature points of the new image associated with the common feature points to which position information has already been assigned. If a predetermined number or more, specifically, three or more, of common feature points are associated with existing feature points in the new image, the imaging position and imaging attitude can be calculated from the relative positional relationship between the feature points included in the new image and the existing feature points using photogrammetry technology, vision-based navigation technology, or a predetermined algorithm. Specifically, algorithms such as SfM and SLAM can be used. In other words, the imaging position calculation unit 42 can calculate the imaging position and imaging attitude from the associated feature points in the new image without relying on location-based position information such as GNSS and in a situation where markers or the like are not required.

[0034] The differential feature identification unit 43 may extract, from among the feature points included in the new image, feature points that are different from the common feature points and different from the existing feature points stored in the storage unit 35, as differential feature points, and store them newly in the storage unit 35. When a new image is captured, the stored differential feature points can also be matched with the new feature points to calculate position information.

[0035] The DB update unit 44 may have a function of storing a new image in which position information is assigned to each feature point as an existing image in the storage unit 35. The DB update unit 44 may sequentially update the feature point database by storing information on the imaging position and imaging posture of a new image containing the feature point together with the feature points assigned with position information. As a result, after the feature point database is updated, the new feature points also become existing feature points assigned with position information. Each feature point of the new image may also include imaging time information, and the feature points may be managed in chronological order in the feature point database.

[0036] The display control unit 45 has a function of generating a three-dimensional image based on existing feature points and / or existing images stored in the storage unit 35, and displaying the generated three-dimensional image on the display unit 33. For example, when the feature point database is updated by the DB update unit 44, the display control unit 45 updates the three-dimensional image using newly acquired feature points and images including the feature points. In addition, the display control unit 45 can generate a three-dimensional image based on feature points at a predetermined time point in the past, or display three-dimensional images at a plurality of time points in a comparative manner, using imaging time information assigned to the feature points. In other words, the display control unit 45 can generate three-dimensional images according to a time series.

[0037] The selection unit 46 has a function of selecting a work target object or a work point to be displayed superimposed on the new image. Specifically, the worker A can select a work target object or a work point by operating the operation unit 32 of the guide terminal 2 or by inputting voice from the microphone of the voice unit 38. Note that a plurality of work target objects or work points may be selected. The following description mainly uses a work target object as an example.

[0038] In order to reduce the workload of the worker A in selecting using the selection unit 46, the display control unit 45 may have a function of suggesting on the display unit 33 the work target object to be selected by the worker A in the selection unit 46 based on the position information of the shooting position of the new image. The selection unit 46 can calculate the distance information between the planned installation position of the work target object and the shooting position of the new image to suggest the work target object that the worker A should select, and the work target object with a short distance. The suggestion may be made by displaying the name of the candidate work target object on the display unit 33, or by suggesting the name of the candidate work target object by voice using the audio unit 38. The worker A can select the suggested candidate by operating the operation unit 32 to select the suggested candidate, or by replying by voice using the microphone of the audio unit 38. This can reduce the workload of the worker at the work site.

[0039] Furthermore, in order to reduce the workload of worker A when selecting using the selection unit 46, the selection unit 46 may have a function of automatically selecting a work target object to be selected based on position information of the shooting position of the new image. For example, the selection unit 46 can calculate distance information between the planned installation position of the work target object to be selected by worker A and the shooting position of the new image, and automatically select the one work target object with the shortest distance. This can further reduce the workload of the worker at the work site.

[0040] The guide calculation unit 47 has a function of outputting guide information to guide the worker A to approach the planned installation position of the work target object from the shooting position of the new image based on the position information of the planned installation position of the work target object selected by the selection unit 46 and the position and orientation information of the shooting position of the new image. In other words, assuming that the shooting position of the new image is the current position of the worker A, if this position is identified, the relative relationship with the planned installation position of the work target object can be identified, so it is possible to output guide information to guide the worker A to approach the planned installation position of the work target object from the shooting position of the new image.

[0041] The imaging range calculation unit 48 has a function of calculating the imaging range of a new image or the position of an arbitrary point within the imaging range.

[0042] The display control unit 45 further has a function of comparing a position in the imaging range in the absolute coordinate system or the coordinate system specific to the work site with a planned installation position of the object to be worked on, and when the planned installation position of the object to be worked on is included in the imaging range, superimposing and displaying the object to be worked on at a position corresponding to the planned installation position of the object to be worked on on the new image displayed on the display unit 33. Specifically, when a new image acquired on the display unit 33 is displayed, or when the imaging range of the new image can also be displayed on the display unit 33 sequentially, the display range becomes equal to the imaging range. At that time, if the object to be worked on is located in the imaging range, i.e., in the display range, it is possible to display the position by superimposing it on the displayed image. Such processing is possible by calculating the position of the imaging range in the absolute coordinate system or the coordinate system specific to the work site.

[0043] Furthermore, the display control unit 45 further has a function of calculating the difference in distance between an arbitrary point in the imaging range of the new image and the intended installation position of the object to be worked on, using position information of the intended installation position of the object to be worked on and position information in the absolute coordinate system of an arbitrary point in the imaging range of the new image or in the coordinate system specific to the work site, and outputting guide information to the worker A, guiding the worker A to approach the intended installation position of the object to be worked on from the imaging position of the new image based on this difference in distance. Specifically, the display unit 33 can display up, down, left, right arrows or a specific distance (e.g., how many meters left), or output such guidance by voice.

[0044] In addition, if the imaging unit 30 is a stereo camera, a depth camera, a ToF camera, a LIDAR integrated camera, or the like, when a new image captures a state in which a worker is holding a work target object, the shape of the work target object is image-recognized, the distance between the image-recognized work target object and the imaging unit 30 (current position) and the distance between the planned installation position and the current position are calculated, and it is calculated how far the worker needs to move the image-recognized work target object to reach the planned installation position, and guide information is output that guides the worker to approach the object based on the calculation.

[0045] With the above-described configuration, the guide system 1 can output guide information to the operator A via the guide terminal 2.

[0046] Next, FIG. 3 shows a flowchart showing the procedure of the guide executed by the control unit 36. Hereinafter, the guide method of this embodiment will be described based on this flowchart with reference to FIG.

[0047] First, in step S1, the feature extraction unit 40 of the control unit 36 ​​causes the imaging unit 30 to capture a new image Pn. As shown in Fig. 1, worker A captures an image at an arbitrary measurement point X using the guide terminal 2 in response to an operation via the operation unit 32. Strictly speaking, the mounting positions of the imaging unit 30 on the rod-shaped portion 22 and the main body portion 21 are also calculated, but to simplify the explanation, the measurement point X will be explained as the imaging position of the new image.

[0048] In step S2, the feature extraction unit 40 of the control unit 36 ​​extracts feature points from the new image Pn. The feature extraction unit 40 also assigns imaging situation information that can be assigned at that time, such as imaging time information, to the extracted feature points. In Fig. 1, five feature points Fn1 to Fn5 corresponding to multiple corners of a building T, which is a site reference object used as a reference for an absolute position at a work site, are extracted as feature points of the new image Pn.

[0049] In steps S3 to S5, the feature matching unit 41 of the control unit 36 ​​uses a plurality of feature points included in the new image to extract and acquire a predetermined number or more of feature points (common feature points) that are common to the feature points included in the new image from among a plurality of existing feature points that are stored in the storage unit 35 and whose position information has already been measured by a surveying device. Then, for the common feature points, the positional relationship between the existing feature points and the feature points included in the new image is associated, and for feature points that are newly detected as feature points different from the existing feature points, position information in the absolute coordinate system or a coordinate system specific to the work site is assigned based on the position information assigned to the existing feature points.

[0050] To this end, in step S3, the feature matching unit 41 extracts existing feature points that are common to the feature points included in the new image from the multiple existing feature points stored in the storage unit .

[0051] As shown in FIG. 4, the feature matching unit 41 extracts existing feature points Fe1 to Fe3 that are common to feature points Fn1 to Fn5 extracted from a building T reflected in the new image.

[0052] In step S4, when matching feature points included in the new image with existing feature points, the feature matching unit 41 determines whether there are a required number of feature points common to the feature points included in the new image and all existing feature points stored in the storage unit 35. If there are not a required number of common feature points stored in the storage unit 35, it is necessary to capture a new new image, and the process returns to step S1.

[0053] After acquiring the existing feature points in step S3, if there are a necessary number or more common feature points in step S4, in step S5, the feature matching unit 41 of the control unit 36 ​​matches the existing feature points Fe with the feature points Fn of the new image Pn. Specifically, the feature matching unit 41 compares the feature amounts of both feature points, and associates the feature points Fn of the new image Pn that correspond to the same feature points as the feature points Fe of the existing image Pe, whose feature amounts match or are close to each other. For example, in FIG. 4, feature points at positions that match between the existing image Pe and the new image Pn are extracted for the building T in both images. In FIG. 4, the existing feature points Fe1 to Fe3 of the existing image Pe match the feature points Fn1 to Fn3 of the new image Pn, and the feature matching unit 41 associates the existing feature points Fe1 and Fn1, the existing feature points Fe2 and Fn2, and the existing feature points Fe3 and Fn3 as common feature points, respectively. Note that, while FIG. 4 shows an example in which one existing image Pe contains three feature points Fe1 to Fe3, the common feature points are not limited to existing feature points extracted from one image as long as they can match a predetermined number or more of any of all existing feature points stored in the storage unit 35.

[0054] In step S6, the imaging position calculation unit 42 of the control unit 36 ​​calculates the imaging position of the new image using the common feature points between the new image and the existing image that were associated in step S5. That is, as shown in Fig. 4, the imaging position calculation unit 42 calculates imaging conditions such as the position coordinates of the imaging position (measurement point X) and the imaging attitude from the associated feature points Fn1 to Fn3 using existing feature points Fe1 to Fe3 whose position information is known and that have been extracted from an existing image Pe that has been captured and acquired in advance while tracking the prism 23 of the guide terminal 2 by the surveying device 5.

[0055] Here, in step S6, the differential feature identifying section 43 of the control section 36 may extract differential feature points that are not common to the existing feature points from among the feature points extracted from the new image, and may assign position information of the associated feature points to the differential feature points based on the imaging position of the new image calculated in step S6. That is, as shown in Fig. 4, the differential feature identifying section 43 may extract differential feature points Fn4 and Fn5 that are not associated with the feature points Fe1 to Fe3 of the existing image from among the feature points Fn1 to Fn5 included in the new image Pn, and store them in the storage section 35.

[0056] Also, the DB update unit 44 of the control unit 36 ​​may update the feature point DB by adding information about the new image Pn to the storage unit 35. Specifically, the DB update unit 44 may add to the storage unit 35 a new image Pn having feature points Fn1 to Fn3 associated with the existing feature points Fe1 to Fe3 and feature points Fn4 and Fn5 to which relative position information with respect to the associated feature points Fn1 to Fn3 has been added. Also, each of the feature points Fn1 to Fn5 of the new image Pn includes imaging situation information such as imaging time information and imaging posture. The new image Pn including this information may be stored in the storage unit 35 as an existing image.

[0057] In step S7, worker A manually operates the operation unit 32 to give instructions to the selection unit 46, or the selection unit 46 automatically selects a work target object from among multiple objects stored in the memory unit 35, the intended installation location of which should be guided to worker A.

[0058] Here, in order to reduce the workload of worker A, the display control unit 45 may suggest on the display unit 33 a work target object that should be selected by worker A in the selection unit 46 based on absolute position information of the shooting position of the new image. This process can compare the planned installation position information of the work target object with the position information of the shooting position of the new image, and suggest one with a small difference.

[0059] The selection unit 46 may automatically select the work target object to be selected based on the position information of the shooting position of the new image. As in the above, it is possible to compare the position information of the planned installation position of the work target object with the position information of the shooting position of the new image, and automatically select the one with the smallest difference.

[0060] The selected work target object includes or is linked to position information that is the planned installation position of the selected work target object. Therefore, in step S8, the guide calculation unit 47 calculates the difference between the imaging position of the new image and the planned installation position of the selected work target object.

[0061] Then, in step 9, the guide calculation unit 47 outputs guide information to the planned installation position, with the current position being the imaging position of the new image, according to the difference calculated in step 8. Various methods can be used to specifically output the guide information.

[0062] For example, Fig. 5 shows an example of a new image Pn displayed on the display unit 33 of the guide terminal 2. In this figure, planned installation positions wp1 and wp2 of the work target object are displayed on the new image Pn. The planned installation positions may be superimposed and displayed on such a new image to serve as guide information.

[0063] In order to output such guide information, the display control unit 45 compares the position in the imaging range in the absolute coordinate system calculated by the imaging range calculation unit 48 or the coordinate system specific to the work site with the position of the planned installation location of the object to be worked on, and if the planned installation location of the object to be worked on is included in the imaging range, displays the object to be worked on superimposed on the new image displayed on the display unit 33 at a position equivalent to the planned installation location of the object to be worked on.

[0064] In step S9, the guide calculation unit 47 outputs the guide information, and then the routine is returned.

[0065] As described above, according to the guide system 1, guide terminal 2, guide method, and guide program of this embodiment, it is possible to take a new image and match the acquired feature points with the existing feature points by taking the new image without directly using a surveying device, etc., by using existing feature points whose position information is known in advance with high accuracy by a surveying device, etc., calculate the imaging position of the new image, and provide guidance for the work target position based on this imaging position. This allows the worker to intuitively understand the work target location on the spot from his / her viewpoint while ensuring high positional accuracy in the absolute coordinate system or the coordinate system specific to the work site.

[0066] FIG. 6 also shows an application example of the guide system and the guide method shown in FIG. In this application example, the absolute position of the range captured by the new image is also calculated, and when it is desired to align the position of the object to be worked on with an arbitrary point in the new image, such as the center, the difference between the intended installation position and the arbitrary point in the new image is calculated, and guide information can be output so that the arbitrary point approaches the intended installation position.

[0067] In this case, the guide terminal 2 has a display unit 33 which displays the new image captured by the imaging unit 30, an imaging range calculation unit 48 which calculates the absolute position of the imaging range of the new image using absolute position information of the imaging position of the new image and the imaging attitude, and compares the absolute position of the imaging range with the absolute position of the planned installation position of the object to be worked on, and when the planned installation position of the object to be worked on is included in the imaging range, displays the object to be worked on superimposed on the position on the new image displayed on the display unit 33 which corresponds to the planned installation position of the object to be worked on.

[0068] This makes it easier for the worker to intuitively grasp the position of the work target object in real space by comparing the situation of the work site that can currently be seen with the naked eye with the planned installation position of the work target object displayed superimposed on the display unit 33.

[0069] FIG. 9 also shows an application example of the guide system and the guide method shown in FIG. In this application example, when the work target objects Wp1-2 are displayed on the display unit 33, if the work target objects are objects having a spatial size such as pillars or building materials, the spatial size of the objects may also be displayed. In this case, the scale of the displayed objects may be corrected using the distance between the shooting position and the work target position.

[0070] FIG. 10 shows a configuration diagram of a guide system according to a modified example, and a guide system 1A according to the modified example will be described with reference to this drawing.

[0071] The guide terminal 2A of the modified example is a wearable device whose main body can be attached to the head of the worker A, and includes a display unit 33 that is a head mounted display (HMD) capable of so-called MR (mixed reality) display. The imaging unit 30 is provided corresponding to the mounting position of the display unit 33 so as to capture an image in the same direction as the direction in which the worker A is pointing. The other configurations are the same as those of the above embodiment, and therefore description thereof will be omitted.

[0072] Further, as a means for outputting guide information, a voice unit 38 is provided. The voice unit 38 is, for example, a speaker, and can output guide information to the worker A by voice.

[0073] In the guide system 1A in the modified example, for example, the imaging unit 30 captures moving images, and the control unit 36 ​​(not shown) updates the feature point DB in real time and causes the display unit 33 to display a three-dimensional image in MR.

[0074] With the above-mentioned configuration, the feature point DB is updated by the worker A simply by mounting the guide terminal 2A on his head and pointing it at the surroundings while moving around the work site, and by capturing an image of feature points Fe7 to Fe9, which were captured in the past at X1, and are linked to position information, from another location, the current location X2 can be identified, and based on the updated feature points, the three-dimensional image can be updated using the image captured from X2, and can be confirmed in real time via the display unit 33. This makes it even easier to check the work site and grasp the state of the terrain.

[0075] In this way, the guide system 1A, the guide terminal 2A, and the guide method and guide program using the guide system 1A can efficiently manage the work performed by the worker A.

[0076] In the above embodiment and modified example, the guide system 1A is realized by the guide terminal 2A alone, but the present invention is not limited to such a configuration, and some of the configuration may be included in an external device. For example, the feature point DB may be constructed in a storage unit of an external management server. In this case, the management server and one or more data management devices are connected to each other so that they can communicate with each other. According to the data management system configured in this way, the feature point DB can be easily shared with multiple guide terminals, making it easier to check the work site and grasp the state of the terrain.

[0077] This completes the description of the embodiment of the present disclosure, but the aspects of the present disclosure are not limited to this embodiment.

[0078] An example of the configuration of this embodiment is as follows. [1] A guide system for assisting a worker in a work site, comprising: a storage unit that stores design information including position information of planned installation positions of a plurality of objects or points that are elements constituting a planned construction at the work site, and a plurality of existing feature points and position information of site reference objects to be used as a reference for positions at the work site obtained as a result of surveying by a surveying device; a guide terminal held by the worker, the guide terminal having an imaging unit that captures a new image and an attitude detection unit that detects imaging direction information of the new image, The guide terminal includes a feature extraction unit that extracts a plurality of feature points from the new image; a feature matching unit that uses a plurality of feature points included in the new image to extract a predetermined number or more of common feature points that are common to the feature points from a plurality of existing feature points stored in the storage unit, and associates the common feature points with the feature points; an imaging position calculation unit that calculates position information of an imaging position and an imaging attitude of the new image using position information of the associated common feature points and the new image; a selection unit for selecting a work target object or a work target point, the planned installation position of which should be guided, from the plurality of objects or points stored in the storage unit; A guide system comprising: a guide calculation unit that outputs, to the worker, guide information from the shooting position of the new image to the planned installation position of the selected work target object or work target point, based on position information of the planned installation position of the selected work target object or work target point, and position information of the shooting position of the new image and the shooting posture. [2] The guide terminal includes a display unit that displays a new image captured by the imaging unit; an imaging range calculation unit that calculates an imaging range of the new image in an absolute coordinate system or a coordinate system specific to the work site using position information of an imaging position of the new image and an imaging attitude; The absolute position of the imaging range is compared with the absolute position of a planned installation position of the work target object or the work target point, and when the planned installation position of the work target object or the work target point is included in the imaging range, The guide system of [1] further comprises a display control unit that displays the work target object or work target point superimposed on a position on the new image displayed on the display unit that corresponds to the planned installation position of the work target object or work target point. [3] The display control unit uses position information of the planned installation position of the work target object or work target point and absolute position information of an arbitrary point in the imaging range in the absolute coordinate system of the new image or a coordinate system specific to the work site to calculate the difference between the arbitrary point in the imaging range and the planned installation position of the work target object or work target point, and outputs guide information to the worker that guides the worker from the shooting position of the new image to the planned installation position of the work target object or work target point based on the difference. [4] The guide system of any one of [2] to [3], wherein the display control unit suggests on the display unit the work target object or work target point that the worker should select in the selection unit based on location information of the shooting location of the new image. [5] The guide system according to any one of [1] to [3], wherein the selection unit automatically selects the work target object or work target point to be selected based on position information of the shooting position of the new image. [6] The guide terminal further includes a voice unit that generates a voice. The guide system according to [1], wherein the audio unit outputs the guide information output by the guide calculation unit by voice. [7] A guide method for assisting a worker in a work site, comprising: Design information including a plurality of objects or points that are elements constituting the planned construction at the work site and position information of the planned installation positions of the objects or points; a storage step of storing, in a storage unit, a plurality of existing feature points and position information of a site reference object to be used as a reference for a position at the work site, which are obtained as a result of surveying by the surveying device; an imaging step in which the worker captures a new image using a guide terminal having an imaging unit; a new image feature point extraction step of extracting a plurality of feature points from the new image by a feature extraction unit; a feature matching step of extracting a predetermined number or more of common feature points that are common to the feature points included in the new image from among the plurality of existing feature points stored in the storage unit, using the plurality of feature points included in the new image, and associating the common feature points; an imaging position calculation step of calculating, by an imaging position calculation unit, position information of an imaging position and an imaging attitude of the new image using position information of the associated common feature points and the new image; a selection step in which a selection unit selects a work target object or a work target point, the planned installation position of which is to be guided, from the plurality of objects or points stored in the storage unit; A guide method comprising: a guide step in which a guide calculation unit outputs, to the worker, guide information from the imaging position of the new image to the planned installation position of the work target object or work target point, based on absolute position information of the planned installation position of the selected work target object or work target point, and position information of the shooting position of the new image and the imaging posture. [8] A guide program installed in a guide terminal held by a worker in a guide system that assists a worker in a work site, comprising: a storage step of storing, in a storage unit, a plurality of existing feature points and position information of a site reference object to be used as a reference for a position at the work site, which are obtained as a result of surveying by the surveying device; a new image acquisition step in which the worker acquires a new image captured by using a guide terminal having an imaging unit; a new image feature point extraction step of extracting a plurality of feature points from the new image by a feature extraction unit; a feature matching step of extracting a predetermined number or more of common feature points that are common to the feature points included in the new image from among the plurality of existing feature points stored in the storage unit, using the plurality of feature points included in the new image, and associating the common feature points; an imaging position calculation step of calculating absolute position information of an imaging position of the new image and an imaging attitude of the new image by an imaging position calculation unit using position information of the associated common feature points and the new image; a selection step in which a selection unit selects a work target object, the planned installation position of which is to be guided, from the plurality of objects or points stored in the storage unit; A guide program for causing a computer to execute a guide step in which a guide calculation unit outputs, to the worker, guide information from the imaging position of the new image to the planned installation position of the work target object or work target point selected, based on absolute position information of the planned installation position of the work target object or work target point, and position information of the shooting position of the new image and the imaging posture. [Explanation of symbols]

[0079] 1 Guide System 2 Guide terminal 21 Main body 22 Rod-shaped part 23 Prism 30 Imaging unit 31 Attitude detection unit 32 Operation section 33 Display section 34 Communications Department 35 Storage section 36 Control Unit 40 Feature Extraction Unit 41 Feature Matching Unit 42 Shooting position calculation unit 43 Differential feature identification unit 44 DB update section 45 Display control section

Claims

1. A guide system for assisting a worker in a work site, comprising: a storage unit that stores design information including position information of planned installation positions of a plurality of objects or points that are elements constituting a planned construction at the work site, and a plurality of existing feature points and position information of site reference objects to be used as a reference for positions at the work site obtained as a result of surveying by a surveying device; a guide terminal held by the worker, the guide terminal having an imaging unit that captures a new image and an attitude detection unit that detects imaging direction information of the new image, The guide terminal includes a feature extraction unit that extracts a plurality of feature points from the new image; a feature matching unit that uses a plurality of feature points included in the new image to extract a predetermined number or more of common feature points that are common to the feature points from a plurality of existing feature points stored in the storage unit, and associates the common feature points with the feature points; an imaging position calculation unit that calculates position information of an imaging position and an imaging attitude of the new image using position information of the associated common feature points and the new image; a selection unit for selecting a work target object or a work target point, the planned installation position of which should be guided, from the plurality of objects or points stored in the storage unit; A guide system comprising: a guide calculation unit that outputs, to the worker, guide information from the shooting position of the new image to the planned installation position of the selected work target object or work target point, based on position information of the planned installation position of the selected work target object or work target point, and position information of the shooting position of the new image and the shooting posture.

2. The guide terminal includes a display unit that displays a new image captured by the imaging unit; an imaging range calculation unit that calculates an imaging range of the new image in an absolute coordinate system or a coordinate system specific to the work site using position information of an imaging position of the new image and an imaging attitude; The position of the imaging range is compared with the position of a planned installation position of the work target object or the work target point, and when the planned installation position of the work target object or the work target point is included in the imaging range, The guide system of claim 1, further comprising a display control unit that displays the work target object or work target point superimposed on a position on the new image displayed on the display unit that corresponds to the planned installation position of the work target object or work target point.

3. The display control unit uses position information of the planned installation position of the work target object or work target point and position information of an arbitrary point in the imaging range in the absolute coordinate system of the new image or a coordinate system specific to the work site to calculate a difference between the arbitrary point in the imaging range and the planned installation position of the work target object or work target point, and outputs guide information to the worker that guides the worker from the shooting position of the new image to the planned installation position of the work target object or work target point based on the difference.

4. A guide system as described in any one of claims 2 to 3, wherein the display control unit suggests on the display unit the work target object or work target point that the worker should select in the selection unit based on location information of the shooting location of the new image.

5. The guide system according to any one of claims 1 to 3, wherein the selection unit automatically selects the work target object or work target point to be selected based on position information of a shooting position of the new image.

6. The guide terminal further includes a voice unit that generates a voice. The guide system according to claim 1 , wherein the audio unit outputs the guide information output by the guide calculation unit by voice.

7. A guide method for assisting a worker in a work site, comprising: Design information including a plurality of objects or points that are elements constituting the planned construction at the work site and position information of the planned installation positions of the objects or points; a storage step of storing, in a storage unit, a plurality of existing feature points and position information of a site reference object to be used as a reference for a position at the work site, which are obtained as a result of surveying by the surveying device; an imaging step in which the worker captures a new image using a guide terminal having an imaging unit; a new image feature point extraction step of extracting a plurality of feature points from the new image by a feature extraction unit; a feature matching step of extracting a predetermined number or more of common feature points that are common to the feature points included in the new image from among the plurality of existing feature points stored in the storage unit, using the plurality of feature points included in the new image, and associating the common feature points with the feature points included in the new image; an imaging position calculation step of calculating, by an imaging position calculation unit, position information of an imaging position and an imaging attitude of the new image using position information of the associated common feature points and the new image; a selection step in which a selection unit selects a work target object or a work target point, the planned installation position of which is to be guided, from the plurality of objects or points stored in the storage unit; A guide method comprising: a guide step in which a guide calculation unit outputs, to the worker, guide information from the imaging position of the new image to the planned installation position of the work target object or work target point, based on absolute position information of the planned installation position of the selected work target object or work target point, and position information of the shooting position of the new image and the imaging posture.

8. A guide program installed in a guide terminal held by a worker in a guide system that assists a worker in a work site, comprising: a storage step of storing, in a storage unit, a plurality of existing feature points and position information of a site reference object to be used as a reference for a position at the work site, which are obtained as a result of surveying by the surveying device; a new image acquisition step in which the worker acquires a new image captured by using a guide terminal having an imaging unit; a new image feature point extraction step of extracting a plurality of feature points from the new image by a feature extraction unit; a feature matching step of extracting a predetermined number or more of common feature points that are common to the feature points included in the new image from among the plurality of existing feature points stored in the storage unit, using the plurality of feature points included in the new image, and associating the common feature points with the feature points included in the new image; an imaging position calculation step of calculating absolute position information of an imaging position of the new image and an imaging attitude of the new image by an imaging position calculation unit using position information of the associated common feature points and the new image; a selection step in which a selection unit selects a work target object, the planned installation position of which is to be guided, from the plurality of objects or points stored in the storage unit; A guide program for causing a computer to execute a guide step in which a guide calculation unit outputs, to the worker, guide information from the imaging position of the new image to the planned installation position of the work target object or work target point selected, based on absolute position information of the planned installation position of the work target object or work target point, and position information of the shooting position of the new image and the imaging posture.

Images