Information processing system and information processing method

Abstract

The objective is to provide an information processing system and information processing method that can promote labor saving and automation even in environments where communication is difficult. [Solution] The information processing system is characterized by comprising: an imaging unit 11 positioned on the side of a tunnel T, which captures images of at least three markers 2 whose positions in a desired absolute coordinate system are known, and acquires a still image 4A; a three-dimensional measurement unit 12 located inside the tunnel T, which performs three-dimensional measurement of a measurement target whose position in an absolute coordinate system is unknown, and acquires point cloud data 5 in a measurement coordinate system; a target identification means (measurement control unit 15) for identifying the markers 2 in the still image 4A; and a coordinate transformation means (measurement control unit 15) for converting the point cloud data 5 acquired by the three-dimensional measurement unit 12 into an absolute coordinate system based on position coordinate information indicating the position of the identified markers 2 in an absolute coordinate system.

Description

Technical Field

[0001] The present invention relates to an information processing system and an information processing method that output point cloud data of an absolute coordinate system suitable for a construction machine corresponding to machine guidance, for example, inside a tunnel where it is difficult to perform position measurement using a global navigation satellite system.

Background Art

[0002] In recent years, for example, at construction sites and work sites, there is a demand for labor saving, reduction of the burden on workers by automation, and improvement of work efficiency. Therefore, various support technologies that display various information based on the current position to support the operation of the user, and automatic driving technologies based on the current position have been proposed.

[0003] For example, Patent Document 1 discloses machine guidance that supports the operation of an operator by displaying three-dimensional data of a construction site obtained by three-dimensional measurement of the construction site and three-dimensional data indicating the design information of the construction site on a display of a construction machine.

[0004] In this Patent Document 1, by associating the coordinate information of an absolute coordinate system obtained by position measurement using a global navigation satellite system with the current position of the construction machine and the three-dimensional data of the construction site, it is possible to compare with the three-dimensional data of the design information with which the coordinate information of the absolute coordinate system has been associated in advance.

[0005] However, at sites where the communication state with the outside is unstable, such as inside a tunnel, or at sites where communication is difficult, such as sites where communication with the outside is impossible, position measurement using a global navigation satellite system is restricted. Therefore, for example, there is a problem that it is impossible to associate coordinate information of a high-precision absolute coordinate system with the three-dimensional data obtained by three-dimensional measurement of the construction site. Then, when the site where labor saving and automation are required is a site where communication with the outside is difficult, various support technologies and automatic driving technologies cannot be fully utilized, so there is room for improvement.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Patent Publication No. 2021-170719 [Overview of the project] [Problems that the invention aims to solve]

[0007] In view of the above-mentioned problems, the present invention aims to provide an information processing system and an information processing method that can promote labor saving and automation even in environments where communication is difficult. [Means for solving the problem]

[0008] This invention is an information processing system comprising: an imaging unit that captures images of at least three targets, which are placed on a stationary object in a target space and whose positions in a desired predetermined coordinate system are known, and acquires imaging information; a three-dimensional measurement unit that three-dimensionally measures a measurement target located in the target space and whose position in the predetermined coordinate system is unknown, and acquires point cloud data in the measurement coordinate system; a target identification means that identifies the target in the imaging information; and a coordinate transformation means that converts the point cloud data acquired by the three-dimensional measurement unit into the predetermined coordinate system based on position coordinate information indicating the position of the identified target in the predetermined coordinate system.

[0009] Furthermore, this invention is an information processing method characterized by comprising: an imaging step in which an imaging unit images at least three or more targets that are placed in a stationary object in a target space and whose position in a desired predetermined coordinate system is known, and acquires imaging information; a three-dimensional measurement step in which a three-dimensional measurement unit performs three-dimensional measurement on a measurement target located in the target space and whose position in the predetermined coordinate system is unknown, and acquires point cloud data of the measurement coordinate system; a target identification step in which a target identification means identifies the target in the imaging information; and a coordinate transformation step in which a coordinate transformation means transforms the point cloud data acquired by the three-dimensional measurement unit into the predetermined coordinate system based on position coordinate information indicating the position of the identified target in the predetermined coordinate system.

[0010] The aforementioned predetermined coordinate system refers to an absolute coordinate system (also called a reference coordinate system) with the Earth's center of mass as its origin, or an arbitrary coordinate system with any predetermined location as its origin. The above-mentioned target space refers to, for example, a space formed by excavating underground, the interior of a building, or a predetermined area outdoors. The above-mentioned targets refer to objects that can be identified based on imaging information, such as AR markers attached to inanimate objects in the target space, two-dimensional barcodes or predetermined patterns, or three-dimensional structures placed on inanimate objects in the target space.

[0011] The above imaging information refers to video or still images captured of the target space. The objects of measurement mentioned above refer to the walls, floors, people, machines, and devices that make up the target space. The above-mentioned measurement coordinate system refers to a coordinate system whose position in a predetermined coordinate system is unknown, for example, a coordinate system that uses the position of the three-dimensional measurement unit as the reference point.

[0012] According to this invention, point cloud data in a measurement coordinate system can be converted to a predetermined coordinate system based on the position coordinate information of a target in a predetermined coordinate system. For example, point cloud data in a desired predetermined coordinate system can be obtained without using a global navigation satellite system.

[0013] Furthermore, as long as the three-dimensional measurement unit is not moved, the relative position between the target and the three-dimensional measurement unit does not change. Therefore, the information processing system and information processing method can stably and repeatedly acquire point cloud data in a predetermined coordinate system that corresponds to the state of the target space, even if the state of the target space changes over time, such as in tunnel excavation work.

[0014] Even if the three-dimensional measurement unit is moved, the information processing system and information processing method can easily reacquire point cloud data in a predetermined coordinate system based on the target's position coordinate information.

[0015] As a result, the information processing system and information processing method can sequentially acquire point cloud data of a desired predetermined coordinate system, even in environments where communication with the outside is difficult, environments where the conditions within the target space are constantly changing, or environments where the three-dimensional measurement unit needs to be moved.

[0016] Therefore, the information processing system and information processing method can acquire point cloud data in a predetermined coordinate system that can be used for various support technologies and autonomous driving technologies, regardless of the state of communication with the outside world. This promotes labor saving and automation even in environments where communication is difficult.

[0017] In one aspect of this invention, the three-dimensional measurement unit is configured to perform three-dimensional measurement within the target space which includes at least three targets, and the coordinate transformation means assigns the position coordinate information of the target to corresponding measurement points that correspond to the target identified by the target identification means among the measurement points that constitute the point cloud data, and transforms the point cloud data acquired by the three-dimensional measurement unit into the predetermined coordinate system based on the position coordinate information assigned to the corresponding measurement points.

[0018] With this configuration, position coordinate information of the target is assigned to the corresponding measurement point in the point cloud data corresponding to the target of the imaging information, making it possible to obtain position coordinate information indicating the position of the corresponding measurement point in a predetermined coordinate system with high accuracy.

[0019] Furthermore, since both the imaging information and the point cloud data include three or more targets, the information processing system can accurately calculate position coordinate information indicating the position of the three-dimensional measurement unit in a predetermined coordinate system, as well as position coordinate information indicating the positions of measurement points other than the corresponding measurement points in the predetermined coordinate system, based on the position coordinate information of three or more predetermined coordinate systems assigned to the point cloud data. This allows the information processing system to accurately convert the point cloud data acquired by the three-dimensional measurement unit into a desired predetermined coordinate system.

[0020] As an aspect of the present invention, the coordinate conversion means may be configured to calculate position coordinate information indicating the position of the three-dimensional measurement unit in the predetermined coordinate system based on the position coordinate information of the corresponding measurement points, and to convert the point cloud data acquired by the three-dimensional measurement unit into the predetermined coordinate system based on the position coordinate information of the three-dimensional measurement unit.

[0021] According to this configuration, since the position coordinate information of the three-dimensional measurement unit in the predetermined coordinate system can be obtained, once the position coordinate information of the three-dimensional measurement unit is calculated, in subsequent processing, the position coordinate information of the predetermined coordinate system can be easily associated with the measurement points of the point cloud data acquired by the three-dimensional measurement unit. Thereby, since the information processing system can efficiently and continuously acquire the point cloud data of the predetermined coordinate system, it can acquire the point cloud data suitable for various support technologies and autonomous driving technologies.

[0022] As an aspect of the present invention, the imaging unit and the three-dimensional measurement unit may be provided integrally. According to this configuration, since the three-dimensional measurement unit can perform three-dimensional measurement at substantially the same position as the imaging unit within the target space including at least three or more targets, it is possible to acquire point cloud data overlapping the imaging information.

[0023] Therefore, the information processing system can not only easily identify the corresponding measurement points corresponding to the targets among the measurement points of the point cloud data, but also accurately acquire the position coordinate information of the corresponding measurement points in the predetermined coordinate system. Thereby, since the information processing system can improve the calculation accuracy of the position coordinate information of the three-dimensional measurement unit, it can acquire the point cloud data of the predetermined coordinate system with high precision.

[0024] As an aspect of the present invention, display means for superimposing and displaying the virtual object of the predetermined coordinate system on the point cloud data of the predetermined coordinate system may be provided. The above-mentioned virtual objects are objects that are represented by lines, images, frames, or polyhedra, and include the shape of the target space, objects within the target space, virtual objects placed within the target space, completed shapes and reference lines based on design drawings, and guide lines that guide the movement of moving objects. The above-mentioned display means refers to display means on a mobile device held by a user within the target space, display means on construction machinery, or display means on an information processing device located outside the target space.

[0025] This configuration allows for the overlaying and display of virtual objects on point cloud data obtained by three-dimensional measurement within the target space, making it easier, for example, to assist users in directly or remotely operating construction machinery. This allows information processing systems to easily reduce labor even in environments with poor communication, thereby reducing the burden on users and improving work efficiency.

[0026] Another embodiment of this invention may consist of a portable measuring device having the imaging unit, the three-dimensional measuring unit, the target identification means, and the coordinate transformation means, and the display means connected to the measuring device via a communication line.

[0027] With this configuration, even when the target space is progressively expanded, such as in tunnel excavation work, the imaging unit, three-dimensional measurement unit, target identification means, and coordinate transformation means can be easily moved in accordance with the expansion of the target space.

[0028] Furthermore, since the measurement device performs a series of processes from imaging and three-dimensional measurement within the target space to acquiring point cloud data in a predetermined coordinate system, the information processing system can utilize existing display means, such as mobile terminals or the display units of construction machinery, as display means for displaying point cloud data in a predetermined coordinate system. This allows the information processing system to improve user convenience through portable measuring devices while keeping costs down by utilizing existing display methods.

[0029] Furthermore, in an embodiment of this invention, the invention may include: a motion identification means for identifying a moving object in the target space based on the imaging information or the point cloud data of the predetermined coordinate system; a motion coordinate acquisition means for acquiring position coordinate information indicating the position of the moving object in the predetermined coordinate system based on the point cloud data of the predetermined coordinate system; and a data output means for outputting the position coordinate information of the moving object in association with the point cloud data of the predetermined coordinate system.

[0030] The term "moving object" refers to an object within the target space whose position changes over time, either as a whole or as a part of an object, such as a person, a vehicle, a transport device, the entirety of a construction machine, or the movable parts of a construction machine.

[0031] With this configuration, the point cloud data of a predetermined coordinate system is output with the position coordinate information of a moving object associated with it. Therefore, the point cloud data associated with the position coordinate information of a moving object can be used, for example, as three-dimensional data to inform the user of the presence and position of a moving object in the target space. Alternatively, an information processing system can use point cloud data, which associates the position coordinate information of a moving object, as three-dimensional data for controlling the movement of, for example, construction machinery.

[0032] This allows the information processing system to utilize point cloud data in a predetermined coordinate system as, for example, three-dimensional data to assist users in operating construction machinery, or three-dimensional data for autonomous operation of construction machinery. As a result, labor savings and automation can be further promoted even in environments where communication is difficult.

[0033] Furthermore, in an embodiment of this invention, an intrusion determination means may be provided that determines whether the moving body has entered a restricted area set within the target space based on the position coordinate information of the moving body acquired by the moving body coordinate acquisition means, and a notification means that notifies the intrusion into the restricted area if the determination result by the intrusion determination means is that the moving body has entered the restricted area.

[0034] This configuration allows for notification of the intrusion of moving objects into a restricted area based on point cloud data in a predetermined coordinate system, without the need to place sensors or other devices within the target space. This allows the information processing system to improve safety within the target space with a simple configuration that keeps costs down.

[0035] In another aspect of this invention, the invention may also include a movement determination means that determines whether the moving body has moved outside a restricted area set within the target space based on the position coordinate information of the moving body acquired by the moving body coordinate acquisition means, and a notification means that notifies the movement outside the restricted area if the determination result by the movement determination means is that the body has moved outside the restricted area.

[0036] This configuration allows for notification of the movement of a moving object outside a restricted area based on point cloud data in a predetermined coordinate system, without the need to place sensors or other devices within the target space. This allows the information processing system to improve safety within the target space with a simple configuration that keeps costs down.

[0037] In another aspect of this invention, the target may be positioned on the inner surface of the target space formed by excavating underground, and may be provided with positional coordinate information in an absolute coordinate system. The inner surface of the target space formed by excavating underground as described above refers to, for example, the bottom, sides, or top of a tunnel.

[0038] This configuration allows for easy acquisition of desired point cloud data in an absolute coordinate system in underground spaces where communication with the outside world is difficult. Therefore, the information processing system can utilize point cloud data in an absolute coordinate system as three-dimensional data necessary for, for example, assisting users in operating construction machinery or for the automated operation of construction machinery in underground space. This allows information processing systems to further promote labor saving and automation in excavation sites where communication with the outside world is difficult.

[0039] Furthermore, this invention is characterized by an information processing system comprising: an imaging unit that captures images of at least one target, which is placed in a stationary object in a target space and whose position in a desired predetermined coordinate system is known, and acquires imaging information; a three-dimensional measurement unit whose position in the predetermined coordinate system is known and which three-dimensionally measures a measurement target located in the target space and whose position in the predetermined coordinate system is unknown, and acquires point cloud data in the measurement coordinate system; a target identification means that identifies the target in the imaging information; and a coordinate transformation means that converts the point cloud data acquired by the three-dimensional measurement unit into the predetermined coordinate system based on position coordinate information indicating the position of at least one identified target in the predetermined coordinate system and position coordinate information indicating the position of the three-dimensional measurement unit in the predetermined coordinate system.

[0040] The aforementioned predetermined coordinate system refers to an absolute coordinate system (also called a reference coordinate system) with the Earth's center of mass as its origin, or an arbitrary coordinate system with any predetermined location as its origin. The above-mentioned target space refers to, for example, a space formed by excavating underground, the interior of a building, or a predetermined area outdoors. The above-mentioned targets refer to objects that can be identified based on imaging information, such as AR markers, two-dimensional barcodes or predetermined patterns attached to inanimate objects in the target space, or parts of three-dimensional structures.

[0041] The above imaging information refers to video or still images captured of the target space. The objects of measurement mentioned above refer to the walls, floors, people, machines, and devices that make up the target space. The above-mentioned measurement coordinate system refers to a coordinate system whose position in a predetermined coordinate system is unknown, for example, a coordinate system that uses the position of the three-dimensional measurement unit as the reference point.

[0042] When there is one target that can be imaged by the imaging unit, the three-dimensional measurement unit is positioned so that its height in a predetermined coordinate system is specified and it is horizontal. When there are two or more targets that can be imaged by the imaging unit, the height in a predetermined coordinate system does not need to be specified, and it does not need to be positioned horizontally.

[0043] According to this invention, point cloud data of a measurement coordinate system can be converted to a predetermined coordinate system based on the position coordinate information of a target in a predetermined coordinate system and the position coordinate information of a three-dimensional measurement unit. Therefore, point cloud data of a desired predetermined coordinate system can be obtained without using, for example, a global navigation satellite system.

[0044] Furthermore, as long as the three-dimensional measurement unit is not moved, the relative position between the target and the three-dimensional measurement unit does not change. Therefore, the information processing system can stably and repeatedly acquire point cloud data in a predetermined coordinate system that corresponds to the state of the target space, even if the state of the target space changes over time, such as in tunnel excavation work.

[0045] Even if the three-dimensional measurement unit is moved, the information processing system can easily acquire the positional coordinate information of the three-dimensional measurement unit after the move by moving the unit to a location where the positional coordinate information of a predetermined coordinate system is known, such as a target in the target space. Therefore, even when the three-dimensional measurement unit is moved, the information processing system can easily acquire point cloud data of the new predetermined coordinate system.

[0046] As a result, the information processing system can sequentially acquire point cloud data of a desired predetermined coordinate system, even in environments where communication with the outside is difficult, environments where the conditions within the target space are constantly changing, or environments where the three-dimensional measurement unit needs to be moved. Therefore, the information processing system can acquire point cloud data in a predetermined coordinate system that can be used for various assistive technologies and autonomous driving technologies, regardless of the state of communication with the outside world. This enables labor saving and automation even in environments where communication is difficult. [Effects of the Invention]

[0047] The present invention provides an information processing system and an information processing method that can promote labor saving and automation even in environments where communication is difficult. [Brief explanation of the drawing]

[0048] [Figure 1] A schematic diagram illustrating the drilling support system. [Figure 2] A diagram showing the configuration of the drilling support system. [Figure 3] A block diagram showing the internal configuration of the drilling support system. [Figure 4] An explanatory diagram illustrating the outline of the first machine learning model. [Figure 5] An explanatory diagram illustrating the outline of the second machine learning model. [Figure 6] A sequence diagram showing the flow of processing operations in the drilling support system. [Figure 7] A flowchart illustrating the calibration process in a measuring device. [Figure 8] A schematic diagram illustrating an example of a still image. [Figure 9] A schematic diagram illustrating an example of point cloud data. [Figure 10] An explanatory diagram illustrating the process of adding absolute coordinate information to point cloud data. [Figure 11] A flowchart illustrating the flow of monitoring processes in a measuring device. [Figure 12] A flowchart illustrating the flow of support processing on the management terminal. [Figure 13] An explanatory diagram illustrating the state in which various types of information are superimposed on point cloud data. [Figure 14] A flowchart showing the calibration process in Example 2. [Modes for carrying out the invention]

[0049] One embodiment of this invention will be described below with reference to the drawings. [Examples]

[0050] This embodiment is an excavation support system 1 that assists in the excavation of tunnel T based on absolute coordinate information of an absolute coordinate system, in the interior of tunnel T where position measurement using a global navigation satellite system is difficult. Such an excavation support system 1 and an excavation support method using the excavation support system 1 will be explained with reference to Figures 1 to 5.

[0051] Figure 1 shows a schematic diagram illustrating the drilling support system 1, Figure 2 shows a configuration diagram of the drilling support system 1, Figure 3 shows a block diagram of the internal configuration of the drilling support system 1, Figure 4 shows an explanatory diagram illustrating the outline of the first machine learning model 17, and Figure 5 shows an explanatory diagram illustrating the outline of the second machine learning model 18.

[0052] Furthermore, in Figure 1, arrow L represents the tunnel axis direction (hereinafter referred to as tunnel axis direction L), arrow Li in the figure indicates the excavation direction of tunnel T (hereinafter referred to as excavation direction Li), and arrow Lo indicates the direction of the tunnel entrance opposite to the excavation direction Li (hereinafter referred to as tunnel entrance direction Lo).

[0053] First, as shown in Figure 1, the excavation support system 1 includes three or more markers 2 placed inside the tunnel T, a portable measuring device 10 that measures the inside of the tunnel T and processes various information, and a management terminal 20 and a mobile terminal 30 (see Figure 2) that function as display means for displaying output data from the measuring device 10 to the user.

[0054] Furthermore, the excavation support system 1 is configured such that the measuring device 10, the management terminal 20, and the mobile terminal 30 are interconnected and able to communicate with each other via a communication line 3 such as a wireless LAN, and are also connected to construction machinery inside the tunnel T via the communication line 3.

[0055] Such an excavation support system 1 is configured to output point cloud data 5 in an absolute coordinate system suitable for known construction machinery that supports machine guidance and machine control. In this embodiment, as shown in Figures 1 and 2, a hydraulic excavator 40 that supports machine guidance using point cloud data in an absolute coordinate system will be used as an example of a construction machine for explanation.

[0056] Although a detailed explanation is omitted, the hydraulic excavator 40, as shown in Figure 3, includes a display unit 41 that displays various information to support operator operation and also functions as a display means to display output data from the measuring device 10 to the user, a line connection unit 42 that connects to the communication line 3, a speaker that outputs sound, and a control unit (not shown) that controls the operation of these components.

[0057] More specifically, as shown in Figure 1, the multiple markers 2 in the excavation support system 1 are attached to both sides of the tunnel T, which is a stationary object inside the tunnel T, at predetermined intervals in the tunnel axis direction L. These multiple markers 2 are targets that function as positional markers readable by the measuring device 10, and are composed of AR markers, for example, each marker 2 is provided with a design that functions as a unique marker identifier.

[0058] Furthermore, the measuring device 10 is small enough for a user to carry and has the function of imaging and three-dimensionally measuring the inside of the tunnel T as the subject, the function of performing various information processing based on the still images 4A and point cloud data 5 described later, and the function of transmitting various information via the communication line 3.

[0059] The measuring device 10 is a device that requires calibration each time its position inside the tunnel T changes. As shown in Figure 1, for example, it is arranged so that three or more markers 2 are included as subjects on the Lo side in the direction of the tunnel entrance in the tunnel T, and imaging and three-dimensional measurement of the inside of the tunnel T are performed at a fixed point.

[0060] As shown in Figure 3, the measuring device 10 consists of an imaging unit 11 that captures images of the subject, a three-dimensional measuring unit 12 that performs three-dimensional measurement on objects inside the tunnel T, a line connection unit 13 that connects to the communication line 3, a storage unit 14 that stores various information, and a measurement control unit 15 that controls the operation of these units.

[0061] Specifically, the imaging unit 11 is composed of a camera or the like and has the function of capturing a still image 4A (see Figure 8) of the subject based on a control signal from the measurement control unit 15, and the function of outputting the captured still image of the subject to the measurement control unit 15.

[0062] Furthermore, the three-dimensional measurement unit 12 is composed of, for example, a 3D-LiDAR capable of three-dimensional measurement within a range of approximately 360 degrees centered on itself, and as shown in Figure 2, it is integrally provided in close proximity to the imaging unit 11 so that the inside of the tunnel T can be measured in three dimensions at approximately the same position as the imaging unit 11.

[0063] This three-dimensional measurement unit 12 has the function of performing three-dimensional measurements of the inside of the tunnel T based on a control signal from the measurement control unit 15 and acquiring point cloud data 5 (see Figure 9), and the function of outputting the acquired point cloud data 5 to the measurement control unit 15. Furthermore, the point cloud data 5 acquired by the three-dimensional measurement unit 12 has attribute information such as three-dimensional coordinate information associated with each measurement point, which represents the object being measured.

[0064] Furthermore, the line connection unit 13 is composed of, for example, a wireless LAN module and has the function of connecting to the communication line 3 and the function of sending and receiving various information via the communication line 3. Furthermore, the storage unit 14 is composed of, for example, a hard disk or non-volatile memory, and has the function of writing and storing various types of information, and the function of reading and storing various types of information.

[0065] As shown in Figure 3, this memory unit 14 stores a processing program (not shown) for performing various processes based on the still image 4A and point cloud data 5, marker information 16 in which various information about the marker 2 is registered, and a first machine learning model 17 and a second machine learning model 18 for identifying objects inside the tunnel T.

[0066] More specifically, the marker information 16 is registered by associating the design of marker 2, which functions as a unique marker identifier for each marker 2, with coordinate information indicating the location where marker 2 is attached. The coordinate information for marker 2 is absolute coordinate information in an absolute coordinate system with the Earth's center of gravity as the origin. The absolute coordinate system position information obtained by surveying the corner 2a of marker 2 (see Figure 10) using a known surveying method is registered.

[0067] The first machine learning model 17 and the second machine learning model 18 are image recognition algorithms using convolutional neural networks, as shown in Figures 4 and 5, respectively, consisting of input layers 17a and 18a into which input information is input, hidden layers 17b and 18b, and output layers 17c and 18c into which output information is output.

[0068] Of these, the first machine learning model 17, as shown in Figure 4, is constructed to analyze the still image 4A captured by the imaging unit 11 to identify each object (still objects and moving objects) in the still image 4A, and to assign a name identifier indicating the name of each identified object to the identified object.

[0069] Specifically, the first machine learning model 17 is constructed to take a still image 4A as input information, detect each object in the still image 4A, such as the inner surface Ta of the tunnel T including the face Tb, people, and construction machinery, and obtain an identified image 4B as output information for the input information, which associates a name identifier indicating the name of each identified object.

[0070] This first machine learning model 17 is constructed by repeatedly training multiple training images, which are still images different from the input still image 4A, using them as training data V1 for the input information. In this process, the first machine learning model 17 learns by repeatedly adjusting its weights and judgment thresholds based on the training data V1 so that the output information for the input information becomes the identified image 4B.

[0071] The training images, which constitute training data V1, are images to which name identifiers indicating the names of objects have been pre-assigned, and include images showing the characteristics of the ground, images showing the characteristics of humans, and images showing the characteristics of construction machinery.

[0072] On the other hand, as shown in Figure 5, the second machine learning model 18 is constructed to analyze the identification image 4B output by the first machine learning model 17 to identify the movable parts of the construction machinery and to assign a name identifier indicating the name of the identified movable part to the identified part.

[0073] Specifically, the second machine learning model 18 takes the identification image 4B as input information, detects the pivot points and ends of the movable parts of the construction machinery within the identification image 4B, and is constructed to obtain an identification image 4C as output information for the input information, which associates a name identifier indicating the name of the identified movable part with the identification image 4C.

[0074] For example, if the construction machine is a hydraulic excavator 40, the second machine learning model 18 detects the connection point 43 between the upper slewing body and the boom (see Figure 13), the connection point 44 between the boom and the arm (see Figure 13), the connection point 45 between the arm and the bucket 46 (see Figure 13), and the bucket 46 as movable parts in the hydraulic excavator 40 in the identification image 4B, and outputs an identification image 4C associated with a name identifier as output information.

[0075] This second machine learning model 18 is constructed by repeatedly training multiple training images, which are static images separate from the input image 4B, using them as training data V2 for the input information. In this process, the second machine learning model 18 learns by repeatedly adjusting its weights and judgment thresholds based on the training data V2 so that the output information for the input information becomes the identified image 4C.

[0076] The training images, which are the training data V2, are images to which name identifiers indicating the names of the movable parts of construction machinery have been pre-assigned. For example, in the case of a hydraulic excavator 40, these may include images showing the features of the upper rotating body, images showing the features of the boom and arm, and images showing the features of the bucket 46.

[0077] Furthermore, the measurement control unit 15 consists of hardware such as a CPU and memory, and software such as a control program. This measurement control unit 15 has processing functions for exchanging various signals with the imaging unit 11, the three-dimensional measurement unit 12, the line connection unit 13, and the storage unit 14, a calibration function for adjusting so that the desired point cloud data 5 can be obtained, a function for controlling the operation of each unit connected via a predetermined bus, and a processing function for exchanging various information via the communication line 3.

[0078] More specifically, the measurement control unit 15 has the function of a target identification means for identifying a target marker 2 from a still image 4A of a subject captured by the imaging unit 11, and the function of a position coordinate assignment means for assigning absolute coordinate system position coordinate information to the point cloud data 5 of the measurement coordinate system based on position coordinate information indicating the position of the identified marker 2 in the absolute coordinate system.

[0079] Furthermore, the measurement control unit 15 has the function of a coordinate transformation means that converts the point cloud data 5 acquired by the three-dimensional measurement unit 12 into an absolute coordinate system based on the assigned absolute coordinate system position coordinate information, and the function of a data output means that outputs various information associated with the point cloud data 5 in the absolute coordinate system.

[0080] Furthermore, the management terminal 20 of the excavation support system 1 is a terminal used by workers or construction managers at an excavation site, for example, and functions as a request means for requesting the transmission of output data to the measuring device 10, and a display means for displaying the output data from the measuring device 10 to the user.

[0081] As shown in Figure 3, this management terminal 20 consists of a line connection unit 21 that connects to the communication line 3, a display unit 22 that displays various information, an operation reception unit 23 that accepts various operations from the user, a storage unit 24 that stores various information, a speaker (not shown) that outputs sound, and a terminal control unit 25 that controls the operation of these.

[0082] Specifically, the line connection unit 21 is composed of, for example, a wireless LAN module, and has the function of connecting to the communication line 3 and the function of sending and receiving various information via the communication line 3. Furthermore, the display unit 22 is composed of a liquid crystal display or the like, and has the function of displaying various information based on control signals from the terminal control unit 25.

[0083] Furthermore, the operation reception unit 23 is composed of, for example, a keyboard (symbol omitted) or a mouse (symbol omitted), and has the function of receiving input operations from the user and the function of outputting information indicating the received input content to the terminal control unit 25.

[0084] Furthermore, the storage unit 24 is composed of, for example, a hard disk or non-volatile memory, and has the function of writing and storing various types of information, and the function of reading and storing various types of information. As shown in Figure 3, this memory unit 24 stores design data 26 that is superimposed on the point cloud data 5 acquired from the measuring device 10.

[0085] More specifically, the design data 26 consists of data showing the design cross-sectional shape of tunnel T, to which absolute coordinate system position coordinate information is associated as attribute information; three-dimensional model data showing the design shape of tunnel T; or data showing virtual stakeouts.

[0086] Furthermore, the terminal control unit 25 consists of hardware such as a CPU and memory, and software such as a control program. This terminal control unit 25 has processing functions related to the exchange of various signals with the line connection unit 21, display unit 22, operation reception unit 23, storage unit 24 and speaker, a function to control the operation of each unit connected via a predetermined bus, and a processing function related to the exchange of various information via the communication line 3.

[0087] Furthermore, the portable terminal 30 of the excavation support system 1 is a tablet terminal used by workers and construction managers at the excavation site, and functions as a display means that displays output data from the measuring device 10 to the user via the management terminal 20.

[0088] Although a detailed explanation is omitted, this mobile terminal 30 consists of an operation display unit 31 that displays various information and accepts various operations from the user, a line connection unit 32 that connects to the communication line 3, a storage unit (not shown) that stores various information, a speaker (not shown) that outputs sound, and a control unit (not shown) that controls the operation of these components.

[0089] Next, the processing flow of the excavation support system 1 with the above configuration will be explained using Figures 6 to 13. Figure 6 shows a sequence diagram of the processing flow in the drilling support system 1, Figure 7 shows a flowchart of the calibration processing flow in the measuring device 10, Figure 8 shows a schematic diagram illustrating an example of still image 4A, and Figure 9 shows a schematic diagram illustrating an example of point cloud data 5.

[0090] Furthermore, Figure 10 shows an explanatory diagram illustrating the process of adding absolute coordinate information to the point cloud data 5, Figure 11 shows a flowchart of the monitoring process in the measurement device 10, Figure 12 shows a flowchart of the support process in the management terminal 20, and Figure 13 shows an explanatory diagram illustrating the state in which various information is superimposed on the point cloud data 5.

[0091] First, with the power to the measuring device 10 turned on, when a user operates the management terminal 20 to execute a predetermined program, the terminal control unit 25 of the management terminal 20 displays a menu screen (not shown) on the display unit 22 that provides various instructions to the user.

[0092] At this point, when the user presses the button to start the calibration of the measuring device 10 in accordance with the instructions on the menu screen, the terminal control unit 25 transmits a calibration signal to start the calibration of the measuring device 10 to the measuring device 10 via the communication line 3, as shown in Figure 6 (step S101). The measurement control unit 15 of the measurement device 10, which has acquired the calibration signal, starts a calibration process to calibrate the device so that point cloud data 5 in absolute coordinate system is output, as shown in Figure 6 (step S102).

[0093] Specifically, when calibration processing is started, the measurement control unit 15 of the measurement device 10 performs, in parallel processing, an imaging process (step S111) in which the imaging unit 11 images the inside of the tunnel T, and a three-dimensional measurement process (step S112) in which the three-dimensional measurement unit 12 measures the inside of the tunnel T, as shown in Figure 7.

[0094] In this process, the imaging unit 11 captures a still image 4A of the inside of the tunnel T based on a control signal from the measurement control unit 15, and outputs the captured still image 4A to the measurement control unit 15 as imaging information.

[0095] This still image 4A captures, for example, the inner surface Ta and face Tb of the tunnel T, the uncut ground Tc at the bottom of the tunnel T, a hydraulic excavator 40 as construction machinery, multiple workers M, and three or more markers 2 attached to the side of the tunnel T.

[0096] Meanwhile, the three-dimensional measurement unit 12, based on the control signal from the measurement control unit 15, performs three-dimensional measurement of objects in the tunnel T at approximately the same timing as imaging by the imaging unit 11, generates point cloud data 5, and then outputs the generated point cloud data 5 to the measurement control unit 15.

[0097] As shown in Figure 9, for example, this point cloud data 5 consists of a point cloud (not shown) indicating the inner surface Ta of the tunnel T containing three or more markers 2, a point cloud indicating the tunnel face Tb, a point cloud indicating the ground Tc, a point cloud indicating the hydraulic excavator 40, and a point cloud indicating the worker M, all within approximately the same field of view as the still image 4A.

[0098] Furthermore, the point cloud data 5 has associated attribute information, specifically the measurement coordinate information of the measurement coordinate system with the three-dimensional measurement unit 12 as the origin. In other words, the point cloud data 5 output by the three-dimensional measurement unit 12 is data whose position in the absolute coordinate system is unknown.

[0099] After performing the imaging process in step S111 and the three-dimensional measurement process in step S112, the measurement control unit 15 identifies the patterns of all markers 2 in the still image 4A based on the color and shading in the still image 4A, as shown in Figure 7 (step S113). Subsequently, the measurement control unit 15 determines whether the number of markers 2 that were able to identify the pattern is three or more (step S114).

[0100] If the number of markers 2 whose patterns were identified is less than three (step S114: No), the measurement control unit 15 cannot perform the subsequent processing, so it sends an error notification indicating an identification error of marker 2 to the management terminal 20 (step S115), and then terminates the calibration process. In this case, the terminal control unit 25 of the management terminal 20, which has received the error notification, displays an error screen (not shown) on the display unit 22 and requests the user to reposition the measuring device 10, etc.

[0101] On the other hand, if the number of markers 2 whose patterns can be identified is three or more (step S114: Yes), the measurement control unit 15 identifies the corresponding measurement points of the point cloud data 5 corresponding to the markers 2 based on the markers 2 whose patterns can be identified, as shown in Figure 7, and assigns absolute coordinate system position coordinate information to the identified corresponding measurement points (step S116).

[0102] Specifically, as shown in Figure 10, the measurement control unit 15 reads marker information 16 from the storage unit 14 and obtains and temporarily stores the coordinate information of the corner 2a of the marker 2 corresponding to the identified pattern. At this time, since the coordinate information of the marker information 16 is in an absolute coordinate system, the coordinate information of the corner 2a of the marker 2 that is temporarily stored is the position coordinate information in an absolute coordinate system.

[0103] Furthermore, as shown in Figure 10, the measurement control unit 15 identifies the measurement points of the point cloud data 5 that overlap with the corner 2a of the marker 2 in the still image 4A as corresponding measurement points 5a, and assigns the temporarily stored position coordinate information of the corner 2a of the marker 2 to the identified corresponding measurement points 5a of the point cloud data 5.

[0104] Therefore, the corresponding measurement point 5a of the point cloud data 5 that overlaps with the corner 2a of marker 2 will be assigned absolute coordinate information in addition to the position coordinate information of the measurement coordinate system with the three-dimensional measurement unit 12 as the origin.

[0105] Then, for each marker 2 whose pattern was identified in step S113, the measurement control unit 15 identifies the corresponding measurement point 5a of the point cloud data 5 that overlaps with the corner 2a of the marker 2 in the still image 4A and assigns absolute coordinate information to it.

[0106] Subsequently, as shown in Figure 7, the measurement control unit 15 uses the position coordinate information of three or more corresponding measurement points 5a acquired in step S116 to calculate and estimate absolute coordinate information of an absolute coordinate system including coordinate axes, and position coordinate information indicating the position of the three-dimensional measurement unit 12 in the absolute coordinate system (step S117).

[0107] Specifically, the measurement control unit 15 calculates absolute coordinate information of the absolute coordinate system, including coordinate axes, and position coordinate information of the three-dimensional measurement unit 12 in the absolute coordinate system, based on the relative positions of the position coordinate information (origin) of the three-dimensional measurement unit 12 in the measurement coordinate system, the position coordinate information of three or more corresponding measurement points 5a, and the position coordinate information of three or more corresponding measurement points 5a in the absolute coordinate system acquired in step S116.

[0108] Subsequently, the measurement control unit 15 calculates and stores calibration values ​​for converting the point cloud data 5 acquired from the three-dimensional measurement unit 12 into an absolute coordinate system, based on the absolute coordinate information of the absolute coordinate system calculated in step S117 and the position coordinate information of the three-dimensional measurement unit 12 (step S118).

[0109] In this process, the measurement control unit 15 converts the point cloud data 5, whose position in the absolute coordinate system is unknown and was acquired in step S112, into point cloud data 5 in the absolute coordinate system, and then returns to step S102 in Figure 6 to complete the calibration process.

[0110] Next, we will explain the processing operation of the measuring device 10 when the calibration of the measuring device 10 is completed and the user presses the button to start monitoring the inside of tunnel T, following the instructions on the menu screen of the management terminal 20. When the button to start monitoring tunnel T is pressed, the terminal control unit 25 transmits a start signal to the measuring device 10 via the communication line 3 to start monitoring inside tunnel T, as shown in Figure 6 (step S103).

[0111] As shown in Figure 6, the measurement control unit 15 of the measurement device 10, having received a start signal from the management terminal 20, starts a monitoring process that monitors the inside of the tunnel T and outputs output data indicating the monitoring results to the management terminal 20 (step S104).

[0112] Specifically, as shown in Figure 11, the measurement control unit 15, which has started the monitoring process, performs in parallel the imaging process (step S121) in which the imaging unit 11 images the inside of the tunnel T, and the three-dimensional measurement process (step S122) in which the three-dimensional measurement unit 12 measures the inside of the tunnel T.

[0113] In this process, the imaging unit 11 captures a still image 4A of the inside of the tunnel T based on a control signal from the measurement control unit 15, and outputs the captured still image 4A to the measurement control unit 15 as imaging information.

[0114] Meanwhile, the three-dimensional measurement unit 12, based on the control signal from the measurement control unit 15, measures objects in the tunnel T in three dimensions at approximately the same timing as imaging by the imaging unit 11, generates point cloud data 5 whose position in the absolute coordinate system is unknown, and then outputs the generated point cloud data 5 to the measurement control unit 15.

[0115] When point cloud data 5 is acquired from the three-dimensional measurement unit 12, the measurement control unit 15 converts the acquired point cloud data 5 into absolute coordinate system point cloud data 5 based on the calibration value calculated in the calibration process, as shown in Figure 11 (step S123).

[0116] The measurement control unit 15 repeatedly performs the imaging process by the imaging unit 11 and the three-dimensional measurement process by the three-dimensional measurement unit 12 at predetermined time intervals to acquire a video based on still images 4A and a video based on point cloud data 5, with the inside of the tunnel T as the subject.

[0117] Subsequently, as shown in Figure 11, the measurement control unit 15 starts object recognition processing to identify still objects and moving objects in the still image 4A using the first machine learning model 17 read from the storage unit 14 (step S124).

[0118] Specifically, as shown in Figure 4, the measurement control unit 15 inputs the still image 4A captured by the imaging unit 11 as input information to the input layer 17a of the first machine learning model 17 read from the storage unit 14. In this process, the first machine learning model 17 passes input information from the input layer 17a to the hidden layer 17b, and from the hidden layer 17b to the output layer 17c, while performing comparisons and judgments with pre-trained weights and thresholds to identify objects in the still image 4A. Furthermore, the first machine learning model 17 outputs an identified image 4B as output information, in which each object in the still image 4A is assigned a name identifier.

[0119] For example, the first machine learning model 17 identifies an object in a still image 4A that approximates information indicating human characteristics as a human, and outputs an identified image 4B in which a name identifier indicating "human" is assigned to the identified object.

[0120] Once the first machine learning model 17 has completed the identification of each object in the still image 4A, the measurement control unit 15 acquires position coordinate information indicating the position of each identified object in the absolute coordinate system based on the identified image 4B and the point cloud data 5 in the absolute coordinate system, as shown in Figure 11 (step S125).

[0121] Specifically, the measurement control unit 15 identifies multiple measurement points in the absolute coordinate system point cloud data 5 that overlap with each object in the identification image 4B as point clouds corresponding to the objects in the identification image 4B.

[0122] Furthermore, the measurement control unit 15 acquires the position coordinate information and size of the identified point cloud in the absolute coordinate system, and temporarily stores the object name identifier assigned to the identification image 4B as object information associated with the identified point cloud.

[0123] Subsequently, as shown in Figure 11, the measurement control unit 15 starts a construction machinery identification process to identify the movable parts of the construction machinery in the identification image 4B using the second machine learning model 18 read from the storage unit 14 (step S126).

[0124] Specifically, the measurement control unit 15 inputs the identification image 4B output by the first machine learning model 17 as input information to the input layer 18a of the second machine learning model 18, which is read from the storage unit 14. In this process, the second machine learning model 18 passes input information from the input layer 18a to the hidden layer 18b, and from the hidden layer 18b to the output layer 18c, while performing comparison and judgment with pre-learned weights and thresholds to identify the movable parts of the hydraulic excavator 40 in the identification image 4B.

[0125] More specifically, the second machine learning model 18 identifies the connection point 43 between the upper slewing body and the boom, the connection point 44 between the boom and the arm, the connection point 45 between the arm and the bucket 46, and the bucket 46 from the object in the identification image 4B which is assigned a name identifier indicating the hydraulic excavator 40. Furthermore, the second machine learning model 18 outputs an identified image 4C as output information, in which a name identifier is associated with the movable part in the identified image 4B.

[0126] Once the second machine learning model 18 has completed the identification of the movable parts of the hydraulic excavator 40, the measurement control unit 15 acquires position coordinate information indicating the position of the movable parts of the hydraulic excavator 40 in the absolute coordinate system based on the identification image 4C and the point cloud data 5 in the absolute coordinate system, as shown in Figure 11 (step S127).

[0127] Specifically, the measurement control unit 15 acquires position coordinate information of the connection point 43 between the upper slewing body and the boom, the connection point 44 between the boom and the arm, the connection point 45 between the arm and the bucket 46, and the position coordinate information of the bucket 46, as well as the size of the point cloud representing the bucket 46, based on the point cloud data 5 that overlaps with the movable parts of the hydraulic excavator 40 in the identification image 4C, in an absolute coordinate system, and temporarily stores this information as construction equipment information, associating it with a name identifier.

[0128] When the position coordinate information of the movable part of the hydraulic excavator 40 is acquired, the measurement control unit 15 generates output data that associates the still image 4A acquired in step S121, the absolute coordinate system point cloud data 5 acquired in step S123, the object information acquired in step S125, and the construction machine information acquired in step S127, as shown in Figure 11, and then transmits it to the management terminal 20 via the communication line 3 (step S128).

[0129] Subsequently, the measurement control unit 15 repeats the process from step S121 to step S128 until it receives a stop signal from the management terminal 20 to stop imaging by the imaging unit 11 and three-dimensional measurement by the three-dimensional measurement unit 12, thereby repeatedly acquiring point cloud data 5 in an absolute coordinate system without using, for example, the Global Navigation Satellite System.

[0130] Next, we will explain the processing operation of the management terminal 20 that sent the start signal in step S103 of Figure 6, based on the user's operation. As shown in Figure 6, the terminal control unit 25 of the management terminal 20 that transmitted the start signal starts support processing to assist the user based on the output data acquired from the measuring device 10 (step S105).

[0131] Specifically, when support processing is started, the terminal control unit 25 of the management terminal 20 determines whether or not output data has been acquired from the measuring device 10 (step S131), as shown in Figure 12. If output data has not been acquired from the measuring device 10 (step S131: No), the system waits for the output data to be acquired.

[0132] On the other hand, when output data is acquired from the measuring device 10 (step S131: Yes), the terminal control unit 25 overlays various information onto a three-dimensional model of the inside of the tunnel T generated based on the point cloud data 5 of the output data and displays it on the display unit 22 (step S132).

[0133] Specifically, as shown in Figure 13, the terminal control unit 25 generates a motion frame 6 that surrounds the point cloud representing the moving worker M, based on object information acquired from the output data, and displays the generated motion frame 6 superimposed on the three-dimensional model generated from the point cloud data 5 as various information.

[0134] Furthermore, as shown in Figure 13, the terminal control unit 25 reads the design data 26 from the storage unit 24, and then displays the cross-sectional shape line 26a, which is data indicating the design cross-sectional shape of the tunnel T registered in the design data 26, as various information on the display unit 22, superimposed on the three-dimensional model generated from the point cloud data 5.

[0135] In addition, the terminal control unit 25 generates a motion frame 7 surrounding the point cloud representing the moving bucket 46 based on the construction machine information acquired from the output data, and displays the generated motion frame 7 as various pieces of information superimposed on the three-dimensional model generated from the point cloud data 5.

[0136] Furthermore, the terminal control unit 25, based on the construction equipment information acquired from the output data, displays the connection points 43 between the upper slewing body and the boom, the connection points 44 between the boom and the arm, the connection points 45 between the arm and the bucket 46, and the skeletal line 8 connecting the bucket 46 in this order as various pieces of information on the display unit 22, superimposed on the three-dimensional model generated from the point cloud data 5.

[0137] When various information is superimposed on the three-dimensional model generated from the point cloud data 5 and displayed on the display unit 22, the terminal control unit 25 generates superimposed data that associates the point cloud data 5, which is the information to be superimposed on the display unit 22, with the various information mentioned above, as shown in Figure 12, and then transmits it to the hydraulic excavator 40 and the mobile terminal 30 via the communication line 3 (step S133). The processing operations of the hydraulic excavator 40 and the mobile terminal 30 that acquired the superimposed data will be described in detail later.

[0138] When the superimposed data is transmitted, the terminal control unit 25 of the management terminal 20 determines, based on the point cloud data 5 in absolute coordinate system, whether or not it has detected human intrusion into the restricted area centered on the hydraulic excavator 40 (step S134), as shown in Figure 12.

[0139] Specifically, the terminal control unit 25 determines whether the straight-line distance from the point cloud position coordinate information representing the hydraulic excavator 40 to the point cloud position coordinate information representing the human, in the absolute coordinate system, is less than or equal to a predetermined distance. The predetermined distance is, for example, the slewing radius of the bucket 46 of the hydraulic excavator 40.

[0140] If the terminal control unit 25 detects human intrusion into the restricted area (step S134: Yes), it starts an intrusion notification process to notify the system of human intrusion into the restricted area, as shown in Figure 12 (step S135).

[0141] Specifically, the terminal control unit 25 displays a warning message, which is a string of characters indicating that a person has entered the restricted area, on the display unit 22, and also outputs an audio message notifying the management terminal 20 of the person's entry into the restricted area via the management terminal 20's speaker.

[0142] Furthermore, the terminal control unit 25 transmits intrusion notification information to the hydraulic excavator 40 and the mobile terminal 30 via the communication line 3, respectively, to notify them of a person's intrusion into the restricted area. The processing operations of the hydraulic excavator 40 and the mobile terminal 30 that received the intrusion notification information will be described in detail later.

[0143] On the other hand, if no human intrusion into the restricted area is detected in step S134 (step S134: No), or if intrusion notification information is sent in step S135, the terminal control unit 25 determines, based on the point cloud data 5 in absolute coordinate system, whether a part of the hydraulic excavator 40 has exceeded the height limit set inside the tunnel T (step S136).

[0144] Specifically, the terminal control unit 25 determines whether the straight-line distance from the point cloud position coordinate information indicating the connection point 44 between the boom and the arm or the bucket 46, to the point cloud position coordinate information indicating the inner surface Ta of the tunnel T, which is a stationary object, is less than or equal to a predetermined distance in the height direction of the absolute coordinate system.

[0145] If a portion of the hydraulic excavator 40 exceeds the height limit (step S136: Yes), the terminal control unit 25 starts construction equipment notification processing to notify that a portion of the hydraulic excavator 40 has moved outside the height limit area, as shown in Figure 12 (step S137).

[0146] Specifically, the terminal control unit 25 displays a warning message in text format on the display unit 22 that a portion of the hydraulic excavator 40 has exceeded the height limit, and also outputs an audio message via the speaker of the management terminal 20 notifying that a portion of the hydraulic excavator 40 has exceeded the height limit.

[0147] Furthermore, the terminal control unit 25 transmits construction equipment notification information to the hydraulic excavator 40 via the communication line 3, informing the hydraulic excavator 40 that a portion of it has exceeded the height limit. The processing actions of the hydraulic excavator 40 after receiving the construction equipment notification information will be described in detail later.

[0148] On the other hand, if part of the hydraulic excavator 40 does not exceed the height limit (step S136: No), and if construction equipment notification information is sent in step S137, the terminal control unit 25 returns to step S131 and repeats the process from step S131 to step S137 (described later) until it accepts a stop operation from the user.

[0149] Then, when the user presses the button to end monitoring of tunnel T in accordance with the instructions on the menu screen of the management terminal 20, the terminal control unit 25 sends stop information to the measuring device 10 via the communication line 3 to end monitoring inside tunnel T, as shown in Figure 6 (step S108), and then terminates the support process. At this point, the measurement control unit 15 of the measurement device 10 stops the imaging by the imaging unit 11 and the three-dimensional measurement by the three-dimensional measurement unit 12, as described above, and terminates the monitoring process.

[0150] Here, we will explain the processing operation when the hydraulic excavator 40 receives the superimposed data transmitted in step S133 of Figure 12, the intrusion notification information transmitted in step S135, or the construction equipment notification information transmitted in step S137.

[0151] When superimposed data is acquired from the management terminal 20, the control unit of the hydraulic excavator 40 displays the superimposed data acquired via the communication line 3 on the display unit 41 as terminal processing, as shown in Figure 6 (step S106).

[0152] In this case, as shown in Figure 13, the display unit 41 of the hydraulic excavator 40 displays a three-dimensional model of the inside of the tunnel T, generated based on point cloud data 5 in an absolute coordinate system, with the cross-sectional shape lines 26a indicating the design cross-sectional shape of the tunnel T superimposed on it.

[0153] Therefore, the excavation support system 1 enables the operator of the hydraulic excavator 40 to proceed with excavating the ground Tc while comparing the current interior of the tunnel T with the design cross-sectional shape.

[0154] Furthermore, when intrusion notification information is received from the management terminal 20, the control unit of the hydraulic excavator 40, in the terminal processing of step S106 in Figure 6, displays a warning message on the display unit 41 and outputs an audio message via the speaker to notify the operator of human intrusion into the restricted area.

[0155] Alternatively, if construction equipment notification information is acquired, the control unit of the hydraulic excavator 40, in the terminal processing of step S106 in Figure 6, displays a warning message on the display unit 41 and outputs an audio message via the speaker to inform the operator that a part of the hydraulic excavator 40 has exceeded the height limit.

[0156] In this scenario, the operator of the hydraulic excavator 40 will operate the excavator 40 while confirming the safety of the surroundings in accordance with messages based on intrusion notification information or construction equipment notification information. Therefore, the excavation support system 1 enables the operator of the hydraulic excavator 40 to safely proceed with excavation of the ground Tc without the need to install sensors or other devices separate from the measuring device 10.

[0157] Next, we will explain the processing operation when the mobile terminal 30 acquires the superimposed data transmitted in step S133 of Figure 12 described above, or the intrusion notification information transmitted in step S135. When superimposed data is acquired from the management terminal 20, the control unit of the mobile terminal 30 displays the acquired superimposed data via the communication line 3 on the operation display unit 31 as terminal processing, as shown in Figure 6 (step S107).

[0158] In this case, as shown in Figure 13, the operation display unit 31 of the mobile terminal 30 displays a three-dimensional model of the inside of the tunnel T, generated based on the point cloud data 5 of the absolute coordinate system, with the cross-sectional shape line 26a indicating the design cross-sectional shape of the tunnel T superimposed on it.

[0159] Therefore, the excavation support system 1 enables construction managers and workers using the mobile terminal 30 to manage the progress of excavation while comparing the current interior of the tunnel T with the design cross-sectional shape.

[0160] Furthermore, when intrusion notification information is obtained from the management terminal 20, the control unit of the mobile terminal 30, in the terminal processing of step S107 in Figure 6, displays a warning message on the operation display unit 31 and outputs an audio message via the speaker to notify the user of the mobile terminal 30 of the intrusion into the restricted area.

[0161] In this case, users using the mobile terminal 30 can be informed that they have entered a restricted area by a message based on intrusion notification information. Therefore, the excavation support system 1 makes it possible to ensure safety at the tunnel T excavation site without installing sensors or other devices separate from the measuring device 10.

[0162] In this way, the drilling support system 1 and drilling support method enable the output of absolute coordinate system point cloud data 5 suitable for known construction machinery that supports machine guidance and machine control at tunnel T excavation sites where position measurement using the Global Navigation Satellite System is difficult, and also enable the safety of users by using the absolute coordinate system point cloud data 5.

[0163] As described above, the excavation support system 1 of this embodiment includes an imaging unit 11 that is positioned on the side of the tunnel T and captures images of at least three markers 2 whose positions in a desired absolute coordinate system are known, thereby acquiring a still image 4A.

[0164] Furthermore, the excavation support system 1 includes a three-dimensional measurement unit 12 that obtains point cloud data 5 of the measurement coordinate system by three-dimensionally measuring a measurement target located inside the tunnel T whose position in the absolute coordinate system is unknown. In addition, the drilling support system 1 is equipped with a target identification means (measurement control unit 15) that identifies the marker 2 in the still image 4A.

[0165] Furthermore, the drilling support system 1 includes a coordinate transformation means (measurement control unit 15) that converts the point cloud data 5 acquired by the three-dimensional measurement unit 12 into an absolute coordinate system based on position coordinate information indicating the position of the identified marker 2 in an absolute coordinate system.

[0166] Furthermore, the excavation support method of this embodiment includes an imaging step (step S111) in which the imaging unit 11 captures images of at least three markers 2, which are positioned on the side of the tunnel T and whose positions in a desired absolute coordinate system are known, in order to acquire a still image 4A.

[0167] Furthermore, the excavation support method includes a three-dimensional measurement process (step S112) in which the three-dimensional measurement unit 12 performs three-dimensional measurement of a measurement target located inside the tunnel T, whose position in the absolute coordinate system is unknown, and acquires point cloud data 5 of the measurement coordinate system. In addition, the excavation support method performs a target identification step (step S113) in which the target identification means identifies the marker 2 in the still image 4A.

[0168] The excavation support method involves a coordinate transformation process (step S118) in which the coordinate transformation means transforms the point cloud data 5 acquired by the three-dimensional measurement unit 12 into an absolute coordinate system, based on position coordinate information indicating the position of the identified marker 2 in an absolute coordinate system.

[0169] According to this configuration, point cloud data 5 in the measurement coordinate system is converted to the absolute coordinate system based on the position coordinate information of marker 2 in the absolute coordinate system. Therefore, it is possible to obtain the desired point cloud data 5 in the absolute coordinate system without using, for example, a global navigation satellite system.

[0170] Furthermore, as long as the three-dimensional measurement unit 12 is not moved, the relative position between the marker 2 and the three-dimensional measurement unit 12 does not change. Therefore, the excavation support system 1 and the information processing method can stably and repeatedly acquire point cloud data 5 in an absolute coordinate system that corresponds to the state of the tunnel T, even if the state inside the tunnel T changes over time.

[0171] Even if the three-dimensional measurement unit 12 is moved, the excavation support system 1 and the information processing method can easily reacquire the point cloud data 5 in the absolute coordinate system based on the position coordinate information of the marker 2.

[0172] As a result, the excavation support system 1 and the information processing method can sequentially acquire point cloud data 5 of the desired absolute coordinate system, even in environments where communication with the outside is difficult, where the conditions inside the tunnel T are constantly changing, or where the three-dimensional measurement unit 12 needs to be moved.

[0173] Therefore, the excavation support system 1 and the information processing method can acquire point cloud data 5 in an absolute coordinate system that can be used for various support technologies and autonomous driving technologies, regardless of the communication status with the outside, thus promoting labor saving and automation even in environments where communication is difficult.

[0174] Furthermore, the three-dimensional measurement unit 12 is configured to perform three-dimensional measurements inside the tunnel T, which contains at least three or more markers 2. The coordinate transformation means (measurement control unit 15) assigns position coordinate information of marker 2 to corresponding measurement points 5a that correspond to marker 2 identified by the target identification means, among the measurement points that make up the point cloud data 5, and transforms the point cloud data 5 acquired by the three-dimensional measurement unit 12 into an absolute coordinate system based on the position coordinate information assigned to the corresponding measurement points 5a.

[0175] With this configuration, position coordinate information for marker 2 is assigned to the corresponding measurement point 5a in the point cloud data 5 corresponding to marker 2 in the still image 4A. This makes it possible to obtain position coordinate information indicating the position of the corresponding measurement point 5a in a highly accurate absolute coordinate system.

[0176] Furthermore, since both the still image 4A and the point cloud data 5 contain three or more markers 2, the drilling support system 1 can accurately calculate position coordinate information indicating the position of the three-dimensional measurement unit 12 in the absolute coordinate system, as well as position coordinate information indicating the positions of measurement points other than the corresponding measurement points 5a in the absolute coordinate system, based on the position coordinate information of three or more absolute coordinate systems assigned to the point cloud data 5. As a result, the excavation support system 1 can accurately convert the point cloud data 5 acquired by the three-dimensional measurement unit 12 into the desired absolute coordinate system.

[0177] Furthermore, the coordinate transformation means (measurement control unit 15) calculates position coordinate information indicating the position of the three-dimensional measurement unit 12 in an absolute coordinate system based on the position coordinate information of the corresponding measurement point 5a, and is configured to transform the point cloud data 5 acquired by the three-dimensional measurement unit 12 into an absolute coordinate system based on the position coordinate information of the three-dimensional measurement unit 12.

[0178] With this configuration, the position coordinate information of the three-dimensional measurement unit 12 in the absolute coordinate system can be obtained. Therefore, once the position coordinate information of the three-dimensional measurement unit 12 is calculated, the position coordinate information in the absolute coordinate system can be easily associated with the measurement points of the point cloud data 5 acquired by the three-dimensional measurement unit 12 in subsequent processing. As a result, the excavation support system 1 can efficiently and continuously acquire point cloud data 5 in an absolute coordinate system, enabling it to acquire point cloud data 5 that is more suitable for various support technologies and automated driving technologies.

[0179] Furthermore, the drilling support system 1 has an imaging unit 11 and a three-dimensional measurement unit 12 integrated into one unit. With this configuration, the three-dimensional measurement unit 12 can perform three-dimensional measurements inside the tunnel T containing at least three markers 2 at approximately the same position as the imaging unit 11, thereby enabling the acquisition of point cloud data 5 that overlaps with the still image 4A.

[0180] Therefore, the drilling support system 1 can not only easily identify the corresponding measurement point 5a that corresponds to the marker 2 among the measurement points of the point cloud data 5, but can also accurately acquire the positional coordinate information of the corresponding measurement point 5a in the absolute coordinate system. As a result, the excavation support system 1 can improve the accuracy of calculating the position coordinate information of the three-dimensional measurement unit 12, and thus acquire highly accurate point cloud data 5 in an absolute coordinate system.

[0181] Furthermore, the excavation support system 1 is equipped with display means (management terminal 20, mobile terminal 30, and hydraulic excavator 40) that overlays the cross-sectional shape line 26a of the absolute coordinate system onto the point cloud data 5 of the absolute coordinate system.

[0182] With this configuration, the cross-sectional shape lines 26a are superimposed and displayed on the point cloud data 5 obtained by three-dimensional measurement of the inside of the tunnel T, making it easier, for example, to assist in the direct or remote operation of the hydraulic excavator 40 by the user. As a result, the excavation support system 1 can easily reduce labor even in environments where communication is difficult, thereby reducing the burden on users and improving work efficiency.

[0183] Furthermore, the excavation support system 1 consists of a portable measuring device 10 having an imaging unit 11, a three-dimensional measuring unit 12, a target identification means and a coordinate transformation means, and display means (management terminal 20, mobile terminal 30 and hydraulic excavator 40) connected to the measuring device 10 via a communication line 3.

[0184] With this configuration, even when the tunnel T is sequentially expanded, the imaging unit 11, the three-dimensional measurement unit 12, the target identification means, and the coordinate transformation means can be easily moved in accordance with the expansion of the tunnel T.

[0185] Furthermore, since the measurement device 10 performs a series of processes from imaging and three-dimensional measurement of the inside of tunnel T to acquiring point cloud data 5 in an absolute coordinate system, the excavation support system 1 can utilize existing display means such as the operation display unit 31 of the mobile terminal 30 and the display unit 41 of the hydraulic excavator 40 as display means for displaying the point cloud data 5 in an absolute coordinate system. As a result, the excavation support system 1 improves user convenience with the portable measuring device 10, and suppresses cost increases by using existing display means.

[0186] Furthermore, the excavation support system 1 is equipped with motion identification means (first machine learning model 17 and second machine learning model 18) that identify moving objects inside the tunnel T based on still images 4A. Furthermore, the excavation support system 1 includes a motion coordinate acquisition means (measurement control unit 15) that acquires position coordinate information indicating the position of a moving object in an absolute coordinate system based on point cloud data 5 in an absolute coordinate system, and a data output means (measurement control unit 15) that outputs the position coordinate information of the moving object (object information and construction machine information) in association with the point cloud data 5 in an absolute coordinate system.

[0187] With this configuration, the point cloud data 5 in absolute coordinate system is output with the position coordinate information of the moving object associated with it. Therefore, the point cloud data 5 with the position coordinate information of the moving object associated with it can be used as three-dimensional data to inform users, for example, of the presence and position of a moving object inside a tunnel T. Alternatively, the excavation support system 1 can use the point cloud data 5, which associates the position coordinate information of the moving object, as three-dimensional data to control the movement of, for example, the hydraulic excavator 40.

[0188] As a result, the excavation support system 1 can utilize the point cloud data 5 in absolute coordinate system as, for example, three-dimensional data to support the operation of the hydraulic excavator 40 by the user, or as three-dimensional data for the automatic operation of the hydraulic excavator 40, thereby further promoting labor saving and automation even in environments where communication is difficult.

[0189] Furthermore, the excavation support system 1 includes an intrusion determination means (terminal control unit 25) that determines whether a person has entered a restricted area set inside the tunnel T, based on the position coordinate information of a moving body acquired by the moving body coordinate acquisition means.

[0190] Furthermore, the excavation support system 1 is equipped with notification means (management terminal 20, portable terminal 30, and hydraulic excavator 40) that notifies the intrusion into a restricted area if the intrusion detection means determines that a person has entered the restricted area.

[0191] With this configuration, human intrusion into a restricted area can be reported based on point cloud data 5 in an absolute coordinate system, without the need to place sensors or other devices inside the tunnel T. As a result, the excavation support system 1 can improve safety inside the tunnel T with a simple configuration that keeps costs down.

[0192] Furthermore, the excavation support system 1 includes a movement determination means (terminal control unit 25) that determines whether a part of the hydraulic excavator 40 has moved outside the height limit set inside the tunnel T, based on the position coordinate information of the moving body acquired by the moving body coordinate acquisition means.

[0193] Furthermore, the excavation support system 1 is equipped with notification means (management terminal 20, portable terminal 30, and hydraulic excavator 40) that notifies the system of movement outside the height limit if the movement determination means determines that the system is moving outside the height limit.

[0194] With this configuration, the movement of the hydraulic excavator 40 outside the height limit can be reported based on point cloud data 5 in an absolute coordinate system, without the need to place sensors or other devices inside the tunnel T. As a result, the excavation support system 1 can improve safety inside the tunnel T with a simple configuration that keeps costs down.

[0195] Furthermore, marker 2 is positioned on the inner surface Ta of the tunnel T formed by excavating underground, and is configured to be associated with absolute coordinate system position information. This configuration makes it possible to easily acquire point cloud data 5 in the desired absolute coordinate system in underground spaces where communication with the outside is difficult. Therefore, the excavation support system 1 can use absolute coordinate system point cloud data 5 as three-dimensional data necessary to, for example, assist the user in operating the hydraulic excavator 40 in the underground space, or to automatically operate the hydraulic excavator 40. This allows the drilling support system 1 to further promote labor saving and automation in drilling sites where communication with the outside is difficult. [Examples]

[0196] The drilling support system 1 of Example 2 differs from that of Example 1 in that the position coordinate information of the measuring device 10 in the absolute coordinate system is known. This drilling support system 1 of Example 2 will be explained using Figure 14, which shows a flowchart of the calibration process in Example 2. Note that components identical to those in the above-described embodiments are denoted by the same reference numerals, and their detailed descriptions are omitted.

[0197] First, the excavation support system 1 consists of one or more markers 2 placed inside the tunnel T, a portable measuring device 10 that measures the inside of the tunnel T and processes various information, and a management terminal 20 and a mobile terminal 30 that display output data from the measuring device 10 to the user, all interconnected via a communication line 3.

[0198] In Example 2, all markers 2 are positioned such that the position of the corner 2a in the height direction, i.e., the position coordinate information of the corner 2a in the Z-axis direction of the absolute coordinate system, is the same.

[0199] Furthermore, the measurement device 10 of Embodiment 2 consists of an imaging unit 11 that captures images of the subject, a three-dimensional measurement unit 12 that performs three-dimensional measurement on objects inside the tunnel T, a line connection unit 13 that connects to the communication line 3, a storage unit 14 that stores various information, and a measurement control unit 15 that controls the operation of these units.

[0200] This measuring device 10 is positioned inside the tunnel T such that the mechanical height of the measuring device 10 in the absolute coordinate system is determined, the three-dimensional measuring unit 12 is positioned horizontally, and the absolute coordinate system position information corresponding to the position of the three-dimensional measuring unit 12 is pre-assigned.

[0201] Therefore, the storage unit 14 of the measuring device 10 stores a processing program (not shown) for performing various processes, marker information 16, a first machine learning model 17, and a second machine learning model 18, as well as the position coordinate information of the measuring device 10 in an absolute coordinate system. Specifically, the storage unit 14 stores the position coordinate information of the measuring device 10 as the position coordinate information of the measuring device 10. This information is based on the machine height of the measuring device 10 in an absolute coordinate system obtained by surveying using a known surveying method, and the distance from the machine height to the three-dimensional measuring unit 12.

[0202] Next, we will explain the processing flow in the excavation support system 1 with the configuration described above. First, with the power to the measuring device 10 turned on, when a user operates the management terminal 20 to execute a predetermined program, the terminal control unit 25 of the management terminal 20 displays a menu screen (not shown) on the display unit 22 that provides various instructions to the user.

[0203] At this point, when the user presses the button to start the calibration of the measuring device 10 in accordance with the instructions on the menu screen, the terminal control unit 25 transmits a calibration signal to start the calibration of the measuring device 10 to the measuring device 10 via the communication line 3, as shown in Figure 6 (step S101). The measurement control unit 15 of the measurement device 10, which has acquired the calibration signal, starts a calibration process to calibrate the device so that point cloud data 5 in absolute coordinate system is output, as shown in Figure 6 (step S102).

[0204] Specifically, when calibration processing is started, the measurement control unit 15 of the measurement device 10 performs, in parallel processing, an imaging process (step S141) in which the imaging unit 11 images the inside of the tunnel T, and a three-dimensional measurement process (step S142) in which the three-dimensional measurement unit 12 measures the inside of the tunnel T in three dimensions, as shown in Figure 14. Note that the imaging process in step S141 and the three-dimensional measurement process in step S142 are the same as the imaging process in step S111 and the three-dimensional measurement process in step S112 in the above-described Embodiment 1, so a detailed explanation thereof will be omitted.

[0205] After performing the imaging process in step S141 and the three-dimensional measurement process in step S142, the measurement control unit 15 identifies the patterns of all markers 2 in the still image 4A based on the color and shading in the still image 4A, as shown in Figure 14 (step S143). Subsequently, the measurement control unit 15 determines whether the number of markers 2 that were able to identify the pattern is one or more (step S144).

[0206] If the number of markers 2 whose patterns were identified is less than one (step S144: No), the measurement control unit 15 cannot perform the subsequent processing, so it sends an error notification indicating an identification error of marker 2 to the management terminal 20 (step S145), and then terminates the calibration process.

[0207] On the other hand, if the number of markers 2 whose patterns can be identified is one or more (step S144: Yes), the measurement control unit 15 identifies the corresponding measurement point 5a of the point cloud data 5 corresponding to the corner 2a of the marker 2, based on the markers 2 whose patterns can be identified, as shown in Figure 14, and assigns absolute coordinate system position coordinate information to the identified corresponding measurement point 5a (step S146).

[0208] In this case, the measurement control unit 15, in the same manner as in step S116 in the embodiment 1 described above, identifies the corresponding measurement point 5a of the point cloud data 5 that overlaps with the corner 2a of the marker 2 in the still image 4A for each marker 2 whose pattern was identified in step S143, and assigns absolute coordinate information to it.

[0209] Subsequently, as shown in Figure 14, the measurement control unit 15 uses the position coordinate information of one or more corresponding measurement points 5a acquired in step S146 and the position coordinate information of the measurement device 10 in the absolute coordinate system to calculate and estimate the absolute coordinate information of the absolute coordinate system, including the coordinate axes (step S147).

[0210] For example, if the number of markers 2 identified in step S143 is 1, the measurement control unit 15 estimates the orientation in the X-axis direction and the Y-axis direction, etc., by calculation based on the position coordinate information of the corresponding measurement point 5a and the position coordinate information of the measurement device 10, in much the same manner as in optical wave surveying, and acquires absolute coordinate information of the absolute coordinate system.

[0211] Subsequently, the measurement control unit 15 calculates and stores calibration values ​​for converting the point cloud data 5 acquired from the three-dimensional measurement unit 12 into an absolute coordinate system, based on the absolute coordinate information of the absolute coordinate system calculated in step S147 (step S148).

[0212] In this process, the measurement control unit 15 converts the point cloud data 5, whose position in the absolute coordinate system is unknown and was acquired in step S142, into point cloud data 5 in the absolute coordinate system, and then returns to step S102 in Figure 6 to complete the calibration process. The subsequent processes (steps S103 to S108 in Figure 6) are the same as those in Example 1 described above, so a detailed explanation will be omitted.

[0213] In this way, the excavation support system 1 of Example 2, similar to Example 1 described above, is capable of outputting point cloud data 5 in an absolute coordinate system suitable for known construction machinery that supports machine guidance and machine control at tunnel T excavation sites where position measurement using global navigation satellite systems is difficult.

[0214] As described above, the excavation support system 1 of Embodiment 2 includes an imaging unit 11 that is positioned on the side of the tunnel T and captures images of at least one marker 2 whose position in a desired absolute coordinate system is known, thereby acquiring a still image 4A.

[0215] Furthermore, the excavation support system 1 includes a three-dimensional measurement unit 12 located inside the tunnel T, which has a known position in the absolute coordinate system. This unit performs three-dimensional measurements on a measurement target whose position in the absolute coordinate system is unknown, and acquires point cloud data 5 of the measurement coordinate system. In addition, the drilling support system 1 is equipped with a target identification means (measurement control unit 15) that identifies the marker 2 in the still image 4A.

[0216] Furthermore, the drilling support system 1 includes a coordinate transformation means (measurement control unit 15) that transforms the point cloud data 5 acquired by the three-dimensional measurement unit 12 into an absolute coordinate system based on position coordinate information indicating the position of at least one identified marker 2 in an absolute coordinate system, and position coordinate information indicating the position of the three-dimensional measurement unit 12 in an absolute coordinate system.

[0217] With this configuration, based on the position coordinate information of marker 2 in the absolute coordinate system and the position coordinate information of the three-dimensional measurement unit 12, the point cloud data 5 in the measurement coordinate system is converted to the absolute coordinate system. Therefore, it is possible to obtain the desired point cloud data 5 in the absolute coordinate system without using, for example, the Global Navigation Satellite System.

[0218] Furthermore, as long as the three-dimensional measurement unit 12 is not moved, the relative position between the marker 2 and the three-dimensional measurement unit 12 does not change. Therefore, even if the internal state of the tunnel T changes over time, the excavation support system 1 can stably and repeatedly acquire point cloud data 5 in an absolute coordinate system that corresponds to the state of the tunnel T.

[0219] Even if the three-dimensional measurement unit 12 is moved, the excavation support system 1 can easily acquire the positional coordinate information of the three-dimensional measurement unit 12 after the move by moving the three-dimensional measurement unit 12 to a location where the positional coordinate information of the absolute coordinate system is known, such as a marker 2 inside the tunnel T. Therefore, even when the three-dimensional measurement unit 12 is moved, the excavation support system 1 can easily acquire new point cloud data 5 in the absolute coordinate system.

[0220] As a result, the excavation support system 1 can sequentially acquire point cloud data 5 in the desired absolute coordinate system, even in environments where communication with the outside is difficult, where the conditions inside the tunnel T are constantly changing, or where the three-dimensional measurement unit 12 needs to be moved. Therefore, the excavation support system 1 can acquire point cloud data 5 in an absolute coordinate system that can be used for various support technologies and autonomous driving technologies, regardless of the state of communication with the outside world. This makes it possible to promote labor saving and automation even in environments where communication is difficult.

[0221] In the correspondence between the structure of this invention and the embodiments described above, The space covered by this invention corresponds to the interior of tunnel T in the embodiment, The same applies to the following: The still life corresponds to the side of tunnel T, The given coordinate system corresponds to the absolute coordinate system. The target corresponds to marker 2, The imaging information corresponds to still image 4A, The measurement targets correspond to the inner surface Ta of tunnel T, the face Tb, the ground Tc, the hydraulic excavator 40, the worker M, and marker 2. The target identification means, coordinate transformation means, dynamic coordinate acquisition means, and data output means correspond to the measurement control unit 15. The information processing system corresponds to the excavation support system 1. The virtual object corresponds to the cross-sectional shape line 26a, The display and notification means correspond to the management terminal 20, the mobile terminal 30, and the hydraulic excavator 40. The motion recognition means corresponds to the first machine learning model 17 and the second machine learning model 18, The restricted area corresponds to the no-entry zone and height restrictions. The intrusion detection means and the movement detection means correspond to the terminal control unit 25. The imaging process corresponds to step S111, The three-dimensional measurement process corresponds to step S112. The target identification step corresponds to step S113, The coordinate transformation process corresponds to step S118, This invention is not limited to the configuration of the embodiments described above, and many other embodiments can be obtained.

[0222] For example, in the embodiment described above, a hydraulic excavator 40 compatible with machine guidance was used as an example of a construction machine, but the invention is not limited to this. For example, the construction machine may be a hydraulic excavator compatible with machine control, another construction machine compatible with machine guidance, or another construction machine compatible with machine control.

[0223] Furthermore, although the measuring device 10 is described as portable, it is not limited to this, and may be a measuring device integrally mounted on construction machinery such as a hydraulic excavator 40. Furthermore, while the notification method for intruding into a restricted area was described using a mobile terminal 30 owned by the user, it is not limited to this, and the notification method for intruding into a restricted area may also be configured using a wearable terminal worn by the user.

[0224] Furthermore, although the target space for imaging by the imaging unit 11 and three-dimensional measurement by the three-dimensional measurement unit 12 is set to the inside of the tunnel T, it is not limited to this, and the target space may be a predetermined area indoors, such as in a building, or outdoors. Furthermore, although Marker 2 is defined as an AR marker, it is not limited to this. Any suitable target that functions as a positional marker readable by the measuring device 10 may be used as Marker 2. For example, a two-dimensional barcode or a predetermined design affixed inside the tunnel T, or a three-dimensional structure placed on the floor of the tunnel T, may be used as Marker 2.

[0225] Furthermore, although marker 2 was attached to the side of tunnel T, it is not limited to this, and marker 2 may be placed at any appropriate position on the inner surface Ta of tunnel T. In addition, the spacing of marker 2 may be uneven. Furthermore, although the coordinate system assigned to marker 2 is assumed to be an absolute coordinate system, it is not limited to this. The coordinate system assigned to marker 2 may be a predetermined coordinate system, for example, one in which a predetermined position outside tunnel T is used as the reference point. In this case, the excavation support system 1 will acquire point cloud data 5 of the predetermined coordinate system.

[0226] Furthermore, although not mentioned in the embodiments described above, the point cloud data 5 in the absolute coordinate system may also be associated with the RGB values ​​of the still image 4A corresponding to the measurement point as attribute information for each measurement point. Furthermore, the processing operations in the drilling support system 1 in Figure 6, the processing operations in the calibration process in Figures 7 and 14, the processing operations in the monitoring process in Figure 11, and the processing operations in the support process in Figure 12 are examples and are not limited thereto; any appropriate processing operations may be used.

[0227] For example, the calibration process shown in Figure 7 may be performed each time imaging is performed by the imaging unit 11 and three-dimensional measurement is performed by the three-dimensional measurement unit 12. In addition, the absolute coordinate system point cloud data 5 output in step S118 may be used to perform the processing from step S124 onwards in the monitoring process shown in Figure 11.

[0228] Furthermore, in step S111 in Figure 7, step S121 in Figure 11, and step S141 in Figure 14, the imaging unit 11 is configured to output a still image 4A. However, the system is not limited to this configuration, and the imaging unit 11 may also output a video captured with the inside of the tunnel T as the subject. In this case, one frame from the video is used as a still image in subsequent processing.

[0229] Furthermore, in step S124 of the monitoring process in Figure 11, the first machine learning model 17 was used to identify each object in the still image 4A, and in step S126, the second machine learning model 18 was used to identify the movable parts of the hydraulic excavator 40 in the identified image 4B, but this is not limited to the above. For example, the first machine learning model 17 may be configured to identify each object and the movable parts of the construction machinery in the still image 4A.

[0230] Alternatively, markers attached to workers' helmets or other parts of construction machinery may be detected in still image 4 to identify each object or movable part of construction machinery in still image 4A without using a machine learning model.

[0231] Furthermore, in step S124 of the monitoring process in Figure 11, moving objects were identified based on the still image 4A and the identification image 4B. However, the process is not limited to this, and moving objects in the point cloud data 5 may also be identified by analyzing the point cloud data 5 using a machine learning model.

[0232] Even in this case, the excavation support system 1 can use the point cloud data 5 in absolute coordinate system as, for example, three-dimensional data to support the operation of the hydraulic excavator 40 by the user, or three-dimensional data for the automatic operation of the hydraulic excavator 40, similar to the embodiment described above.

[0233] Furthermore, in step S132 of the support processing in Figure 12, the cross-sectional shape line 26a, which is data indicating the design cross-sectional shape of the tunnel T, is displayed by overlaying it onto the point cloud data in absolute coordinate system. However, the system is not limited to this, and three-dimensional model data indicating the design shape of the tunnel T or data indicating virtual stakes may also be displayed by overlaying them. Alternatively, an image of a person may be superimposed on the point cloud data representing a person, and an image of a construction machine may be superimposed on the point cloud data representing a construction machine.

[0234] Alternatively, guide lines indicating the direction of movement of the hydraulic excavator 40, guide lines indicating the direction of operation of the bucket 46 of the hydraulic excavator 40, and frame lines indicating the excavation site in the ground Tc may be overlaid and displayed on the point cloud data in an absolute coordinate system.

[0235] Furthermore, in step S132 of the support processing shown in Figure 12, the still image 4A acquired in step S121 of the monitoring processing may be overlaid and displayed on the point cloud data 5 in absolute coordinate system, the still image 4A and the point cloud data 5 may be displayed separately, or various information may be overlaid and displayed on the still image 4A instead of the point cloud data 5.

[0236] Furthermore, in Example 1, based on the still image 4A acquired by the imaging unit 11 and the point cloud data 5 of the measurement coordinate system acquired by the three-dimensional measurement unit 12, absolute coordinate system position information was assigned to the corresponding measurement point 5a of the point cloud data 5 corresponding to the marker 2 in the still image 4A, and the position coordinate information of the three-dimensional measurement unit 12 in the absolute coordinate system was calculated, but the invention is not limited to this.

[0237] For example, the position coordinate information of the imaging unit 11 in the absolute coordinate system is estimated based on the size of marker 2 in the still image 4A and the position coordinate information of marker 2 in the absolute coordinate system. Subsequently, the position coordinate information of the three-dimensional measurement unit 12 in the absolute coordinate system may be estimated based on the estimated position coordinate information of the imaging unit 11 and the relative position between the imaging unit 11 and the three-dimensional measurement unit 12.

[0238] In addition, although the measuring device 10 transmits output data to the management terminal 20, and the management terminal 20 generates superimposed data and transmits it to the hydraulic excavator 40 and the mobile terminal 30, the system is not limited to this, and the measuring device 10 may directly transmit output data to the management terminal 20, the hydraulic excavator 40 and the mobile terminal 30.

[0239] In this case, the management terminal 20, the hydraulic excavator 40, and the mobile terminal 30 initiate the support processing shown in Figure 12, overlaying various information onto the point cloud data 5 and determining whether a person has entered a restricted area or whether a portion of the hydraulic excavator 40 has exceeded the height limit. Furthermore, in step S134 of the support process in Figure 12, the intrusion of a person into a restricted area was detected and reported, but this is just one example and is not limited to this; the intrusion of moving objects such as vehicles or construction machinery may also be detected.

[0240] Furthermore, in step S134 of the support process shown in Figure 12, it was determined whether the straight-line distance from the point cloud position coordinate information representing the hydraulic excavator 40 to the point cloud position coordinate information representing the human was less than or equal to a predetermined distance, but this is just one example and is not limited to this. For example, it may be determined whether or not the position coordinate information of a point cloud representing a human is located within a predetermined coordinate range that includes the hydraulic excavator 40.

[0241] Furthermore, in step S136 of the support process shown in Figure 12, the connection point 44 between the boom and the arm or the bucket 46 is considered as part of the hydraulic excavator 40, and a notification is given if a part of the hydraulic excavator 40 exceeds the height limit. However, this is just one example and is not limited to this. For example, a notification may be given if the bucket 46 or the upper rotating body of the hydraulic excavator 40 is located outside the restricted area.

[0242] Furthermore, in step S136 of the support processing shown in Figure 12, it was determined whether the straight-line distance from the point cloud position coordinate information indicating the connection point 44 between the boom and the arm or the bucket 46 to the point cloud position coordinate information indicating the inner surface Ta of the tunnel T is less than or equal to a predetermined height distance. However, this is just one example and is not limited to this.

[0243] For example, it may be determined whether the position coordinate information of a point group indicating the connection point 44 between the boom and the arm or the bucket 46 is located above the position coordinate information indicating the height limit in the height direction.

[0244] Also, in the excavation site of the tunnel T, although the excavation support system 1 and the excavation support method that perform various supports using the point group data in the absolute coordinate system have been described as the information processing system and the information processing method of the present invention, it is not limited thereto.

[0245] For example, in a notification system that notifies a user of various information based on the point group data in the absolute coordinate system, or a support system that displays various information based on the point group data in the absolute coordinate system to support the user, an information processing system and an information processing method for acquiring the point group data in the absolute coordinate system in an environment where communication with the outside is difficult may be used.

[0246] Alternatively, in an automatic driving system that controls the operations of a vehicle, an automatic transport device, etc. based on the position coordinate information in the absolute coordinate system, an information processing system and an information processing method for acquiring the point group data in the absolute coordinate system in an environment where communication with the outside is difficult may be used.

[0247] Also, not limited to the excavation of the tunnel T, for example, in the construction of a water discharge channel or a water storage tank provided underground as a flood control facility, an information processing system and an information processing method for a support system or an automatic driving system that supports a user may be used. Alternatively, in various manufacturing, various operations, and various construction works in a building, an information processing system and an information processing method for a support system or an automatic driving system that supports a user may be used.

[0248] Also, in Example 2, although the machine height in the absolute coordinate system was specified and the measuring device 10 was arranged so that the three-dimensional measuring unit 12 became horizontal, it is not limited thereto. For example, when there are at least two markers 2 that can be imaged by the imaging unit 11, it is not necessary to specify the machine height in the absolute coordinate system, and further, the three-dimensional measurement unit 12 does not have to be arranged horizontally. Alternatively, the three-dimensional measurement unit 12 may be arranged horizontally and at the same height position as the marker 2.

[0249] Also, in the first and second embodiments, the position coordinate information of the marker 2 and the position coordinate information of the three-dimensional measurement unit 12 are the position coordinate information in the absolute coordinate system obtained by surveying using a known surveying method, but it is not limited to this. For example, by arranging the marker 2 and the three-dimensional measurement unit 12 at positions where the position coordinate information in the absolute coordinate system is known, which are surveyed at another timing, the position coordinate information of the marker 2 and the position coordinate information of the three-dimensional measurement unit 12 may be obtained.

[0250] Furthermore, when moving the measuring device 10, for example, by moving the measuring device 10 so that the three-dimensional measurement unit 12 is located at a position where the position coordinate information in the absolute coordinate system is known, such as the marker 2 which is the imaging target before movement, the position coordinate information of the destination where the position in the absolute coordinate system is known can be obtained as the position coordinate information of the three-dimensional measurement unit 12 after movement. Thereby, the excavation support system 1 can eliminate the need to survey the position of the three-dimensional measurement unit 12 after movement.

Explanation of Reference Numerals

[0251] 1... Excavation support system 2... Marker 3... Communication line 4A... Still image 5... Point cloud data 5a... Corresponding measurement point 10... Measuring device 11... Imaging unit 12... Three-dimensional measurement unit 15... Measurement control unit 17... First machine learning model 18... Second machine learning model 20... Management terminal 25... Terminal control unit 26a...Cross-sectional shape line 30… Mobile devices 40... Hydraulic excavator M…Worker T...Tunnel Tc...Ground

Claims

1. An imaging unit that captures images of at least three targets placed on a stationary object in the target space, whose positions in a desired predetermined coordinate system are known, and acquires imaging information; A three-dimensional measurement unit that performs three-dimensional measurement on a measurement target located within the aforementioned target space and whose position in the predetermined coordinate system is unknown, and acquires point cloud data of the measurement coordinate system, A target identification means for identifying the target in the imaging information, The system is equipped with coordinate transformation means that transforms the point cloud data acquired by the three-dimensional measurement unit into the predetermined coordinate system based on position coordinate information indicating the position of the identified target in the predetermined coordinate system. Information processing system.

2. The three-dimensional measurement unit is The configuration is for performing three-dimensional measurements within the target space which includes at least three of the aforementioned targets. The aforementioned coordinate transformation means is The configuration involves assigning the position coordinate information of the target to corresponding measurement points among the measurement points that constitute the point cloud data, which correspond to the target identified by the target identification means, and converting the point cloud data acquired by the three-dimensional measurement unit into the predetermined coordinate system based on the position coordinate information assigned to the corresponding measurement points. The information processing system according to claim 1.

3. The aforementioned coordinate transformation means is Based on the position coordinate information of the corresponding measurement points, position coordinate information indicating the position of the three-dimensional measurement unit in the predetermined coordinate system is calculated, and the point cloud data acquired by the three-dimensional measurement unit is converted to the predetermined coordinate system based on the position coordinate information of the three-dimensional measurement unit. The information processing system according to claim 2.

4. The imaging unit and the three-dimensional measurement unit are provided integrally. The information processing system according to claim 3.

5. The system is equipped with a display means that displays the virtual object of the predetermined coordinate system superimposed on the point cloud data of the predetermined coordinate system. The information processing system according to claim 1.

6. A portable measuring device having the imaging unit, the three-dimensional measurement unit, the target identification means and the coordinate transformation means, It consists of the display means connected to the measuring device via a communication line. The information processing system according to claim 5.

7. A motion identification means for identifying a moving object in the target space based on the imaging information or the point cloud data of the predetermined coordinate system, A motion coordinate acquisition means that acquires position coordinate information indicating the position of the motion in the predetermined coordinate system based on the point cloud data of the predetermined coordinate system, The system is equipped with data output means for outputting the position coordinate information of the moving body in association with the point cloud data of the predetermined coordinate system. The information processing system according to claim 1.

8. An intrusion determination means determines, based on the position coordinate information of the moving body acquired by the moving body coordinate acquisition means, that the moving body enters a restricted area set within the target space. If the intrusion detection means determines that the moving object has entered the restricted area, the system is provided with a notification means for notifying the intrusion into the restricted area. The information processing system according to claim 7.

9. A movement determination means determines whether the moving body has moved outside the restricted area set within the target space, based on the position coordinate information of the moving body acquired by the moving body coordinate acquisition means, If the determination result by the movement determination means indicates movement outside the restricted area, the system is provided with notification means for notifying that movement outside the restricted area has occurred. The information processing system according to claim 7.

10. The aforementioned target is It is positioned on the inner surface of the target space formed by excavating underground, and is configured to be assigned positional coordinate information in an absolute coordinate system. The information processing system according to claim 1.

11. The imaging process involves the imaging unit capturing images of at least three targets, which are placed on a stationary object in the target space and whose position in a desired predetermined coordinate system is known, in order to acquire imaging information. A three-dimensional measurement process involves a three-dimensional measurement unit performing a three-dimensional measurement on a measurement target located within the aforementioned target space, whose position in the predetermined coordinate system is unknown, and acquiring point cloud data of the measurement coordinate system. A target identification step in which the target identification means identifies the target in the imaging information, Based on position coordinate information indicating the position of the identified target in the predetermined coordinate system, the three-dimensional measurement unit performs a coordinate transformation step in which the coordinate transformation means transforms the point cloud data acquired by the three-dimensional measurement unit into the predetermined coordinate system. Information processing methods.

12. An imaging unit that captures imaging information by imaging at least one target that is placed on a stationary object in the target space and whose position in a desired predetermined coordinate system is known, A three-dimensional measurement unit, whose position in the predetermined coordinate system is known, performs three-dimensional measurement on a measurement target located within the aforementioned target space and whose position in the predetermined coordinate system is unknown, and acquires point cloud data of the measurement coordinate system. A target identification means for identifying the target in the imaging information, The system includes position coordinate information indicating the position of at least one identified target in the predetermined coordinate system, and coordinate transformation means that transforms the point cloud data acquired by the three-dimensional measurement unit into the predetermined coordinate system based on position coordinate information indicating the position of the three-dimensional measurement unit in the predetermined coordinate system. Information processing system.

Images