Detection device and detection method

Abstract

To acquire an excavation position of an excavator by means of an inexpensive method.SOLUTION: A detection device 100 comprises: an image acquisition unit 30 which acquires an object image obtained by imaging the foundation and an excavator 10; a processing detection unit 31 which detects a tip of the excavator in the object image by applying image processing to the object image; an energy information acquisition unit 32 which acquires information about an excavation energy value of excavating the foundation by the excavator in association with the object image; and a position specification unit 33 which specifies the position of the tip of the excavator 10 detected in the associated object image as an excavation position when the excavation energy value equal to or greater than a prescribed threshold is acquired.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a detection device and a detection method for detecting the excavation position of an excavator. [Background technology]

[0002] Patent Document 1 describes a method for evaluating the natural ground ahead of the tunnel face using drilling energy measured by a jumbo drill during the excavation of a mountain tunnel. Patent Document 1 also describes the automatic acquisition of drilling data, such as drilling position, drilling energy, and drilling direction angle data, during drilling, using a computer-controlled jumbo drill. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-127655 Summary of the Invention [Problem to be solved by the invention]

[0004] In the method described in Patent Document 1, various sensors are provided in the drill jumbo to sense the operation of the drill jumbo, thereby automatically obtaining the drilling position.

[0005] However, acquiring the drilling position from a sensor attached to the drill jumbo requires multiple sensors. It is also necessary to measure the sensor installation position and the shape of the drill jumbo and calculate the tip position of the drill jumbo. Therefore, the method of acquiring the drilling position by using sensors to sense the operation of the drill jumbo is costly, and there is a need for a cheaper method of acquiring the drilling position.

[0006] An object of the present invention is to obtain the excavation position of an excavator in an inexpensive manner. [Means for solving the problem]

[0007] The present invention is a detection device for detecting the excavation position of ground by an arm-shaped excavator, and comprises an image acquisition unit that acquires a target image of the ground and the excavator, a processing detection unit that performs image processing on the target image to detect the tip of the excavator in the target image, an energy information acquisition unit that acquires information on the excavation energy value for excavating the ground by the excavator in association with the target image, and a position identification unit that, when an excavation energy value equal to or greater than a predetermined threshold is acquired, identifies the position of the tip of the excavator detected in the associated target image as the excavation position.

[0008] The present invention is a detection method for detecting the excavation position of ground by an arm-shaped excavator, which involves acquiring a target image of the ground and the excavator, applying image processing to the target image to detect the tip of the excavator in the target image, acquiring information on the excavation energy value used to excavate the ground by the excavator in association with the target image, and when an excavation energy value greater than or equal to a predetermined threshold is acquired, identifying the position of the tip of the excavator detected in the associated target image as the excavation position. [Effects of the Invention]

[0009] According to the present invention, the excavation position can be obtained inexpensively. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic diagram of a detection system according to an embodiment of the present invention; [Figure 2] 1 is a block diagram showing a configuration of a detection system according to an embodiment of the present invention. [Figure 3] 3 is a flowchart illustrating a detection method according to an embodiment of the present invention. [Figure 4] 1 is a flowchart illustrating an evaluation method according to an embodiment of the present invention. [Figure 5] 10A and 10B are diagrams illustrating a state in which an excavator is detected in a target image in a detection method according to an embodiment of the present invention. [Figure 6]10A and 10B are diagrams illustrating a method for estimating the position of the tip of an excavator in an object image in a detection method according to an embodiment of the present invention; [Figure 7] FIG. 10 is a diagram showing a state in which a working face is divided into a plurality of regions in a target image in an evaluation method according to an embodiment of the present invention. [Figure 8] 1A and 1B are diagrams illustrating evaluation images in an evaluation method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] A detection system 101 including a detection device 100 and a detection method according to an embodiment of the present invention will be described below with reference to the drawings.

[0012] The detection system 101 is applied to a work machine 102 that has an arm-shaped excavator 10 and excavates the ground with the excavator 10, and detects the position where the excavator 10 excavates the ground.

[0013] First, the overall configuration of a work machine 102 will be described with reference to FIG.

[0014] In this embodiment, the work machine 102 is a jumbo drill that drills blast holes H for loading explosives into a face S of a tunnel T, which is the ground to be excavated. The work machine 102 includes an arm-shaped excavator 10 and a mobile carriage 1 on which the excavator 10 is mounted.

[0015] 1, the excavator 10 has a rod 12 with a bit 11 attached to one end, a drifter 13 that supports the other end of the rod 12, and a guide cell 14 that guides the movement of the drifter 13. When the drifter 13 moves back and forth along the guide cell 14 by a drive mechanism (not shown), the rod 12 also moves back and forth along the guide cell 14.

[0016] The drifter 13 is capable of applying a rotational force and an impact force to the rod 12. The bit 11 is pressed against the rock at the working face S, and the bit 11 is rotated and used to impact the rock, thereby drilling a blast hole H. The work machine 102 normally includes multiple excavators 10, but in this embodiment, for ease of explanation, only a single excavator 10 is shown.

[0017] The mobile carriage 1 supports the base end of an extendable boom 15 so that it can swing freely in the vertical and horizontal directions, and the guide cell 14 of the excavator 10 is supported at the tip of the boom 15 so that it can swing freely in the vertical and horizontal directions. The boom 15 swings and expands relative to the mobile carriage 1 by a drive mechanism (not shown). The excavator 10 moves up, down, left, and right as the boom 15 swings and expands. The guide cell 14 of the excavator 10 swings relative to the boom 15 by a drive mechanism (not shown). As the guide cell 14 swings, the orientation of the excavator 10 (the orientation of the rod 12) changes.

[0018] The excavator 10 is operated by an operator to drill the working face S and perforate the blast hole H. Normally, the mobile carriage 1 does not move while the excavator 10 is drilling the working face S, and the positional relationship between the mobile carriage 1 and the working face S does not change.

[0019] As shown in Figures 1 and 2, the detection system 101 includes a camera 20 as an imaging unit that images the working face S, a measurement unit 25 for acquiring the drilling energy value by the excavator 10, and a detection device 100 that detects the drilling position of the excavator 10 at the working face S.

[0020] The camera 20 is attached to the mobile carriage 1 so as to be able to capture an image of the entire working face S being drilled by the excavator 10. The camera 20 continuously captures images of the working face S to obtain image data in the form of a moving image. Note that the camera 20 may also be one that intermittently captures still images. The image data obtained by the camera 20 is input to the detection device 100.

[0021] The measurement unit 25 measures parameters representing the operating state of the excavator 10, which are necessary to obtain the excavation energy value of the excavator 10. The measurement unit 25 measures parameters representing the operating states of the multiple excavators 10 in a manner that allows them to be distinguished from one another. In this embodiment, the measurement unit 25 is configured with various sensors that measure the rotational force and impact force applied to the rod 12 by the drifter 13. For example, although not shown, the measurement unit 25 includes a torque sensor that detects the rotational force of the rod 12, a pressure sensor that detects the impact force of the rod 12, and a speed sensor that detects the moving speed of the rod 12. The measurement unit 25 measures the rotational force, impact force, and moving speed of the rod 12 in accordance with the interval (sampling interval) between still image captures by the camera 20. In other words, the measurement unit 25 measures each measurement item for each frame captured by the camera 20. The measurement results of the measurement unit 25 are input to the detection device 100. Note that the sampling interval of the measurement unit 25 only needs to be set so that the measurement unit 25 can obtain at least the excavation energy when a target image, described below, is captured.

[0022] The detection device 100 is configured by a computer including a CPU (Central Processing Unit) that executes a control program and the like, a ROM (Read-Only Memory) that stores the control program executed by the CPU, a RAM (Random Access Memory) that stores the results of CPU calculations and the like, a communication device, etc. The detection device 100 performs various functions of the detection device 100 described in this specification by the CPU executing the control program stored in the ROM. The detection device 100 may be configured by a single computer, or may be configured by multiple microcomputers and configured to distribute the various controls among the multiple computers.

[0023] The detection device 100 is mounted on, for example, the mobile dolly 1, and is communicatively connected to the camera 20 and the measurement unit 25 via wired or wireless communication. Note that the detection device 100 is not limited to being mounted on the mobile dolly 1, and may be provided in a location other than the mobile dolly 1. For example, the detection device 100 may be configured as a server computer wirelessly connected to the camera 20 and the measurement unit 25 via a network.

[0024] 2, the detection device 100 includes an image acquisition unit 30 that acquires a target image from image data input from the camera 20, a processing and detection unit 31 that performs image processing on the target image to detect the tip of the excavator 10 in the target image, an energy information acquisition unit 32 that acquires information on the excavation energy value for drilling the face S by the excavator 10 in association with the target image, a position identification unit 33 that identifies the excavation position by the excavator 10 in the target image based on the excavation energy value, and an energy calculation unit 34 that divides the imaged face S into multiple regions and calculates the excavation energy value for each region based on the excavation energy value at one or more excavation positions included in each region. Note that each component of the detection device 100 shown in FIG. 2 is shown as a virtual unit representing each function of the detection device 100 and does not mean that it physically exists.

[0025] The image acquisition unit 30 extracts still images at predetermined time intervals from image data input in the form of a moving image. The extracted still images (hereinafter referred to as "target images") are subjected to image processing by the processing detection unit 31. The interval at which still images are extracted from the image data may be the same as the sampling interval at which the camera 20 captures still images (i.e., all frames of the moving image are extracted as target images), or may be longer than the sampling interval. In the detection method described below, each frame of the moving image is extracted as the target image.

[0026] The processing and detection unit 31 detects the front end, rear end, and entire excavator 10 in the target image using a trained model that has been machine-learned using as training data a group of images in which the front end of the excavator 10 is identified. The front end of the excavator 10 here refers to the front end of the rod 12 to which the bit 11 is attached.

[0027] The trained model is constructed by having an operator label the front end, rear end, and entirety of the excavator 10 in a group of images previously taken of the working face S and the excavator 10, and then having a computer perform machine learning using deep learning using the labeled group of images as training data. The trained model is trained to be able to identify each of multiple excavators 10. For example, a trained model that can identify the excavators 10 can be generated by attaching markers or the like to each excavator 10 and using a group of images labeled based on the markers as training data.

[0028] Through image processing and object recognition by the processing and detection unit 31, rectangular regions (bounding boxes) B1, B2, B3 corresponding to the front end, rear end, and entire excavator 10, respectively, are extracted (see FIG. 5). Furthermore, if the processing and detection unit 31 is unable to extract the front end of the excavator 10, it has the function of estimating and detecting the position of the front end based on the extracted rear end and entire excavator. Details of this function will be described later.

[0029] The energy information acquisition unit 32 acquires the excavation energy value of the excavator 10 when the target image was captured based on the measurement results of the measurement unit 25. The excavation energy value is calculated based on the rotational force, impact force, and movement speed of the excavator 10. A known method can be used to calculate the excavation energy value from the rotational force, impact force, and movement speed of the excavator 10, so a detailed explanation will be omitted. The target image and the excavation energy value when the target image was captured are stored in association with each other and input to the position identification unit 33.

[0030] If the drilling energy value of the target image input from the energy information acquisition unit 32 is greater than a preset threshold, the position identification unit 33 identifies the position of the tip of the excavator 10 in the target image as the drilling position relative to the face S. In this way, the excavation position in the target image is identified. The threshold is set to a value greater than zero so that fluctuations in the excavation energy value caused by errors or noise in the sensors of the measurement unit 25 when the excavator 10 is not excavating the face S will not have an effect.

[0031] When the position specifying unit 33 specifies the excavation position in the target image, it compares the target image with a reference image in which the shape of the face S is specified in advance, and specifies the shape of the face S in the target image. This specifies the relative positional relationship between the detected excavation position and the face S.

[0032] The energy calculation unit 34 creates a grid on the target image and divides the working face S in the target image into multiple regions. The energy calculation unit 34 calculates the excavation energy value for each divided region based on the position of the working face S and the excavation position in the target image identified by the position identification unit 33, and the excavation energy value for excavating the excavation position. The excavation energy value for each region is one excavation energy value (representative value) corresponding to each region. If multiple blast holes H are provided in a region, the excavation energy value for each region can be the average value of the excavation energy values for excavating each blast hole H, or the maximum value of the excavation energy values of the blast holes H in the region. In the evaluation method for the working face S described below, an example will be explained in which the maximum value is used as the representative value.

[0033] Furthermore, the energy calculation unit 34 generates an evaluation image by assigning color and brightness to multiple regions divided within the target image according to the magnitude of the excavation energy value. The generated evaluation image is output from the detection device 100 to an external monitor. This allows the worker to understand the hardness or softness of the working face S by checking the distribution of the excavation energy value expressed by differences in color and brightness in the evaluation image.

[0034] Next, a method for detecting an excavation position and a method for evaluating the excavation face S by the detection system 101 will be described with reference to FIGS.

[0035] Fig. 3 is a flowchart showing a method for detecting an excavation position executed by the detection device 100, and Fig. 4 is a flowchart showing a method for evaluating the face S. The detection device 100 detects the excavation position of the excavator 10 by performing image processing on the target image, and evaluates the condition of the face S based on the excavation position and the excavation energy value required for excavation.

[0036] First, a method for detecting an excavation position will be described. The detection device 100 executes the excavation position detection process shown in Fig. 3 at a predetermined processing interval. In this embodiment, the processing interval of the detection process executed by the detection device 100 matches the time interval between frames captured by the camera 20. In other words, the detection system 101 executes the process shown in Fig. 3 for each frame of the video captured by the camera 20. Note that the processing interval executed by the detection system 101 only needs to be set to be equal to or longer than the time interval between frames.

[0037] In step S10, one frame of a still image is extracted from the moving image captured by the camera 20 and acquired as a target image.

[0038] In step S11, the trained machine learning model is used to extract from the target image the regions of the front end, rear end, and entire excavator 10. Specifically, as shown in Fig. 5, a rectangular region B1 indicating the front end of the excavator 10, a rectangular region B2 indicating the rear end, and a rectangular region B3 indicating the entire excavator 10 are extracted.

[0039] Here, since the tip of the excavator 10 comes into contact with the working face S and excavates, it is difficult to extract an area in the target image compared to the rear end or the entire excavator in a method of object recognition in an image using a machine learning model. For this reason, the rear end and the entire excavator 10 may be detected from the target image, but the tip may not be detected. Note that the entire excavator 10 is easier to detect than the tip, because it exists within a certain range in the target image and feature values can be extracted even if the tip is not detected.

[0040] In such a case, the detection system 101 detects the tip by estimating the position of the tip based on the relative positional relationship between the rear end of the excavator 10 and the entire excavator. Specifically, as shown in FIG. 6, a rectangular area B2 indicating the rear end of the excavator 10 is usually located within a rectangular area B3 indicating the entire excavator 10. The tip of the excavator 10 is located at the end of the body extending from the rear end of the excavator 10. Therefore, if only the tip of the excavator 10 is not detected, the detection device 100 estimates the position P1 of the tip in the target image to be point-symmetrical to the center point P3 of the rectangular area B3 indicating the entire excavator 10 and the center point P2 of the rectangular area B2 at the rear end. That is, on an imaginary line (a two-dot chain line in FIG. 6) connecting the center point P3 of a rectangular area B3 showing the entire excavator 10 with the center point P2 of a rectangular area B2 showing the rear end, a point P1 that is equidistant from the center point P2 of the entire excavator 10 to the center point P3 of the rear end is detected as the tip. This makes it possible to detect the hole-boring position of the excavator 10 with a certain degree of accuracy even if the tip of the excavator 10 cannot be directly detected from the target image.

[0041] Steps S12 and S13 are executed in parallel with the execution of steps S10 and S11. Note that steps S10 and S11 and steps S12 and S13 are not limited to parallel processing and may be processed serially, and the order of these steps may be either. In step S12, the measurement result (measurement result corresponding to the target image) by the measurement unit 20 when the target image is captured is obtained.

[0042] In step S13, the excavation energy value of the excavator 10 is calculated from the measurement results acquired in step S12.

[0043] When the excavator 10 is extracted in step S11 and the excavation energy value is calculated in step S13, it is determined in step S14 whether the excavation energy is equal to or greater than a predetermined threshold. If the excavation energy value is smaller than the threshold, the process ends. If the excavation energy is equal to or greater than the threshold, the process proceeds to step S15, where the target image (more specifically, the position of the tip of the excavator 10 within the target image) and the excavation energy value are stored in association with each other. The position of the tip of the excavator 10 at this time is detected as the position of the hole being drilled by the excavator 10. When step S15 is completed, the process ends.

[0044] The detection device 100 continues to detect such drilling positions, for example, until a predetermined number of blast holes H are formed. Once the predetermined number of blast holes H have been detected, the detection device 100 performs the evaluation process shown in Fig. 4 to evaluate the working face S. The detection device 100 may be configured to perform the process shown in Fig. 4 when an operator inputs an operation, for example. In this way, the timing for performing the process shown in Fig. 4 can be set arbitrarily.

[0045] As shown in FIG. 4, in step S20, the shape of the ground (shape of the face S) to be excavated in the target image is identified. Specifically, a reference image in which the shape of the face S (position in the image) is identified is stored in advance in the detection system 101. The reference image is, for example, an image showing the design shape of the face S in the excavation of the tunnel T. The shape of the face S in the target image is identified by fitting the target image to the reference image. The fitting involves moving and scaling the reference image so that the face S in the reference image overlaps with the face S in the target image, thereby superimposing the target image and the reference image. The fitting of the reference image to the target image may be performed by the detection device 100 or may be performed manually by an operator. As a result, the excavation position, which was a coordinate position in the target image, is recognized as a relative position with respect to the face S.

[0046] Next, in step S21, the working face S in the target image is divided into a plurality of regions by grid lines, as shown in Fig. 7. Note that the spacing and shape of the grid lines are not limited to the example shown in Fig. 6, and can be set arbitrarily. Also, each point shown in Fig. 7 indicates the excavation position of the excavator 10.

[0047] In step S22, one excavation energy value (representative value) is calculated for each cell C divided by grid lines. If multiple blast holes H are formed in cell C, the representative value of the excavation energy value is the maximum value of the excavation energy values for the multiple blast holes H. Also, if no blast holes H are formed in cell C, the excavation energy value of that cell C is calculated as 0 (zero).

[0048] Next, in step S23, an evaluation image is generated in which cells C are color-coded according to the magnitude of the excavation energy value (see FIG. 8). Then, in step S24, the evaluation image is output to, for example, an external monitor, and the process ends. This allows the worker to determine the hardness or softness of the ground by checking the evaluation image.

[0049] According to the above embodiment, the following advantageous effects are achieved.

[0050] In this embodiment, the tip of the excavator 10 is detected by image processing, and it is possible to determine whether the excavator 10 is excavating the ground based on the excavation energy value. In other words, the tip of the excavator 10 while excavating the ground can be detected by image processing, and the excavation position relative to the ground can be identified. In this way, because the position of the tip of the excavator 10 can be identified by image processing without using a sensor, it is possible to reduce the number of sensors that sense the operation of the excavator 10, and the excavation position can be obtained more inexpensively.

[0051] Furthermore, in this embodiment, only when the excavation energy value is equal to or greater than a threshold value is the position of the tip of the excavator 10 when the excavation energy value is calculated detected as the excavation position. An evaluation image is generated based on the excavation position and the excavation energy value. This makes it possible to grasp the hardness of the face S. By grasping the hardness of the face S, it is possible to manage the charge loading for the blast hole H, predict the hardness of the ground to be excavated (such as the new face S that will be created after blasting), evaluate and manage the supports to be installed in the tunnel T, and so on.

[0052] Next, a modification of this embodiment will be described.

[0053] In the above embodiment, the front and rear ends of the excavator 10 are detected in the target image by a machine-learned learning model. The detection system 101 may calculate the angle of the excavator 10, and therefore the angle at which the hole is drilled, based on the front and rear ends. Because the total length of the excavator 10 can be known in advance, it is possible to calculate the excavation angle from the total length of the excavator 10 and the positions of the front and rear ends of the excavator 10. The excavation angle calculated in this manner may be used, for example, to evaluate the accuracy of drilling.

[0054] In addition, in the above embodiment, the detection device 100 is applied to a drill jumbo that drills the working face S, but is not limited to this and may also be applied to, for example, a road header that similarly excavates the ground using an excavator 10.

[0055] Furthermore, in the above embodiment, when the excavation energy value becomes equal to or greater than the threshold value, the position of the tip of the excavator 10 at that time is acquired as the excavation position. This configuration does not prevent the position of the tip of the excavator 10 from being stored when the excavation energy value does not reach the threshold value.

[0056] In the above embodiment, the shape of the tunnel face S in the target image is identified by comparing (fitting) the target image with the reference image, and the relative position of the blast hole H with respect to the tunnel face S is determined. In the above embodiment, the reference image is based on the design shape of the tunnel face S. However, for example, the tunnel face S may actually be imaged, and the shape of the tunnel face S in the image may be detected by image processing or manually by an operator to be used as the reference image. Furthermore, the determination of the relative position of the blast hole H with respect to the tunnel face S is not limited to the method of comparing the target image with the reference image. Even without using the reference image, the shape of the tunnel face S in the target image captured by the camera 20 can be identified by previously determining the actual shape of the tunnel face S and the relative position of the camera 20 with respect to the tunnel face S. Furthermore, the shape of the tunnel face S in the target image captured by the camera 20 can be identified by performing image processing on the target image to detect the boundary between the tunnel face S and the inner wall of the tunnel T.

[0057] Furthermore, in the above embodiment, the processing detection unit 31 detects the excavator 10 in the target image using a machine-learned learning model, but the detection of the excavator 10 is not limited to image processing using a learning model. When using a learned model, if the ground to be excavated is updated or changed, it is desirable to label the image of the ground to create new training data and allow the new training data to be additionally trained.

[0058] Although the embodiments of the present invention have been described above, the above embodiments merely illustrate some of the application examples of the present invention, and it is not intended that the technical scope of the present invention be limited to the specific configurations of the above embodiments.

[0059] The series of processes in the detection device 100 described above may be provided as a program for causing a computer to execute the processes. The program for executing the series of processes described above may be provided by a storage medium readable by the detection device 100. The program may also be provided to the detection device 100 via a network line. The various programs executed by the detection device 100 may be stored in a non-transitory storage medium such as a CD-ROM. [Explanation of symbols]

[0060] 100 Detection device 10. Excavator 20 Camera (imaging unit) 25 Measuring part 30 Image acquisition unit 31 Processing detection unit 32 Energy information acquisition unit 33 Location identification part 34 Energy calculation unit

Claims

1. A detection device for detecting an excavation position of a ground by an arm-shaped excavator, an image acquisition unit that acquires a target image obtained by capturing an image of the ground and the excavator; a processing and detection unit that applies image processing to the target image to detect the tip of the excavator in the target image; an energy information acquisition unit that acquires information on an excavation energy value used to excavate the ground by the excavator in association with the target image; and a position specifying unit that specifies, when the excavation energy value is equal to or greater than a predetermined threshold, the position of the tip of the excavator detected in the associated target image as the excavation position. A detection device characterized by:

2. 2. The detection device according to claim 1, the processing and detection unit detects the tip of the excavator using a trained model that has been machine-learned using training data. A detection device characterized by:

3. 3. The detection device according to claim 1 or 2, the processing and detection unit is configured to detect the front end, rear end, and entirety of the excavator, and if the front end cannot be detected, detects the position of the front end from the positional relationship between the rear end and the entirety; A detection device characterized by:

4. 3. The detection device according to claim 1 or 2, the position specifying unit specifies the shape of the ground in the target image by comparing the target image with a reference image in which the shape of the ground in the image is specified, and specifies a positional relationship between the ground and the excavation position; A detection device characterized by:

5. 5. The detection device according to claim 4, The ground is divided into a plurality of regions, and an energy calculation unit is further provided which calculates the excavation energy value in each region based on the excavation energy value at one or more excavation positions included in each region. A detection device characterized by:

6. A detection method for detecting an excavation position of a ground by an arm-shaped excavator, comprising: acquiring a target image of the ground and the excavator; Image processing is performed on the target image to detect the tip of the excavator in the target image. Acquire information on an excavation energy value for excavating the ground by the excavator in association with the target image; When the excavation energy value is equal to or greater than a predetermined threshold, the position of the tip of the excavator detected in the associated target image is identified as the excavation position. A detection method characterized by:

Images