Face evaluation support device and face evaluation support method

Abstract

To provide a face evaluation supporting apparatus capable of improving evaluation quality in face evaluation.SOLUTION: A face evaluation supporting apparatus 4 is provided with an image data acquisition unit 11 for acquiring image data in which a target region including a tunnel face is photographed, a point cloud data acquisition unit 12 for acquiring point cloud data in which a three-dimensional shape is measured, a face evaluation unit 13 for evaluating the face based on the image data and the point cloud data, and a support information output unit 14 for outputting support information based on an evaluation result of the face evaluation unit 13. The face evaluation unit 13 is provided with a face extraction unit 130 for extracting a face region corresponding to the face from the target region, a roughness quantification unit 131 for quantifying the roughness of the face surface in the face region as a roughness feature amount based on the point cloud data, an area division unit 132 for dividing the face region into divided areas based on the distribution of representative values of the roughness feature amount, and a divided area evaluation processing unit 133 for evaluating an observation item related to the face as an evaluation result for each divided area.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to a face evaluation support device and a face evaluation support method.

Background Art

[0002] In order to ensure the safety and workability of tunnel construction, it is required to appropriately evaluate the state of the face of the tunnel. Conventionally, a professional engineer visually observes the face and makes a sketch record of the observation results. However, in recent years, in order to cope with the shortage of professional engineers and ensure the quality of the observation results, a face evaluation device using image data (photographs) of the face taken and point cloud data obtained by measuring the three-dimensional shape of the face has been developed.

[0003] For example, Patent Document 1 discloses a rock weathering and alteration evaluation device that derives an evaluation category of weathering and alteration based on an imaging image of the rock face of the face. Further, Patent Document 2 discloses a crack evaluation device that evaluates cracks developed in the rock based on point cloud data obtained by measuring three-dimensional solid information of the face of the tunnel.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the devices disclosed in Patent Document 1 and Patent Document 2, since the face of the tunnel heading is evaluated using image data and point cloud data respectively, there is a limit to improving the evaluation quality when evaluating the face of the tunnel heading by simply evaluating the image data and the point cloud data individually. For example, in the image data, in addition to the face of the tunnel heading, supports, roadbed parts, etc. are also photographed. Therefore, on the image data, a process for separating the area of the face of the tunnel heading and the area other than the face of the tunnel heading is required. If this process is insufficient, the evaluation quality of the face of the tunnel heading may be degraded. In addition, the face of the tunnel heading may contain a plurality of rock masses with different characteristics. If these rock masses are not classified and evaluated for various observation items, the state of the face of the tunnel heading may not be appropriately evaluated.

[0006] The present invention has been made in view of the above-described problems, and an object thereof is to provide a face evaluation support device and a face evaluation support method that can improve the evaluation quality when evaluating the face of the tunnel heading.

Means for Solving the Problems

[0007] In order to achieve the above object, a face evaluation support device according to an aspect of the present invention includes an image data acquisition unit that acquires image data in which a target area including the face of a tunnel is photographed, a point cloud data acquisition unit that acquires point cloud data in which the three-dimensional shape of the target area is measured, a face evaluation unit that evaluates the face of the tunnel based on the image data and the point cloud data, and a support information output unit that outputs support information based on an evaluation result by the face evaluation unit. The face evaluation unit includes a face extraction unit that extracts a face area corresponding to the face of the tunnel from the target area based on at least one of the image data and the point cloud data, a roughness quantification unit that quantifies the roughness of the face surface in the face area as a roughness feature value along a plurality of scan lines set in a plurality of scan directions with respect to the face area based on the point cloud data. For each intersection point where a plurality of the scanning lines intersect, a representative value of the roughness feature amount is obtained, and based on the distribution of the representative values for each intersection point, a region dividing unit that divides the face area into divided regions; For each divided region obtained by dividing the face area by the region dividing unit, an observation item evaluate the cutting edge and output the evaluation result is used as Output a divided region evaluation processing unit for performing the evaluation.

Advantages of the Invention

[0008] According to the face evaluation support device according to one aspect of the present invention, the face extraction unit extracts a face area corresponding to the face based on at least one of image data and point cloud data, and the roughness quantification unit quantifies the roughness of the face surface in the face area as a roughness feature amount based on the point cloud data. The region dividing unit divides the face area into divided regions based on the distribution of the roughness feature amount, and the divided region evaluation processing unit evaluates the observation items related to the face for each divided region. Therefore, the region corresponding to the face is divided into divided regions, and the observation items related to the face are evaluated for each divided region. Therefore, the evaluation quality can be improved when evaluating the face.

[0009] Problems, configurations, and effects other than those described above will be clarified in the embodiments for carrying out the invention described later.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. In the following, the range necessary for the description for achieving the object of the present invention is schematically shown, and mainly the range necessary for the description of the corresponding part of the present invention will be described, and the parts where the description is omitted are assumed to be based on known techniques.

[0012] FIG. 1 is an overall view showing an example of the face evaluation support system 1. FIG. 2 is a diagram showing an example of the data processed by the face evaluation support system 1, showing (a) image data and (b) point cloud data.

[0013] The face evaluation support system 1 is a system used to observe and evaluate the state of the face F at the tunnel site. The face evaluation support system 1 mainly includes an image capturing device 2, a three-dimensional shape measuring device 3, a face evaluation support device 4, an operator terminal device 5, and a manager terminal device 6. Each of the devices 2 to 6 is connected to a wired or wireless network 7 so as to be able to transmit and receive various data to and from each other. Note that the number of each of the devices 2 to 6 may be plural, and the configuration of the network 7 is not limited to the example of FIG. 1.

[0014] The image capturing device 2 is a device that captures a target area 100 including the face F of the tunnel, and is composed of, for example, a digital camera having a color image sensor. The image capturing device 2 is operated by, for example, a field worker, and captures the target area 100 under predetermined shooting conditions (shooting position, installation position of the color sample 20, lighting, etc.) to generate color image data in which pixel values for three components of RGB are recorded for each pixel. The color image data is sent to the face evaluation support device 4 via the network 7 or a recording medium or the like.

[0015] The imaging device 2 is used by being installed on the roadbed section with a tripod or the like. However, for example, it may be attached to construction machinery, vehicles, drones, unmanned aerial vehicles (UAVs), flying objects such as balloons, or may be remotely operated. Further, the imaging device 2 may generate grayscale image data instead of color image data. In the present embodiment, the case where the imaging device 2 generates color image data and performs color correction by a color correction unit 110 described later will be described. Hereinafter, the color image data will be abbreviated as "image data".

[0016] In the image data, not only the face F but also the struts and the roadbed section are photographed. Further, when photographing the image data, as shown in FIG. 2(a), in order to photograph with the color sample 20 installed near the front of the face F, the color sample 20 is also photographed in the image data.

[0017] The color sample 20 has a predetermined black-and-white pattern composed of two colors, black and white. For example, it is composed of three rectangular regions (each having approximately the same area) arranged in the left-right direction in the order of black, white (having black edges at the top and bottom), and black. The color sample 20 adopts a point-symmetric or line-symmetric pattern so that it has the same pattern even when the top and bottom are reversed. Further, since the color sample 20 has only a black-and-white pattern, for example, it can be easily produced by printing the pattern of the black part on copy paper or printing paper (A3, B4, A4 size, etc.) for white paper with a printer (even if it does not support color printing) and pasting it on a mount such as a blackboard, a whiteboard, or cardboard.

[0018] The three-dimensional shape measurement device 3 is a device that measures the three-dimensional shape of a target region 100 including the face F of the tunnel, and is composed of, for example, a distance measurement sensor using a laser or ultrasonic wave, a stereo camera, or the like. The three-dimensional shape measurement device 3 measures the target region 100 under predetermined measurement conditions (measurement position, etc.) to generate point cloud data in which the distances (depths) to each position of the target region 100 are recorded with respect to the direction of the tunnel axis (z-axis). The point cloud data is sent to the face evaluation support device 4 via the network 7 or a recording medium or the like.

[0019] The three-dimensional shape measurement device 3 may be either an active method or a passive method as a method for measuring a three-dimensional shape. The image capturing device 2 and the three-dimensional shape measurement device 3 may be integrally configured. For example, they may be configured by an RGB-D (Red Green Blue Depth) camera capable of generating both image data and point cloud data.

[0020] As shown in FIG. 2(b), in the point cloud data, with the axial direction of the tunnel as the z direction, the three-dimensional shape of the target area 100 is recorded by the distance to the face F. At this time, it is not necessary to install the color sample 20. Note that the imaging range by the image data and the measurement range by the point cloud data are preferably in the same range, such as the target area 100, but may be different ranges as long as both ranges correspond to the target area 100 including at least the entire face F. Also, the imaging time of the image data and the measurement time of the point cloud data are preferably in basically the same time zone, but there may be a time difference as long as the state of the face F does not change.

[0021] The face evaluation support device 4 is configured by a general-purpose or dedicated computer. For example, a stationary computer or a portable computer is used. The face evaluation support device 4 acquires image data and point cloud data from the image capturing device 2 and the three-dimensional shape measurement device 3 respectively, and evaluates various observation items regarding the face F by performing various processes on the image data and the point cloud data. The face evaluation support device 4 generates support information based on the evaluation result, and provides the support information to, for example, the operator terminal device 5, the administrator terminal device 6, etc.

[0022] Further, the face evaluation support device 4 includes a face management database 400 that stores the image data, the point cloud data, and the support information in association with each other. A plurality of the above data regarding the face F are registered in the face management database 400, and are managed in an identifiable manner by adding attribute information such as an identifier (ID), a site name, a site location, an observation date and time, etc.

[0023] The operator terminal device 5 is used, for example, by on-site operators. The operator terminal device 5 is composed of a general-purpose or dedicated computer, and for example, a portable computer such as a tablet terminal or a smartphone is used. The operator terminal device 5 accepts various input operations and outputs various information via a display screen or voice by software such as an app or a web browser. Further, the operator terminal device 5 transmits and receives various data to and from the face evaluation support device 4 etc. via the network 7.

[0024] The administrator terminal device 6 is used, for example, by on-site administrators (who may be the same person as the operator). The administrator terminal device 6 is composed of a general-purpose or dedicated computer, and for example, a stationary computer or a portable computer is used. The administrator terminal device 6 accepts various input operations and outputs various information via a display screen or voice by software such as an app or a web browser. Further, the administrator terminal device 6 transmits and receives various data to and from the face evaluation support device 4 etc. via the network 7.

[0025] FIG. 3 is a block diagram showing an example of the face evaluation support device 4. The face evaluation support device 4 includes a storage unit 40 composed of an HDD, an SSD, a memory, etc., a control unit 41 composed of a processor such as a CPU, a GPU, an MPU, etc., an input unit 42 composed of a keyboard, a mouse, etc., a display unit 43 composed of a display, etc., a communication unit 44 which is an interface with the network 7 based on a predetermined communication standard (either wired or wireless is acceptable), an external device interface (I / F) unit 45 which is an interface with external devices such as a printer, a scanner, a USB memory, etc., and a media input / output unit 46 which is an interface with a storage medium such as a CD, a DVD, etc. Note that the input unit 42 and the display unit 43 may be composed of a shared device such as a touch panel.

[0026] The storage unit 40 stores an operating system (OS) which is a basic program, a face evaluation support program 10 for controlling the operation of the face evaluation support device 4, various types of data (such as the face management database 400, etc.) used in the face evaluation support program 10, spreadsheet software for executing spreadsheet processing, document creation software for executing document creation processing, web browser software, and the like.

[0027] Note that the face evaluation support program 10 and various types of data used in the face evaluation support program 10 are basically stored in the storage unit 40, but these programs and data may be acquired from an external storage device via the communication unit 44 or the external device I / F unit 45, or may be acquired from a storage medium via the media input / output unit 46. Also, the various types of data used in the face evaluation support program 10 are preferably created in a format that can be displayed and edited by the spreadsheet software, document creation software, and web browser software.

[0028] By executing the face evaluation support program 10, the control unit 41 functions as an image data acquisition unit 11, a point cloud data acquisition unit 12, a face evaluation unit 13, and a support information output unit 14.

[0029] FIG. 4 is a functional explanatory diagram showing an example of the face evaluation support device 4. Note that the face evaluation support device 4 operates as a main body for implementing the face evaluation support method, and the processing contents performed by each part of the face evaluation support device 4 correspond to each step of the face evaluation support method.

[0030] The image data acquisition unit 11 acquires image data of a target area 100 including the face F of the tunnel. The image data acquisition unit 11 acquires image data from the image capturing device 2 via, for example, the communication unit 44, the external device I / F unit 45, the media input / output unit 46, etc. Further, the image data acquisition unit 11 includes a color correction unit 110 that performs color correction on the color image data. Note that the image data may be those captured in the past and stored in the face management database 400. In that case, the image data acquisition unit 11 may refer to the face management database 400.

[0031] The point cloud data acquisition unit 12 acquires point cloud data obtained by measuring the three-dimensional shape of the target area 100 including the face F of the tunnel. The point cloud data acquisition unit 12 acquires point cloud data from the three-dimensional shape measuring device 3 via, for example, the communication unit 44, the external device I / F unit 45, the media input / output unit 46, etc. Note that the point cloud data may be those measured in the past and stored in the face management database 400. In that case, the point cloud data acquisition unit 12 may refer to the face management database 400.

[0032] The face evaluation unit 13 evaluates the face F based on the image data acquired by the image data acquisition unit 11 and the point cloud data acquired by the point cloud data acquisition unit 12. The image data used by the face evaluation unit 13 is the image data after color correction by the color correction unit 110.

[0033] The face evaluation unit 13 includes, as its configuration, a face extraction unit 130, a roughness quantification unit 131, a region division unit 132, a divided region evaluation processing unit 133, and a face region evaluation processing unit 134. Details of each unit will be described later.

[0034] The divided region evaluation processing unit 133 includes, as its configuration, a crack dominant direction determination unit 133A, a rock group determination unit 133B, a crack interval determination unit 133C, a crack morphology determination unit 133D, and an evaluation division inference unit 133E. Note that any one of the units 133A to 133E may be omitted.

[0035] The cutting edge area evaluation processing unit 134 includes, as its configuration, a first continuity crack detection unit 134A, a second continuity crack detection unit 134B, a third continuity crack detection unit 134C, an artifact detection unit 134D, a first skin peeling risk location prediction unit 134E, a second skin peeling risk location prediction unit 134F, and a skin peeling risk location evaluation unit 134G. Note that any one of the units 134A to 134G may be omitted.

[0036] The support information output unit 14 outputs support information based on the evaluation result of the cutting edge F by the cutting edge evaluation unit 13. The support information is output in a format that can be handled by arbitrary software such as an application or a web browser. Further, as the output destination of the support information, for example, there are a cutting edge management database 400, an operator terminal device 5, a manager terminal device 6, an external device (printer, USB memory), and the like. Note that the support information output unit 14 may output support information in response to the acquisition of image data and point cloud data by the image data acquisition unit 11 and the point cloud data acquisition unit 12, or may output support information, for example, in response to a request from the operator terminal device 5, the manager terminal device 6, or the like. Further, the support information stored in the cutting edge management database 400 may be output, for example, in response to a request from the operator terminal device 5, the manager terminal device 6, or the like.

[0037] Examples of the content of the support information include cutting edge observation sheet information 15 in which the evaluation result is recorded according to a predetermined format, and cutting edge sketch information 16 in which the evaluation result is superimposed and displayed on a display screen for displaying the image data.

[0038] Hereinafter, the details of each part of the cutting edge evaluation support device 4 (each step of the cutting edge evaluation support method) and the evaluation result of the cutting edge F by the cutting edge evaluation support device 4 will be described with reference to FIGS. 5 to 21, respectively.

[0039] (1) Color correction processing by the color correction unit 110 When the image data acquisition unit 110 acquires color image data obtained by photographing a target area 100 including a color sample 20 having a predetermined black-and-white pattern in color together with the cutting edge F as image data, the color correction unit 110 performs color correction on the color image data.

[0040] FIG. 5 is a flowchart showing an example of the color correction process by the color correction unit 110.

[0041] First, in step S101, the color tone of the entire color image is corrected using the white and black pixels of the color sample 20 included in the color image data.

[0042] Next, in step S102, the color image data is converted into a grayscale image, and the brightness of the pixel located in the middle when all the pixels constituting the grayscale image are arranged in ascending order of brightness is obtained as the brightness intermediate value.

[0043] Next, in step S103, all the pixels are classified into a high-brightness pixel group composed of pixels having a brightness higher than the brightness intermediate value and a low-brightness pixel group composed of an image having a brightness equal to or lower than the brightness intermediate value.

[0044] Next, in step S104, for the grayscale image, for example, binarization processing is performed with a threshold value of minimum 110 to maximum 253 to detect the white frame on the outer periphery of the color sample 20. Then, for the grayscale image, in order to remove noise, dilation processing for connecting broken line segments is performed, and contour detection processing is performed. Among the contours obtained as a result, the position of the color sample 20 is detected based on the size, shape, area ratio, etc.

[0045] Next, in step S105, for the pixel group corresponding to the white portion of the color sample 20, the average value of each of the three RGB components is obtained, and the maximum value among the average values is selected. Then, each difference value between the maximum value and each average value is determined as the correction value for the high-brightness pixel group. For example, when the average value of RGB of the pixel group corresponding to the white portion is obtained as "230, 216, 231", "231" is selected as the maximum value, and the correction value for the high-brightness pixel group is determined as "1, 15, 0".

[0046] Next, in step S106, the high-brightness pixel group is color-corrected by adding the correction value for the high-brightness pixel group determined in step S105 to each pixel of the high-brightness pixel group separately for each of the three components. For example, when a specific pixel belonging to the high-brightness pixel group is "123, 234, 135", the pixel after correction by the above correction value is "124, 249, 135".

[0047] Next, in step S107, for the pixel group corresponding to the black portion of the color sample 20, the average value of each of the three components is obtained, and the minimum value among the average values is selected. Then, each difference value between the minimum value and each average value is determined as the correction value for the low-brightness pixel group. For example, when the average value of RGB of the pixel group corresponding to the black portion is obtained as "24, 4, 6", "4" is selected as the minimum value, and the correction value for the low-brightness pixel group is determined as "20, 0, 2".

[0048] Next, in step S108, the low-brightness pixel group is color-corrected by subtracting the correction value for the low-brightness pixel group determined in step S107 from each pixel of the low-brightness pixel group separately for each of the three components. For example, when a specific pixel belonging to the low-brightness pixel group is "43, 30, 35", the pixel after correction by the above correction value is "23, 30, 33".

[0049] As described above, by performing the series of color correction processes shown in FIG. 5, the color correction unit 110 corrects the color deviation in the image data. Therefore, even when various conditions such as the type and illuminance of lighting, the specifications and settings of the image capturing device 2, etc. are different, the face F can be appropriately evaluated based on the corrected image data.

[0050] (2) Face extraction process by the face extraction unit 130 The face extraction unit 130 extracts a face region 101 corresponding to the face F from the target region 100 based on at least one of the image data and the point cloud data. Examples of the method for extracting the face region 101 from the target region 100 include a first face extraction process of analyzing and extracting the point cloud data, a second face extraction process of extracting from the image data using a machine learning model, and a third face extraction process of using both the first face extraction process and the first face extraction process in combination.

[0051] FIG. 6 is an explanatory diagram showing an example of the first face extraction process of extracting the face region 101 from the point cloud data. In the first face extraction process, when the point cloud data is measured in a coordinate system with the tunnel axis as the z-axis (the coordinate value decreases as it approaches the face side), as shown in FIG. 6(a), the face extraction unit 130 extracts the face region 101 based on the z-axis coordinate value of each point. For example, the progress distance L of the face surface in one blasting of tunnel construction is about 1.5 m with respect to the z-axis, and the unevenness of the face surface with respect to the z-axis is within the progress distance L. Also, the support is constructed at a position farther from the face surface than the progress distance L.

[0052] Therefore, as shown in FIG. 6(b), the face extraction unit 130 extracts the range from the minimum value of the z-axis (the farthest point from the measurement position) of the point cloud data acquired by the point cloud data acquisition unit 12 to the progress distance L as the face region 101. Also, as shown in FIG. 6(c), the face extraction unit 130 performs a masking process on the image data acquired by the image data acquisition unit 11 for the range of coordinate values equal to or greater than the coordinate value obtained by adding the progress distance L to the minimum value of the z-axis of the point cloud data, and extracts the face region 101 as the region of the support and the roadbed part.

[0053] In addition, as another face excavation area extraction process using point cloud data, the face excavation area extraction unit 130 may extract the face excavation area 101 by utilizing the characteristic that in the face excavation area 101, the component along the z-axis is dominant in each normal vector of the triangular mesh composed of the point cloud data, while in the rib and roadbed parts, the component orthogonal to the z-axis component is dominant.

[0054] FIG. 7 is an explanatory diagram showing an example of a second face excavation area extraction process for extracting the face excavation area 101 from image data. The machine learning model used in the second face excavation area extraction process is configured by, for example, a convolutional neural network (CNN), and machine learning is performed using teacher data including learning image data in which the face excavation was photographed and the correct label assigned to the face excavation area 101 corresponding to the face excavation F. That is, as shown in FIG. 7, in the learning phase of machine learning, the extraction result of the face excavation area 101 output by inputting the learning image data of the teacher data into the machine learning model is compared with the correct label of the teacher data, and machine learning is performed by adjusting the weight parameters of the convolutional neural network based on the comparison result. Note that, as shown in FIG. 7, the face excavation area extraction unit 130 may extract not only the face excavation area 101 but also the rib area corresponding to the rib and the roadbed area corresponding to the roadbed part.

[0055] The face excavation area extraction unit 130 is the main body that executes the inference phase of machine learning, and extracts the face excavation area 101 from the target area 100 by inputting the image data acquired by the image data acquisition unit 11 into the learned machine learning model.

[0056] As described above, the face excavation area 101 is extracted from the target area 100 by the face excavation area extraction process by the face excavation area extraction unit 130.

[0057] (3) Roughness quantification process by the roughness quantification unit 131 The roughness quantification unit 131 quantifies the roughness (convexity and concavity) of the face of the excavation face in the excavation face area 101 extracted by the excavation face extraction unit 130 along a plurality of scan lines set in a plurality of scan directions with respect to the excavation face area 101 based on the point cloud data as a roughness feature amount. The roughness quantification unit 131 quantifies, as the roughness feature amount, the amplitude value and the wavelength of the roughness approximation model when the point cloud data corresponding to the excavation face area 101 is approximated by a periodic function along a plurality of scan lines.

[0058] FIG. 8 is an explanatory diagram showing an example of the roughness quantification process by the roughness quantification unit 131. The roughness feature amount is a parameter obtained by quantifying the roughness in the z-axis direction on the excavation face based on the point cloud data. As a plurality of scan directions, as shown in FIG. 8(a), for example, the vertical and horizontal directions (x-axis, y-axis) and the diagonal direction on the excavation face are each set, and approximated along a plurality of scan lines i arranged at predetermined intervals d1 and d2.

[0059] The roughness approximated along the scan line i is set from the z-axis coordinate value in the point cloud data located within a certain distance a from the scan line i, as shown in FIG. 8(b). The height difference h from the reference line (plane) on the scan line i i (see FIG. 8(c)) is obtained by using the coordinate value of the point cloud p j included in the point cloud data and the distance r from the scan line i to the point cloud p j using the following equation [Equation 1]. Here, p j,z is the z-axis coordinate value of the point cloud p j , and z 0 is the value of the reference line (plane).

[0060]

Equation

[0061] The roughness of the excavation face is approximated to a roughness approximation model for each scan line i, paying attention to the magnitude of the height difference and the periodicity of the height. The roughness approximation model uses a periodic function to represent the height difference H(t) from the reference plane at the position t of the scan line ii is modeled (approximated) by the following [Equation 2].

[0062]

Equation

[0063] Here, the wave number k j corresponds to a value that is the reciprocal of the wavelength λ j which is the distance of the repetition of the height difference. The coefficients A j and B j are characteristic quantities and can be obtained, for example, by using the least squares method or sparse modeling with respect to the wave number k j . The above roughness approximation model is a modeling that includes the characterization by the average value of the height difference by the superposition sum of j periodic functions. On the other hand, the wave number k j may be set by using the crack interval (frequency) which is an observation item of the face F.

[0064] The roughness quantification unit 131 quantifies, as the roughness characteristic quantity for each scanning line i, for example, the amplitude value and the wavelength in the roughness approximation model. The amplitude value C j of the roughness approximation model is obtained from the coefficients A j and B j by the following [Equation 3]. Also, the wavelength λ j of the roughness approximation model is determined by the wave number k j at which the amplitude value C j (= 1 / λ j ) becomes the maximum value.

[0065]

Equation

[0066] In the above manner, by the roughness quantification process by the roughness quantification unit 131, the roughness of the face in the face area 101 is quantified as the roughness characteristic quantity.

[0067] (4) Region division process by the region division unit 132 For each intersection point where a plurality of scanning lines intersect, the region dividing unit 132 obtains a representative value of the roughness feature quantity digitized by the roughness digitizing unit 131, and divides the cut-off region 101 extracted by the cut-off extraction unit 130 into divided regions 102 based on the distribution of the representative values for each intersection point.

[0068] FIG. 9 is an explanatory diagram showing an example of the region division process by the region dividing unit 132. When dividing the cut-off region 101 into divided regions 102, the region dividing unit 132 obtains a representative value of the roughness feature quantity at each intersection point P where a plurality of scanning lines intersect. For example, as shown in FIG. 9(a), at the intersection point P where four scanning lines in the vertical direction and the diagonal direction intersect i,j,k,l If the amplitude value C and the wavelength λ are digitized as the roughness feature quantities at the intersection point P, the representative value of the roughness feature is obtained by, for example, the average (simple average or weighted average) or vector sum of the roughness feature quantities on the four scanning lines, that is, the amplitude value C and the wavelength λ.

[0069] Next, as shown in FIG. 9(b), the region dividing unit 132 compares the representative value (amplitude value C and wavelength λ) for each intersection point P with the reference value C of the amplitude value C 0 and the reference value λ of the wavelength λ 0 to determine into which of the four feature classifications Ra, Rb, Rc, Rd each intersection point P is classified. Note that by setting a plurality of reference values for the amplitude value C or setting a plurality of reference values for the wavelength λ, the feature classification may be classified into feature classifications other than the above.

[0070] Then, the region dividing unit 132 divides the cut-off region 101 into divided regions 102 based on the distribution of the feature classifications Ra, Rb, Rc, Rd into which each intersection point P is classified. FIG. 9(c) shows, for example, a case where it is divided into two divided regions 102, that is, a divided region 102 (upper left part) which is a set of intersection points P classified into the feature classification Ra and a divided region 102 (lower right part) which is a set of intersection points P classified into the feature classification Rd.

[0071] As described above, by the region division process performed by the region division unit 132, the face advance region 101 is divided into division regions 102 based on the distribution of the roughness feature amount.

[0072] (5) Division region evaluation process by the division region evaluation unit 133 The division region evaluation unit 133 evaluates, as evaluation results, observation items regarding the face advance F for each of the division regions 102 into which the face advance region 101 is divided by the region division unit 132. The division region evaluation unit 133 evaluates various observation items for each of the division regions 102 based on the image data and the point cloud data corresponding to the division region 102. Therefore, as shown in FIG. 4, the division region evaluation unit 133 includes a crack dominant direction determination unit 133A, a rock group determination unit 133B, a crack interval determination unit 133C, a crack form determination unit 133D, and an evaluation division inference unit 133E. Details of each unit (each process) 133A to 133E will be described below.

[0073] (5-1) Crack dominant direction determination unit 133A FIG. 10 is an explanatory diagram showing an example of the determination process of the crack dominant direction by the crack dominant direction determination unit 133A. The crack dominant direction determination unit 133A projects each normal vector of the triangular mesh composed of the point cloud data corresponding to the face advance region 101 onto a stereonet by polar projection for each of the division regions 102, and determines, for each of the division regions 102, the crack dominant direction indicating the strike inclination of the dominant cracks on the face advance surface based on the distribution of the plots polar-projected on the stereonet.

[0074] On the face advance surface, since rock blocks are often separated and excavated along existing cracks, the unevenness of the face advance surface is likely to appear along the cracks. Therefore, in order to determine the crack dominant direction in each division region 102, as shown in FIG. 10(a), the crack dominant direction determination unit 133A uses the triangular mesh composed of the point cloud data corresponding to the division region 102, and as shown in FIGS. 10(b) and 10(c), projects the normal vector of each triangle included in the triangular mesh onto a stereonet (equal-area projection net) by polar projection. In the polar projection onto the stereonet, the arrow base end sides of the respective normal vectors are aligned with the center of the stereonet and projected in the direction that coincides with the tunnel axis (z-axis).

[0075] Then, as shown in FIG. 10(c), the crack-dominant direction determination unit 133A creates a density contour map based on the distribution of the plots corresponding to the arrow tip sides of the respective normal vectors projected onto the stereonet, and in the density contour map, determines the range where the density is higher than a predetermined reference value as the crack-dominant direction. At this time, the crack-dominant direction determination unit 133A determines the crack-dominant direction for each partition region, but there may be a case where a plurality of crack-dominant directions are determined to be included in one partition region.

[0076] Note that when the point cloud data is irregularly distributed on the cross-section plane, it is assumed that the number of plots projected onto the stereonet does not correspond to the area of the triangles on the cross-section plane. In that case, the crack-dominant direction determination unit 133A may perform weighting by the area of the triangles constituting the triangular mesh and plot on the stereonet, and determine the crack-dominant direction in a state reflecting the area on the cross-section plane.

[0077] (5-2) Rock group determination unit 133B FIG. 11 is an explanatory diagram showing an example of the determination process of the rock group by the rock group determination unit 133B. The rock group determination unit 133B determines the rock group indicating the type of rock constituting the cross-section plane for each divided region 102 based on the amplitude value and wavelength of the roughness approximation model digitized by the roughness digitization unit 131.

[0078] As rock groups, for example, by combining any of the weathering classifications of massive and layered, and any of the strength classifications of hard, medium-hard, and soft, they are classified into multiple rock types. For example, in the rock group Ga of hard rock - massive (e.g., granite, etc.), the amplitude value indicating the unevenness of the cracks is large, and the wavelength indicating the interval between the cracks tends to be medium to long. In the rock group Ge of soft rock - layered (e.g., Tertiary shale, phyllite, etc.), since fine bedding and schistosity are present, the amplitude value becomes small and the wavelength also becomes small due to the unevenness along the bedding and schistosity. In the rock group Gc of soft rock - massive (e.g., poorly consolidated tuff breccia, etc.), since the face of the excavation face finishes smoothly, there is almost no unevenness, the amplitude value is small, and the wavelength is long. The rock group determination unit 133B determines the rock group of each division region 102 based on, for example, the amplitude value and wavelength at each intersection point P by paying attention to the tendencies and characteristics of each rock group as described above. Note that the rock group determination unit 133B may also determine the rock group of each division region 102 by referring not only to the roughness feature amount but also to the evaluation classification for other observation items (e.g., weathering alteration, compressive strength, etc.).

[0079] (5-3) Crack interval determination unit 133C FIG. 12 is an explanatory diagram showing an example of the determination process of the crack interval by the crack interval determination unit 133C. The crack interval determination unit 133C performs an edge detection process for detecting edges on the image data corresponding to the division region 102, and refers to the correspondence information in which the relationship between the edge density and the crack interval is defined based on the edge density of the edges detected by the edge detection process, thereby determining the crack interval for each division region 102.

[0080] The crack interval determination unit 133C detects edges from the image data by means of an arbitrary edge detection process such as the Canny method, etc., and calculates the edge density. Then, the crack interval determination unit 133C refers to the correspondence information defined for each type of rock, for example, regarding the relationship between the edge density and the crack interval, and specifies the crack interval associated with the edge density calculated as described above in the correspondence information of the rock group determined by the rock group determination unit 133B, thereby determining the crack interval.

[0081] Although the correspondence information has been described as being defined in a plurality for each type of rock, it is not limited to this. For example, it may be defined in a plurality for each other observation item, or there may be one defined correspondence information.

[0082] (5-4) Crack form determination unit 133D FIG. 13 is an explanatory diagram showing an example of the determination process of the crack form by the crack form determination unit 133D. The crack form determination unit 133D determines the crack form for each division area 102 based on the characteristics of the wavelength corresponding to the scanning direction of the scanning line when the roughness quantification unit 131 approximates the roughness feature amount to the roughness approximation model.

[0083] The roughness feature amount digitized along the scanning line by the roughness quantification unit 131 can be used for estimating the crack form. In the scanning lines set in the vertical, horizontal, and diagonal directions, when the wavelength λ, which is the repeated distance of the height difference, is generally constant regardless of the direction of the scanning line (within a predetermined reference value range), the crack form determination unit 133D determines that the crack form of that section area is "random square" or "earth and sand shape". On the other hand, when the wavelength λ varies greatly depending on the direction of the scanning line (outside the predetermined reference value range), the crack form determination unit 133D determines that the crack form of that section area is "columnar" or "layered".

[0084] (5-5) Evaluation category inference unit 133E FIG. 14 is an explanatory diagram showing an example of the inference process of the evaluation category for the observation items by the evaluation category inference unit 133E. The evaluation category inference unit 133E infers the evaluation category for each division area 102 by inputting the image data corresponding to the division area 102 into the machine learning model. The machine learning model used by the evaluation category inference unit 133E is one obtained by performing machine learning using teacher data including learning image data in which the face is photographed and a correct label to which an evaluation category regarding the observation item is assigned for the face F.

[0085] Examples of the observation items to be inferred include, for example, compressive strength, weathering alteration, crack interval, crack form, crack state, rock group, etc. The evaluation category inference unit 133E may estimate the evaluation category for at least one of these observation items. Note that each of the observation items of crack interval, crack form, crack state, and rock group can be determined by the above-described crack interval determination unit 133C, crack form determination unit 133D, and rock group determination unit 133B. However, the evaluation category inference unit 133E may infer each observation item instead of these, or may infer each observation item in combination with these.

[0086] The machine learning model is preferably configured by, for example, a convolutional neural network and created for each observation item as shown in FIG. 14. For example, the machine learning model regarding compressive strength is one obtained by performing machine learning using teacher data including learning image data in which the face is photographed and a correct label to which an evaluation category regarding compressive strength is assigned for the face F. The evaluation category inference unit 133E infers the evaluation category regarding compressive strength for the division area by inputting the image data corresponding to the division area into the learned machine learning model. At this time, the image data may be input into the machine learning model in a state of being divided into images of small areas having a predetermined size. In that case, the evaluation category inference unit 133E may infer the evaluation category regarding compressive strength in units of small areas included in the division area. Since the machine learning models regarding other observation items are configured in the same manner as above, the description thereof is omitted.

[0087] (6) Evaluation process of the cutting edge area by the cutting edge area evaluation unit 134 The cutting edge area evaluation unit 134 evaluates the observation items regarding the cutting edge F as evaluation results for the cutting edge area 101 extracted by the cutting edge extraction unit 130. As shown in FIG. 4, the cutting edge area evaluation unit 134 includes a first continuity crack detection unit 134A, a second continuity crack detection unit 134B, a third continuity crack detection unit 134C, an artifact detection unit 134D, a first peeling risk location prediction unit 134E, a second peeling risk location prediction unit 134F, and a peeling risk location evaluation unit 134G in order to evaluate various observation items. The details of each unit (each process) 134A to 133G will be described below.

[0088] (6-1) First continuity crack detection unit 134A FIG. 15 is an explanatory diagram showing an example of the determination process of the continuity crack by the first continuity crack detection unit 134A. The first continuity crack detection unit detects the continuity crack continuously formed on the cutting edge surface based on the point cloud data. Specifically, the first continuity crack detection unit extracts candidates for the edges that form part of the continuity crack based on the angle formed by the normal vectors of two triangles including a common side from each side of the triangular mesh composed of the point cloud data corresponding to the cutting edge area 101, and detects the continuity crack existing in the cutting edge area 101 by determining the continuity between the candidates.

[0089] In the point cloud data, the convex and concave shape of the cutting edge surface is represented by the triangular mesh, but each side of each triangle is a mixture of the edges representing the convex and concave of the cutting edge surface and the delimitation lines for convenience. Therefore, the continuity crack is identified by extracting the sides of the triangles representing the convex and concave edges of the cutting edge surface and considering the continuity of the extracted sides.

[0090] The determination of whether the common side ab of two adjacent triangles A and B represents an uneven edge (edge determination process) is performed based on the angle formed by the normal vectors nA and nB of triangles A and B, as shown in FIG. 15. For example, when the angle θ formed by the two normal vectors nA and nB is within a predetermined threshold range with respect to 0° or 180°, since triangles A and B are considered to be on the same plane, it is determined that the side ab is not an uneven edge. Otherwise, it is determined that the side ab is an uneven edge. At that time, the angle θ formed by the normal vectors nA and nB may be obtained and determined, or the determination may be made by the inner product of the normal vectors nA and nB.

[0091] The first continuity crack detection unit performs the above-described edge determination process for all sides shared by two adjacent triangles, and extracts candidates for edges that form part of the continuity crack. Then, for the continuity between the edge candidates, for each of the extracted edge candidates, a determination of whether there is continuity between the edge candidates (continuity determination process) is performed, and the continuous edges are grouped by connecting them as a polyline. In the continuity determination process, the two edges to be determined are used as line segment vectors, and when the angle formed by the two line segment vectors is equal to or less than a predetermined threshold, it is determined that they are continuous. Otherwise, it is determined that they are disconnected. The first continuity crack detection unit deletes polylines shorter than a predetermined threshold among the grouped polylines, and further selects those with higher straightness to detect the continuity crack.

[0092] (6-2) Second Continuity Crack Detection Unit 134B FIG. 16 is an explanatory diagram showing an example of the determination process of the continuous crack by the second continuous crack detection unit 134B. The second continuous crack detection unit detects a continuous crack continuously formed on the face of the blast hole based on the image data. Specifically, the second continuous crack detection unit inputs the image data corresponding to the blast hole area 101 into the machine learning model to detect the continuous crack existing in the blast hole area 101. The machine learning model used by the second continuous crack detection unit is configured by, for example, a convolutional neural network, and machine learning is performed by teacher data including the learning image data when the blast hole was photographed and the correct label given to the continuous crack formed in the blast hole F.

[0093] (6-3) The third continuous crack detection unit 134C FIG. 17 is an explanatory diagram showing an example of the determination process of the continuous crack by the third continuous crack detection unit 134C. The third continuous crack detection unit detects a continuous crack continuously formed on the face of the blast hole based on the image data and the point cloud data. Specifically, the third continuous crack detection unit performs an edge detection process for detecting an edge on the image data corresponding to the blast hole area 101, and fits the edge detected by the edge detection process to the crack dominant direction of each divided area 102 determined by the crack dominant direction determination unit 133A with respect to the point cloud data, thereby detecting the continuous crack existing in the blast hole area 101.

[0094] The third continuous crack detection unit detects an edge from the image data by an arbitrary edge detection process such as the Canny method. Then, the third continuous crack detection unit performs a fitting process of fitting the detected edge to the crack dominant direction of each divided area 102 by, for example, the least squares method or sparse modeling, thereby detecting a continuous crack.

[0095] (6-4) Artifact detection unit 134D FIG. 18 is an explanatory diagram showing an example of the detection process of an artifact by the artifact detection unit 134D. The artifact detection unit 134D detects an artifact existing in the target area 100 based on the image data corresponding to the target area 100. Specifically, the artifact detection unit 134D divides the image data corresponding to the target area 100 into the image data corresponding to the face area 101 and the image data corresponding to areas other than the face area 101 (for example, the support area and the roadbed area), and inputs these image data into a machine learning model respectively, thereby detecting the artifact existing in the face area 101 and the artifact existing outside the face area 101 respectively. The machine learning model used by the artifact detection unit 134D is obtained by performing machine learning with teacher data including the learning image data in which the face is photographed and the correct label assigned to the artifact installed in the face or its surroundings.

[0096] Examples of the artifacts to be detected include, for example, steel supports, rock bolts, mirror bolts, forepoling, blackboards, people, shadows of people and machines, etc. The artifact detection unit 134D may detect at least one of these artifacts. The artifacts to be detected in the face area 101 are, for example, mirror bolts, forepoling, blackboards, people, shadows of people and machines, etc., and the artifacts to be detected in the support area and the roadbed area outside the face area 101 are, for example, rock bolts, steel supports, blackboards, people, shadows of people and machines, etc., but are not limited to these examples. FIG. 18 illustrates a case where the artifact detection unit 134D detects a plurality of forepolings for the face area 101 and a plurality of rock bolts for the support area.

[0097] The machine learning model is composed of, for example, a convolutional neural network. The machine learning model for detecting an artifact from the image data corresponding to the face area 101 and the machine learning model for detecting an artifact from the image data corresponding to areas other than the face area 101 may be created as a common machine learning model or as separate machine learning models. Also, the machine learning model may be created for each type of artifact to be detected. Further, the image data may be input to the machine learning model in a state where it is divided into images of small areas having a predetermined size. In that case, the artifact detection unit 134D may detect artifacts in units of small areas included in the target area 100.

[0098] (6-5) First skin peeling risk location prediction unit 134E FIG. 19 is an explanatory diagram showing an example of the prediction process of the skin peeling risk location. The first skin peeling risk location prediction unit 134E predicts a skin peeling risk location indicating a location where there is a risk of skin peeling in the face F based on the point cloud data. Specifically, as shown in FIG. 19, the first skin peeling risk location prediction unit 134E projects the crack dominant direction of each divided area 102 determined by the crack dominant direction determination unit 133A onto a great circle of a stereonet. When a closed polygon is formed by the crack dominant directions extending in at least three directions, the range of the face corresponding to the closed polygon is predicted as the skin peeling risk location.

[0099] In the first skin peeling risk location prediction unit 134E, the crack dominant direction of each divided area 102 determined by the crack dominant direction determination unit 133A for the point cloud data is geometrically utilized to predict the skin peeling risk location. For example, when the crack dominant direction is projected as a great circle on a stereonet and a spherical triangle is formed on the stereonet, the area corresponding to the spherical triangle is predicted as a block that can come out of the face plane.

[0100] (6-6) Second skin peeling risk location prediction unit 134F The second skin peeling risk location prediction unit 134F predicts a skin peeling risk location indicating a location where there is a risk of skin peeling in the cut surface F based on the image data. Specifically, as shown in FIG. 19, the second skin peeling risk location prediction unit 134F inputs the image data corresponding to the cut surface area 101 into a machine learning model that has been machine-learned with teacher data including the learning image data in which the cut surface was photographed and the correct label assigned to the skin peeling risk location where there is a risk of skin peeling in the cut surface F, thereby predicting the skin peeling risk location existing in the cut surface area 101.

[0101] The machine learning model is composed of, for example, a convolutional neural network. The image data may be input into the machine learning model in a state of being divided into small area images having a predetermined size. In that case, the second skin peeling risk location prediction unit 134F may infer whether there is a risk of skin peeling in units of small areas included in the cut surface area 101.

[0102] (6-7) Skin peeling risk location evaluation unit 134G As shown in FIG. 19, the skin peeling risk location evaluation unit 134G predicts a skin peeling risk location based on the prediction result by the first skin peeling risk location prediction unit 134E and the prediction result by the second skin peeling risk location prediction unit 134F. Specifically, the skin peeling risk location evaluation unit 134G compares the skin peeling risk location predicted by the first skin peeling risk location prediction unit 134E with the skin peeling risk location predicted by the second skin peeling risk location prediction unit 134F, and determines that the risk is high for the skin peeling risk location with a high degree of coincidence as a result of the comparison.

[0103] Note that the peeling risk location evaluation unit 134G may predict the peeling risk location in consideration of other observation items. For example, when it is detected by the artificial object detection unit 134D that, as an artificial object, for example, a mirror bolt is installed, the peeling risk location evaluation unit 134G may determine that the risk of peeling is high. Further, when either one of the first peeling risk location prediction unit 134E and the second peeling risk location prediction unit 134F is omitted, the peeling risk location evaluation unit 134G may output the peeling risk location predicted by the other peeling risk location evaluation unit as it is, or the peeling risk location evaluation unit 134G itself may be omitted.

[0104] (7) Support information output process by the support information output unit 14 As described above, the support information output unit 14 generates, as support information, the face sheet observation sheet information 15, the face sheet sketch information 16, etc. based on the evaluation results of the face sheet F for each observation item by the face sheet evaluation unit 13, and outputs them in an output form such as data storage output, screen display output, print output, etc.

[0105] FIG. 20 is a diagram showing a face sheet observation sheet 150 which is a print output example of the face sheet observation sheet information 15. The face sheet observation sheet 150 records the evaluation results of the face sheet F for each observation item according to a predetermined format. In the example of FIG. 20, the face sheet observation sheet 150 includes a face sheet information column 151 in which information about the face sheet F is recorded, and an evaluation classification column 152 in which the evaluation results of the face sheet F are recorded.

[0106] In the face sheet information column 151, for example, the site name, site location, observation date and time, etc. are recorded. In the evaluation classification column 152, for example, the evaluation classifications for compressive strength, weathering deterioration, crack interval, crack state, strike inclination (predominant crack direction), etc. are recorded.

[0107] In the evaluation classification column 152 shown in FIG. 20, the evaluation classifications for each region when the face F is divided into three regions: the top edge, the left shoulder, and the right shoulder are recorded respectively. This is obtained by weighting and adding the evaluation results evaluated for each observation item for each divided region 102 by the face evaluation unit 13 (particularly the divided region evaluation processing unit 133) based on the area ratio where each divided region 102 overlaps with the three regions: the top edge, the left shoulder, and the right shoulder, and converting them into the evaluation results for the top edge, the left shoulder, and the right shoulder.

[0108] Note that the support information output unit 14 may output the face observation sheet information 15 recording the evaluation results for each of the three regions: the top edge, the left shoulder, and the right shoulder, instead of or in addition to the face observation sheet information 15 recording the evaluation results for each divided region 102. Also, as the format of the face observation sheet information 15, the observation items included in the face observation sheet information 15, the layout of the evaluation results, the output format of the evaluation results (point format, table format, graph format, image overlay format, etc.) are not limited to the example in FIG. 20 and may be changed as appropriate, for example, it may be changeable by the administrator or the operator.

[0109] FIG. 21 is a diagram showing a face sketch screen 160 which is an example of the screen display of the face sketch information 16. The face sketch screen 160 superimposes and displays the evaluation results of the face F for each observation item on the display screen for displaying the image data. In the example of FIG. 21, the face sketch screen 160 includes a face information column 161 for displaying information about the face F, an image display area 162 for displaying the image data of the face F, a display item selection column 163 for selecting the object to be displayed within the image display area 162 from among the face region 101, the divided region 102, and various observation items, and a display marker 164 for displaying the evaluation results of the face F within the image display area 162 according to the selected item in the display item selection column 163.

[0110] In the face information column 161, for example, the site name, site location, observation date and time, etc. are displayed. In the image display area 162, the image data acquired by the image data acquisition unit 11 is displayed. The image display area 162 is configured to be able to change the magnification and reduction of the image, and is also configured to be able to accept a sketch operation on the screen by an operator or a manager. Note that in the image display area 162, an image before color correction by the color correction unit 110 may be displayed, or an image after color correction may be displayed.

[0111] The display item selection column 163 has, as options for observation items, for example, strike inclination (dominant direction of cracks), rock group, compressive strength, weathering alteration, crack interval, crack form, crack state, continuous crack, artificial object, peeling risk location, etc.

[0112] The display marker 164 draws the outer peripheries of the face area 101 and the division area 102 with a frame line, or displays the evaluation results of each observation item respectively. At that time, the display marker 164 may display the evaluation results of each observation item so as to overlap at the corresponding positions on the image, or may display them in such a way that the correspondence relationship can be understood by a leader line or the like. Also, the display form (color, pattern, etc.) of the display marker 164 is not limited to the example of FIG. 21, and may be appropriately changed, for example, it may be changeable by a manager or an operator.

[0113] As described above, according to the face evaluation support device 4 according to the present embodiment, the face extraction unit 130 extracts the face area 101 corresponding to the face F based on at least one of the image data and the point cloud data, the roughness quantification unit 131 quantifies the roughness of the face surface in the face area 101 as a roughness feature amount based on the point cloud data, the area division unit 132 divides the face area 101 into division areas 102 based on the distribution of the roughness feature amount, and the division area evaluation processing unit 133 evaluates the observation items related to the face F for each division area 102. Therefore, the face area 101 corresponding to the face F is divided into division areas 102, and the observation items related to the face F are evaluated for each division area 102. Therefore, the evaluation quality can be improved when evaluating the face F.

[0114] In addition, the section area evaluation processing unit 133 can evaluate each observation item for each section area 102 by including each of the units 133A to 133E as described above. Further, the face area evaluation processing unit 134 can evaluate the entire face area 101 for each observation item by including each of the units 134A to 134G as described above.

[0115] (Other embodiments) The present invention is not limited to the above-described embodiments, and various modifications can be made and implemented without departing from the gist of the present invention. And all of them are included in the technical idea of the present invention.

[0116] In the above embodiment, the face evaluation support program 10 has been described as being stored in the storage unit 40. In contrast, the face evaluation support program 10 may be provided by being recorded on a computer-readable storage medium such as a CD-ROM or a DVD in an installable format or an executable format file. Further, the face evaluation support program 10 may be provided by being downloaded from an external device via the network 7. Also, various machine learning models used in the face evaluation unit 13 may be stored in the storage unit 40 or may be stored in an external device and accessed via the network 7.

[0117] In the above embodiment, the case where a convolutional neural network is adopted as a specific method of machine learning using a machine learning model has been described. However, the machine learning model may adopt any other machine learning method (including not only supervised learning but also unsupervised learning and reinforcement learning). Examples of other machine learning methods include tree types such as decision trees and regression trees, ensemble learning such as bagging and boosting, other neural network types (including deep learning) such as recurrent neural networks, hierarchical clustering, non-hierarchical clustering, clustering types such as the k-nearest neighbor method and the k-means method, multivariate analysis such as principal component analysis, factor analysis, and logistic regression, and support vector machines.

Description of Symbols

[0118] 1…Face evaluation support system, 2…Image capturing device, 3…Three-dimensional shape measurement device, 4…Face evaluation support device, 5…Operator terminal device, 6…Administrator terminal device, 7…Network, 10…Face evaluation support program, 11…Image data acquisition unit, 12…Point cloud data acquisition unit, 13…Face evaluation unit, 14…Support information output unit, 15…Face observation sheet information, 16…Face sketch information, 20…Color sample, 40…Memory unit, 41…Control unit, 42…Input unit, 43…Display unit, 44…Communication unit, 45…External device I / F unit, 46…Media input / output unit, 100…Target area, 101…Face area, 102…Division area, 110…Color correction unit, 130…Face extraction unit, 131…Roughness quantification unit, 132…Area division unit, 133…Division area evaluation processing unit, 133A…Crack dominant direction determination unit, 133B…Rock group determination unit, 133C…Crack interval determination unit, 133D…Crack morphology determination unit, 133E…Evaluation division inference unit, 134…Face area evaluation processing unit, 134A…First continuous crack detection unit, 134B…Second continuous crack detection unit, 134C…Third continuous crack detection unit, 134D…Artifact detection unit, 134E…First peeling risk point prediction unit, 134F…Second peeling risk point prediction unit, 134G…Peeling risk point evaluation unit, 150…Face observation sheet, 151…Face information column, 152…Evaluation division column, 160…Face sketch screen, 161…Face information column, 162…Image display area, 163…Display item selection column, 164…Display marker, 400…Face management database

Claims

1. an image data acquisition unit that acquires image data of a target area including a tunnel face; a point cloud data acquisition unit that acquires point cloud data obtained by measuring a three-dimensional shape of the target area; A face evaluation unit that evaluates the face based on the image data and the point cloud data; A support information output unit that outputs support information based on the evaluation result by the face evaluation unit, The face evaluation unit is A face extraction unit that extracts a face area corresponding to the face from the target area based on at least one of the image data and the point cloud data; a roughness quantification unit that quantifies the roughness of the face in the face region as a roughness feature amount along a plurality of scanning lines set in a plurality of scanning directions with respect to the face region based on the point cloud data; a region division unit that calculates a representative value of the roughness feature amount for each intersection point where a plurality of the scanning lines intersect, and divides the face region into divided regions based on a distribution of the representative value for each intersection point; A divided area evaluation processing unit is provided for evaluating the face for each divided area into which the face area is divided by the area division unit, and outputting the evaluation result as the evaluation result. Face evaluation support device.

2. The segmented area evaluation processing unit includes: Each normal vector of a triangular network composed of the point cloud data corresponding to the face region is polar projected onto a stereo net for each of the divided regions; A fracture dominant direction determination unit that determines a fracture dominant direction indicating a strike and inclination of a fracture dominant on the face for each of the divided regions based on a distribution of plots polar-projected onto the stereonet; The face evaluation support device according to claim 1.

3. The roughness quantifying unit is As the roughness feature amount, an amplitude value and a wavelength of a roughness approximation model obtained by approximating the point cloud data corresponding to the face region by a periodic function along a plurality of the scanning lines are quantified; The segmented area evaluation processing unit includes: A rock group determination unit that determines a rock group indicating a type of rock constituting the face for each divided area based on the amplitude value and the wavelength. The face evaluation support device according to claim 1 or 2.

4. The segmented area evaluation processing unit includes: performing an edge detection process for detecting edges in the image data corresponding to the divided region; a crack spacing determination unit that determines the crack spacing for each of the divided regions by referring to correspondence information that defines a relationship between the edge density and the crack spacing based on the edge density of the edges detected by the edge detection process; The face evaluation support device according to any one of claims 1 to 3.

5. The roughness quantifying unit is As the roughness feature amount, a wavelength of a roughness approximation model obtained by approximating the point cloud data corresponding to the face region by a periodic function along a plurality of the scanning lines is quantified; The segmented area evaluation processing unit includes: a crack shape determining unit that determines a crack shape for each of the divided regions based on the wavelength characteristics according to the scanning direction of the scanning line when the roughness approximation model is approximated, The face evaluation support device according to any one of claims 1 to 4.

6. The segmented area evaluation processing unit includes: The image data corresponding to the divided area is input to a machine learning model in which machine learning is performed using training data including learning image data of the face and a correct answer label to which an evaluation category related to the observation item is assigned for the face, thereby inferring the evaluation category for each divided area. The face evaluation support device according to any one of claims 1 to 5.

7. The face evaluation unit is a first continuous crack detection unit that detects continuous cracks formed continuously on the face of the tunneling tunnel based on the point cloud data; The first continuity crack detection unit includes: extracting edge candidates constituting a part of the continuous crack from each side of a triangular net composed of the point cloud data corresponding to the face region based on an angle formed by normal vectors of two triangles including a common side, and detecting the continuous crack present in the face region by determining the continuity between the candidates; The face evaluation support device according to any one of claims 1 to 6.

8. The face evaluation unit is a second continuous crack detection unit that detects continuous cracks formed continuously on the face of the tunneling tunnel based on the image data; The second continuity crack detection unit includes: The image data corresponding to the face region is input to a machine learning model in which machine learning is performed using training data including learning image data of the face and correct labels assigned to the continuous cracks formed in the face, thereby detecting the continuous cracks present in the face region. The face evaluation support device according to any one of claims 1 to 7.

9. The segmented area evaluation processing unit includes: Each normal vector of a triangular network composed of the point cloud data corresponding to the face region is polar projected onto a stereo net for each of the divided regions; A fracture dominant direction determination unit that determines a fracture dominant direction indicating a strike and inclination of a fracture dominant on the face for each of the divided regions based on a distribution of plots polar-projected onto the stereonet, The face evaluation unit is a third continuous crack detection unit that detects continuous cracks formed continuously on the face of the tunneling tunnel based on the image data and the point cloud data; The third continuity crack detection unit includes: performing an edge detection process for detecting edges on the image data corresponding to the face region; The edge detected by the edge detection process is fitted to the crack dominant direction for each of the divided areas determined by the crack dominant direction determination unit, thereby detecting the continuous crack present in the face area. The face evaluation support device according to any one of claims 1 to 8.

10. The face evaluation unit is an artifact detection unit that detects an artifact present in the target area based on the image data; The artifact detection unit includes: The image data is divided into the image data corresponding to the face region and the image data corresponding to other than the face region, and each of the image data is input to a machine learning model in which machine learning is performed using training data including learning image data of the face and correct answer labels assigned to artificial objects installed at or around the face, thereby detecting the artificial objects present in the face region and the artificial objects present outside the face region. The face evaluation support device according to any one of claims 1 to 9.

11. The segmented area evaluation processing unit includes: Each normal vector of a triangular network composed of the point cloud data corresponding to the face region is polar projected onto a stereo net for each of the divided regions; A fracture dominant direction determination unit that determines a fracture dominant direction indicating a strike and inclination of a fracture dominant on the face for each of the divided regions based on a distribution of plots polar-projected onto the stereonet, The face evaluation unit is A first skin flaking risk prediction unit predicts a skin flaking risk location based on the point cloud data, The first skin loss risk part prediction unit, The crack prominent direction for each of the divided areas determined by the crack prominent direction determination unit is projected onto a stereo net as a great circle; When a closed polygon is formed by the predominant crack directions extending in at least three directions, the range of the face corresponding to the closed polygon is predicted as the skin fall danger area. The face evaluation support device according to any one of claims 1 to 10.

12. The face evaluation unit is A second skin flaking risk part prediction unit that predicts the skin flaking risk part based on the image data; a skin loss danger part evaluation part that predicts the skin loss danger part based on a prediction result by the first skin loss danger part prediction part and a prediction result by the second skin loss danger part prediction part; The second skin loss risk prediction unit is The image data corresponding to the face region is input to a machine learning model in which machine learning is performed using training data including learning image data of the face and a correct answer label assigned to the face-fall danger area at the face, thereby predicting the face-fall danger area; The skin fall risk area evaluation unit is the skin fall risk area predicted by the first skin fall risk area prediction unit and the skin fall risk area predicted by the second skin fall risk area prediction unit are compared, and the skin fall risk area having a high degree of agreement as a result of the comparison is determined to be at a high risk. The face evaluation support device according to claim 11.

13. The image data acquisition unit A color correction unit performs color correction on color image data obtained by photographing the target area in color, the image data including the face and a color sample having a predetermined black and white pattern, The color correction unit includes: converting the color image data into a grayscale image, and arranging all pixels constituting the grayscale image in order of brightness to obtain a brightness median value of the pixel located in the middle; Classifying all the pixels into a high-brightness pixel group consisting of pixels having a brightness higher than the intermediate brightness value and a low-brightness pixel group consisting of pixels having a brightness equal to or lower than the intermediate brightness value; calculating average values ​​of the three RGB components for a pixel group corresponding to a white portion of the color sample, selecting a maximum value among the average values, and adding difference values ​​between the maximum value and each of the average values ​​as correction values ​​for the high-lightness pixel group to each of the pixels of the high-lightness pixel group for each of the three components, thereby color-correcting the high-lightness pixel group; calculating average values ​​of the three components for a pixel group corresponding to a black portion of the color sample, selecting a minimum value from among the average values, and subtracting each difference value between the minimum value and each average value from each pixel of the low-lightness pixel group for each of the three components as a correction value for the low-lightness pixel group, thereby color-correcting the low-lightness pixel group; The face evaluation support device according to any one of claims 1 to 12.

14. An image data acquisition step of acquiring image data of a target area including a tunnel face; a point cloud data acquisition step of acquiring point cloud data obtained by measuring a three-dimensional shape of the target area; A face evaluation process for evaluating the face based on the image data and the point cloud data; A support information output process for outputting support information based on the evaluation result by the face evaluation process, The face evaluation process includes: A face extraction process of extracting a face area corresponding to the face from the target area based on at least one of the image data and the point cloud data; a roughness quantification step of quantifying the roughness of the face in the face region as a roughness feature value along a plurality of scanning lines set in a plurality of scanning directions with respect to the face region based on the point cloud data; A region dividing step of calculating a representative value of the roughness feature amount for each intersection point where a plurality of the scanning lines intersect, and dividing the face region into divided regions based on a distribution of the representative value for each intersection point; For each divided area into which the face area is divided by the area division step, the face is evaluated for the observation items related to the face, and the evaluation result is output as the evaluation result. A method to assist in face evaluation.

Images