Face evaluation support system and face evaluation support method

Abstract

To provide a face evaluation support system 1 and a face evaluation support method that can reduce the burden on users and accurately evaluate a face F.SOLUTION: The face evaluation support system 1 for supporting evaluation based on evaluation items for the face F, includes visible image acquisition means (terminal control unit 26) for acquiring visible image data 35 including visible information of the face F, invisible image acquisition means (terminal control unit 26) for acquiring thermal image data 36 including invisible information of the face F, a first machine learning model 33 for evaluating the visible information of the visible image data 35 based on the evaluation items and calculating a visible information evaluation value, a second machine learning model 34 for evaluating the invisible information of the thermal image data 36 based on the evaluation items and calculating a thermal information evaluation value, and output means (terminal control unit 26) for outputting the evaluation results for the face F for the evaluation items based on the visible information evaluation value and the thermal information evaluation value.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a tunnel face evaluation support system and a tunnel face evaluation support method that support evaluation based on evaluation items for the tunnel face, for example. [Background technology]

[0002] At the tunnel excavation site, the condition of the tunnel face is observed daily as excavation progresses, and the results of the observations are recorded in a face observation record book.The ground classification and support structure are selected based on the face observation record book.

[0003] As an example of a system to assist in the creation of such face observation records, a system such as that described in Patent Document 1 is known in which an observer's evaluation of the face and visible image data of the face captured by a visible light camera are used as input information, and a ground assessment program determines the condition of the face and assists in the creation of a face observation record.

[0004] Incidentally, when an observer evaluates a face, the observer observes and evaluates the face based on geological knowledge for a wide range of evaluation items such as the face condition, compressive strength, and spring water. For this reason, systems such as that described in Patent Document 1, which require observation by an observer, not only place a heavy burden on the observer, but also have the problem of unstable evaluation accuracy, as the evaluation results are prone to variation depending on the observer's geological knowledge. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 10-39042 Summary of the Invention [Problem to be solved by the invention]

[0006] In view of the above-mentioned problems, the present invention aims to provide a face evaluation support system and a face evaluation support method that can reduce the burden on users and accurately evaluate face surfaces. [Means for solving the problem]

[0007] This invention is a tunnel face evaluation support system that supports evaluation based on evaluation items for the tunnel face, and is characterized by comprising: a visible image acquisition means for acquiring visible image data including visible information of the face; an invisible image acquisition means for acquiring invisible image data including invisible information of the face; a visible information evaluation means for evaluating the visible information of the visible image data based on the evaluation items and calculating a visible information evaluation value; an invisible information evaluation means for evaluating the invisible information of the invisible image data based on the evaluation items and calculating an invisible information evaluation value; and an output means for outputting an evaluation result of the face for the evaluation items based on the visible information evaluation value and the invisible information evaluation value.

[0008] The present invention also provides a tunnel face evaluation support method for supporting evaluation based on evaluation items for the tunnel face, and is characterized by carrying out the following steps: a visible image acquisition step in which a visible image acquisition means acquires visible image data including visible information of the face; an invisible image acquisition step in which an invisible image acquisition means acquires invisible image data including invisible information of the face; a visible information evaluation step in which a visible information evaluation means evaluates the visible information of the visible image data based on the evaluation items and calculates a visible information evaluation value; an invisible information evaluation step in which an invisible information evaluation means evaluates the invisible information of the invisible image data based on the evaluation items and calculates an invisible information evaluation value; and an output step in which an output means outputs the evaluation results of the face for the evaluation items based on the visible information evaluation value and the invisible information evaluation value.

[0009] The visible information of the face is information that can be seen by a person when visually inspecting the face, and includes, for example, the surface shape, color, shade, brightness, and saturation of the face. The visible image acquisition means refers to a means for generating and acquiring visible image data based on a signal from an imaging means, a means for acquiring visible image data stored in a storage means, a means for acquiring visible image data via a communication line, or a means for acquiring visible image data via a portable storage medium.

[0010] The invisible information of the face is information that cannot be seen by a person when visually inspecting the face, and includes, for example, heat distribution, density distribution, or strength distribution. The invisible image acquisition means refers to a means for generating and acquiring invisible image data based on a signal from an imaging means, a means for acquiring invisible image data stored in a storage means, a means for acquiring invisible image data based on the results measured by a measurement means, a means for acquiring invisible image data via a communication line, or a means for acquiring invisible image data via a portable storage medium.

[0011] According to this invention, the evaluation results of the face for the evaluation items can be output using visible image data and invisible image data, so that the condition of the face can be known in detail without relying on an observer observing the face.

[0012] Furthermore, since the evaluation of the face for each evaluation item is performed based on the visible information evaluation value obtained by evaluating visible image data and the invisible information evaluation value obtained by evaluating invisible image data, the face evaluation support system can reduce the variation in evaluation results compared to when an observer evaluates the face, and can output evaluation results with high evaluation accuracy. Therefore, the face evaluation support system and face evaluation support method can reduce the burden on users and accurately evaluate the face.

[0013] As an aspect of the present invention, a plurality of evaluation items may be set for the working face, and the visible information evaluation means and the invisible information evaluation means may be provided for each evaluation item. According to this configuration, the visible information evaluation means and the invisible information evaluation means can be optimized for each evaluation item.

[0014] This allows the face evaluation support system to evaluate the face using visible information evaluation means and invisible information evaluation means optimized for each evaluation item. Therefore, the face evaluation support system can improve the evaluation accuracy for each evaluation item compared to when evaluation of the face based on multiple evaluation items is performed using one visible information evaluation means and one invisible information evaluation means.

[0015] In another aspect of the present invention, the evaluation items are set to a plurality of evaluation categories indicating the condition of the face, the visible information evaluation means is configured to calculate the frequency of occurrence of each evaluation category as the visible information evaluation value, and the invisible information evaluation means is configured to calculate the frequency of occurrence of each evaluation category as the invisible information evaluation value.

[0016] According to this configuration, the visible image data and the invisible image data can be evaluated in detail, and therefore the face evaluation support system can improve the evaluation accuracy of the evaluation results output by the output means.

[0017] In another aspect of the present invention, the output means may be configured to calculate the occurrence frequency for each evaluation category based on the visible information evaluation value and the invisible information evaluation value, and to use the evaluation category with the highest occurrence frequency among the calculated occurrence frequencies for each evaluation category as the evaluation result. According to this configuration, the evaluation category with the highest frequency of occurrence is output as the evaluation result, eliminating the need for the user to determine the evaluation category, thereby improving convenience for the user.

[0018] In another aspect of the present invention, the visible information evaluation means may be constructed with a machine learning model that takes the visible image data as input information, evaluates the visible image data based on the evaluation items, and outputs the visible information evaluation value, and the invisible information evaluation means may be constructed with a machine learning model that takes the invisible image data as input information, evaluates the invisible image data based on the evaluation items, and outputs the invisible information evaluation value.

[0019] The machine learning model constituting the visible information evaluation means and the machine learning model constituting the invisible information evaluation means may be machine learning models of the same type or different types.

[0020] According to this configuration, the visible information evaluation means and the invisible information evaluation means are each constructed using a machine learning model, so that the visible information of the visible image data and the invisible information of the invisible image data can be evaluated with high accuracy. This allows the face evaluation support system to further improve the evaluation accuracy of the evaluation results output by the output means, thereby enabling more accurate evaluation of the face.

[0021] As another aspect of the present invention, the output means may be configured to output the evaluation result based on an average value or a weighted average value of the visible information evaluation value and the invisible information evaluation value.

[0022] According to this configuration, an ensemble learning model can be constructed using the output means and the visible information evaluation means and invisible information evaluation means constructed using a machine learning model. Therefore, the face evaluation support system can improve the evaluation accuracy of the face even when the evaluation accuracy of the visible information evaluation means or the invisible information evaluation means is low.

[0023] In another aspect of the present invention, the output means may be constructed using a machine learning model that uses the visible information evaluation value and the invisible information evaluation value as input information and outputs the evaluation result based on the visible information evaluation value and the invisible information evaluation value.

[0024] According to this configuration, an ensemble learning model can be constructed using the visible information evaluation means, invisible information evaluation means, and output means constructed using a machine learning model. Therefore, the face evaluation support system can improve the evaluation accuracy of the face even when the evaluation accuracy of the visible information evaluation means or the invisible information evaluation means is low.

[0025] As another aspect of the present invention, a teacher data storage means may be provided which stores the visible image data as teacher data for the visible information evaluation means and stores the invisible image data as teacher data for the invisible information evaluation means.

[0026] According to this configuration, it is possible to diversify the training data using visible image data and invisible image data obtained at the construction site, thereby further improving the evaluation accuracy of the visible information evaluation means and invisible information evaluation means constructed using a machine learning model.

[0027] As another aspect of the present invention, a processing means may be provided for performing a predetermined processing process on the visible image data and the invisible image data stored in the teacher data storage means. The processing may be, for example, adding noise, modifying the exposure, inverting, or rotating.

[0028] According to this configuration, the diversity of the training data can be further increased, thereby further improving the evaluation accuracy of the visible information evaluation means and invisible information evaluation means constructed using a machine learning model.

[0029] As a result, the face evaluation support system can output more accurate face evaluation results because it improves the calculation accuracy of the visible information evaluation value calculated by the visible information evaluation means and the calculation accuracy of the invisible information evaluation value calculated by the invisible information evaluation means.

[0030] Another aspect of the present invention may include a display means for displaying the visible image data or the invisible image data, an operation reception means for receiving a user's operation to specify any point that will become the contour of the face of the cutting edge in the visible image data or the invisible image data displayed on the display means, and a face contour generation means for generating the contour of the face of the cutting edge based on three or more of the arbitrary points specified by the operation reception means.

[0031] This configuration allows the face to be easily extracted from the visible image data and the invisible image data, respectively, which reduces the user's effort required for extracting the face, thereby improving both user convenience and evaluation efficiency.

[0032] In another aspect of the present invention, an area setting means is provided for setting a plurality of predetermined areas indicating portions of the face for the visible image data and the invisible image data based on the face contour generated by the face contour generating means, and the visible information evaluation means may calculate the visible information evaluation value for each of the predetermined areas of the visible image data, and the invisible information evaluation means may calculate the invisible information evaluation value for each of the predetermined areas of the invisible image data.

[0033] This configuration allows for more efficient evaluation of the face than when the entire face is the target of evaluation. Furthermore, since it is possible to use evaluation results according to the importance and priority of a specific area, the face evaluation support system can improve convenience for users.

[0034] In another aspect of the present invention, the output means may be configured to set the other evaluation value as the evaluation result when one of the visible information evaluation value and the invisible information evaluation value has not been obtained. According to this configuration, even if it is not possible to acquire either the visible image data or the invisible image data, it is possible to output the evaluation result of the working face.

[0035] As another aspect of the present invention, the invisible image data may be thermal image data obtained by capturing an image of the heat distribution on the working face. This configuration makes it easier to generate invisible image data compared to invisible image data based on measurement results such as laser measurement, sonic measurement, etc. Therefore, the face evaluation support system reduces the effort required to obtain invisible image data and enables efficient evaluation of the face.

[0036] Another aspect of the present invention is a system that includes a mobile terminal used by a user and a server connected to the mobile terminal via a communication line, and the mobile terminal is equipped with a first imaging means that captures the visible information of the face, a second imaging means that captures the invisible information of the face, the visible image acquisition means, the invisible image acquisition means, the visible information evaluation means, the invisible information evaluation means, and the output means.

[0037] According to this configuration, by providing the first imaging means and the second imaging means in the portable terminal used by the user, visible image data and invisible image data can be easily obtained on the portable terminal, which has excellent portability.

[0038] Furthermore, since the entire process from capturing images of the face to outputting the face evaluation results can be performed on a mobile terminal, the face evaluation support system can evaluate the face even inside a tunnel where communication conditions are unstable. This allows the face evaluation support system to reduce the burden on users and to more efficiently evaluate face surfaces.

[0039] In another aspect of the present invention, the mobile terminal may be equipped with a transmission means for transmitting the evaluation results of the face output by the output means to the server, and the server may be equipped with a server storage means for storing the evaluation results of the face obtained from the mobile terminal.

[0040] This configuration allows the face evaluation results to be easily shared with terminals other than mobile terminals, for example, a management terminal in a management office away from the construction site. Therefore, the face evaluation support system can easily create a face observation record book using the face evaluation results from terminals other than mobile terminals. This allows the face evaluation support system to improve convenience for users. [Effects of the Invention]

[0041] The present invention can provide a face evaluation support system and a face evaluation support method that can reduce the burden on users and accurately evaluate face surfaces. [Brief explanation of the drawings]

[0042] [Figure 1] FIG. 1 is a schematic diagram illustrating an outline of a face evaluation support system. [Figure 2] FIG. 1 is a diagram showing the configuration of a face evaluation support system. [Figure 3] FIG. 1 is a block diagram showing the internal configuration of a face evaluation support system. [Figure 4] FIG. 2 is an explanatory diagram illustrating visible image data and thermal image data. [Figure 5] FIG. 1 is a schematic explanatory diagram illustrating an overview of a first machine learning model. [Figure 6] FIG. 1 is a schematic diagram illustrating an overview of a second machine learning model. [Figure 7] FIG. 4 is an explanatory diagram illustrating cut-out visible image data and cut-out thermal image data. [Figure 8] FIG. 10 is a schematic explanatory diagram illustrating an outline of a support screen. [Figure 9] 10 is a flowchart showing the processing operation of evaluation support processing. [Figure 10] FIG. 10 is an explanatory diagram illustrating setting of a cutting area. [Figure 11] FIG. 10 is an explanatory diagram illustrating setting of a cutting area. [Figure 12] 10 is a flowchart showing the processing operation of face evaluation processing. [Figure 13] FIG. 10 is an explanatory diagram for explaining calculation of a face evaluation value. [Figure 14] FIG. 10 is an explanatory diagram illustrating an overview of an evaluation result screen. [Figure 15] FIG. 10 is a sequence diagram showing the processing operations in the face evaluation support system. [Figure 16] FIG. 10 is a schematic explanatory diagram illustrating an overview of a calculation process of an evaluation result in another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0043] An embodiment of the present invention will be described below with reference to the drawings. The tunnel face evaluation support system 1 of this embodiment is a system that supports evaluation based on evaluation items of the tunnel face F of a tunnel T. Such a tunnel face evaluation support system 1 will be described with reference to Figs. 1 to 7.

[0044] FIG. 1 shows a schematic diagram illustrating the outline of the face evaluation support system 1, FIG. 2 shows a configuration diagram of the face evaluation support system 1, FIG. 3 shows a block diagram of the face evaluation support system 1, and FIG. 4 shows an explanatory diagram explaining the visible image data 35 and the thermal image data 36.

[0045] Furthermore, Figure 5 shows a schematic diagram illustrating the outline of the first machine learning model 33, Figure 6 shows a schematic diagram illustrating the outline of the second machine learning model 34, and Figure 7 shows an explanatory diagram illustrating the visible image crop data 37 and the thermal image crop data 38.

[0046] In addition, the arrow X in Figure 1 indicates the tunnel axis direction (hereinafter referred to as the tunnel axis direction X), the arrow Xi in the figure indicates the excavation direction of the tunnel T (hereinafter referred to as the excavation direction Xi), and the arrow Xo indicates the mine portal direction (hereinafter referred to as the mine portal direction Xo), which is in the opposite direction to the excavation direction Xi.

[0047] First, as shown in Figures 1 and 2, the face evaluation support system 1 includes a management terminal 10 located remotely away from the excavation site of the tunnel T, a mobile terminal 20 equipped with a camera that is used by a worker or construction manager as user M at the excavation site, and a server 40 connected to the management terminal 10 and the mobile terminal 20 via a communication line 2.

[0048] Such a face evaluation support system 1 is configured so that a user M uses a mobile terminal 20 to photograph a face F inside a tunnel T after excavation, and the mobile terminal 20 evaluates the condition of the face F based on the image data of the face F.

[0049] In more detail, the management terminal 10 is a terminal used by users working in a management office located outside the tunnel T, or by workers or construction managers at the excavation site of another tunnel T. As shown in Figure 3, this management terminal 10 includes an operation reception unit 11 that receives various operations from users, a display unit 12 that displays various information, a memory unit 13 that stores various information, a line connection unit 14 that connects to the communication line 2, and a control unit 15 that controls the operations of these units.

[0050] Specifically, as shown in FIG. 2, the operation reception unit 11 is composed of, for example, a keyboard 11a and a mouse 11b, and has the function of receiving input operations from a user and the function of outputting information indicating the received input content to the control unit 15. The display unit 12 is configured by, for example, a liquid crystal display, and has the function of displaying various information in response to a control signal from the control unit 15 .

[0051] The storage unit 13 is configured with a hard disk or a non-volatile memory, and has the function of writing and storing various information, and the function of reading out various information. The storage unit 13 stores a progress management support program (not shown) that supports progress management of the excavation work of the tunnel T, a record book creation support program (not shown) that supports the creation of a face observation record book, and the like.

[0052] The line connection unit 14 is configured by, for example, a wired LAN board, and has a function of connecting to the communication line 2 and a function of transmitting and receiving various information via the communication line 2. The control unit 15 is composed of hardware such as a CPU and memory, and software such as a control program.

[0053] This control unit 15 has processing functions related to the exchange of various information with the server 40, processing functions related to the exchange of various signals with the operation reception unit 11, display unit 12, memory unit 13 and line connection unit 14, and the function of controlling the operation of each unit connected via a specified bus.

[0054] As shown in FIG. 3, the mobile terminal 20 includes an operation display unit 21 that displays various information and accepts various operations from the user M, a visible light camera 22 and a thermal camera 23 that capture still images of the subject, a terminal memory unit 24 that stores various information, a line connection unit 25 that connects to the communication line 2, and a terminal control unit 26 that controls the operations of these units.

[0055] Specifically, as shown in FIG. 2, the operation display unit 21 is configured with, for example, a touch panel display, and has the function of accepting various operations from the user and the function of displaying various information.

[0056] The visible light camera 22 is a camera that converts visible light reflected by a subject into an electrical signal and outputs it as an image signal. It has the function of photographing the subject based on a control signal from the terminal control unit 26, and the function of outputting the image signal of the subject to the terminal control unit 26.

[0057] The thermal camera 23 is a camera that converts infrared rays emitted by a subject into an electrical signal and outputs it, and is provided to photograph the heat distribution of the working face F. The thermal camera 23 is a camera that is pre-installed in the mobile terminal 20, or a camera that is attached to the mobile terminal 20 later.

[0058] The thermal camera 23 has a function of photographing an object based on a control signal from the terminal control unit 26, and a function of outputting an image signal obtained by photographing the object to the terminal control unit 26. The terminal storage unit 24 is configured by a hard disk or a nonvolatile memory, and has the function of writing and storing various information and the function of reading out various information.

[0059] As shown in Figure 3, this terminal memory unit 24 stores an evaluation support program 31 that supports the evaluation of the face F based on evaluation items, project data 32 set and registered for each excavation site, multiple first machine learning models 33 and multiple second machine learning models 34 that analyze and evaluate the condition of the face F, and the like.

[0060] More specifically, the evaluation support program 31 is a program that performs the evaluation support process described below, and is downloaded and acquired in advance from the server 40 . The project data 32 includes the name of the construction site, the construction location, various information about the progress, and the like, which are registered in association with each other.

[0061] Furthermore, as shown in FIG. 3, visible image data 35 and thermal image data 36 are registered in the project data 32 in association with each other. As shown in Figure 4(a), the visible image data 35 is a still image captured by a visible light camera 22 of the interior of the tunnel T, including the face F, as viewed from the tunnel axis direction X, and the surface shape, color, shade, brightness, and saturation of the surface inside the tunnel T are captured as visible information.

[0062] On the other hand, the thermal imaging data 36 is a still image taken by the thermal camera 23 of the inside of the tunnel T including the face F as seen from the tunnel axis direction X, as shown in Figure 4(b), and the heat distribution inside the tunnel T is captured as invisible information. It is desirable that the thermal imaging data 36 is image data captured at approximately the same photographing position as the visible image data 35.

[0063] 3, the first machine learning model 33 and the second machine learning model 34 are provided for each evaluation item of the working face F. The first machine learning model 33 and the second machine learning model 34 are machine learning models optimized for the evaluation items, and are obtained by downloading from the server 40. The first machine learning model 33 and the second machine learning model 34 will be described in detail later.

[0064] The line connection unit 25 is configured by, for example, a wireless communication module, and has a function of connecting to the communication line 2 and a function of transmitting and receiving various information via the communication line 2. The terminal control unit 26 is composed of hardware such as a CPU and memory, and software such as a control program.

[0065] This terminal control unit 26 has processing functions related to the exchange of various information with the server 40, processing functions related to the exchange of various signals with the operation display unit 21, visible light camera 22, thermal camera 23, terminal memory unit 24 and line connection unit 25, and the function of controlling the operation of each unit connected via a specified bus.

[0066] As shown in FIG. 3, the server 40 includes a line connection unit 41 that connects to the communication line 2, a server storage unit 42 that stores various information, and a server control unit 43 that controls the operations of these units. Specifically, the line connection unit 41 is configured by, for example, a wired LAN board, and has a function of connecting to the communication line 2 and a function of transmitting and receiving various information via the communication line 2.

[0067] The server storage unit 42 is configured with a hard disk or nonvolatile memory, and has the function of writing and storing various information, and the function of reading out various information. As shown in Figure 3, this server memory unit 42 stores a first machine learning model (not shown) and a second machine learning model (not shown) that are delivered to the mobile terminal 20, as well as project data 44 that is set and registered for each excavation site.

[0068] Furthermore, the server memory unit 42 stores project data 44, such as image data 45, such as visible image data 35 and thermal image data 36 acquired from the mobile terminal 20, and evaluation data 46 showing the evaluation results of the face F acquired from the mobile terminal 20, which are registered in association with the project name, etc.

[0069] The image data 45 is stored not only as a record of the evaluation of the working face F, but also for use as training data D1 for a first machine learning model and training data D2 for a second machine learning model, which will be described later. The server control unit 43 is composed of hardware such as a CPU and memory, and software such as a control program.

[0070] This server control unit 43 has processing functions related to the exchange of various information with the management terminal 10 and the mobile terminal 20, processing functions related to the exchange of various signals with the line connection unit 41 and the server memory unit 42, and the function of controlling the operation of each unit connected via a specified bus.

[0071] Next, the first machine learning model 33 and the second machine learning model 34 will be described in more detail. First, as examples of evaluation items for the face F, 10 types of evaluation items have been pre-set, as shown in Table 1: "Condition of the face," "Condition of the bare excavation surface," "Compressive strength," "Weathering," "Proportion of the fractured area in the face," "Frequency of cracks," "Condition of cracks," "Shape of cracks," "Spring water," and "Deterioration due to water."

[0072] Furthermore, although a detailed description of each evaluation item will be omitted, three or more evaluation categories, more specifically, three to six evaluation categories, are set in advance to categorize the condition of the working face F. In this embodiment, as an example, a good condition of the working face F is designated as "A" and a bad condition of the working face F is designated as "D", and four evaluation categories from "A" to "D" are set.

[0073] [Table 1] To facilitate the following explanation, in this embodiment, the 10 types of evaluation items will be referred to as n=1st to n=10th evaluation items, and the evaluation categories for the nth evaluation item will be referred to as "nA," "nB," "nC," and "nD." For example, when n=1, the evaluation item is the first item in Table 1, "Condition of face," and the evaluation categories are "1-A," "1-B," "1-C," and "1-D" in Table 1.

[0074] For such evaluation items, the first machine learning model 33 and the second machine learning model 34 are constructed to calculate the probability that the condition of the face F falls into the evaluation category for each evaluation category. The first machine learning model 33 and the second machine learning model 34 are constructed so that the sum of the probabilities of falling into each evaluation category is "1.0."

[0075] More specifically, the first machine learning model 33 is an estimation algorithm using a convolutional neural network, as shown in FIG. 5, which includes, for example, an input layer 33a to which input information is input, an intermediate layer 33b, and an output layer 33c to which output information is output.

[0076] As shown in Figure 5, this first machine learning model 33 is constructed so that it takes as input information one piece of visible image clipping data 37, which is a portion of visible image data 35, evaluates the visible image clipping data 37 based on the evaluation items, and obtains, as output information for the input information, a visible information evaluation value that indicates the evaluation result of the visible image clipping data 37.

[0077] Specifically, the first machine learning model 33 is constructed to obtain the probability that the condition of the face F in the visible image cut data 37 falls into the evaluation category "nA", the probability that it falls into the evaluation category "nB", the probability that it falls into the evaluation category "nC", and the probability that it falls into the evaluation category "nD" as visible information evaluation values.

[0078] The visible image crop data 37 input to the input layer 33a is of three types, as shown in Figure 7(a): a top image 37a obtained by cropping the top end of the face F in the visible image data 35 into an approximately rectangular shape; a right shoulder image 37b obtained by cropping the right shoulder of the face F into an approximately rectangular shape; and a left shoulder image 37c obtained by cropping the left shoulder of the face F into an approximately rectangular shape.

[0079] Such a first machine learning model 33 is constructed by repeatedly learning visible image crop data 37 (top image 37a, right shoulder image 37b, left shoulder image 37c) and processed image data (not shown) that has been processed by adding noise to the visible image crop data 37, correcting the exposure, inverting, or rotating, as training data D1 for the input information.

[0080] In this case, the first machine learning model 33 identifies the characteristics of the face F in the evaluation item based on the RGB values of the input information, the visible image clipping data 37, and learns by repeatedly adjusting the weighting and judgment threshold based on the training data D1 so that the output information for the input information has a probability of falling into the evaluation category.

[0081] On the other hand, the second machine learning model 34 is an estimation algorithm using a convolutional neural network, as shown in FIG. 6, which includes, for example, an input layer 34a to which input information is input, an intermediate layer 34b, and an output layer 34c to which output information is output.

[0082] As shown in Figure 6, this second machine learning model 34 is constructed so that it takes one piece of thermal image crop data 38, which is a cut-out portion of the thermal image data 36, as input information, evaluates the thermal image crop data 38 based on the evaluation items, and obtains a thermal information evaluation value indicating the evaluation result of the thermal image crop data 38 as output information for the input information.

[0083] Specifically, the second machine learning model 34 is constructed to obtain the thermal information evaluation values as the probability that the condition of the face F in the thermal image cut data 38 falls into the evaluation category "nA", the probability that it falls into the evaluation category "nB", the probability that it falls into the evaluation category "nC", and the probability that it falls into the evaluation category "nD".

[0084] The thermal image crop data 38 input to the input layer 34a is of three types, as shown in Figure 7(b): a top image 38a obtained by cropping the top end of the working face F in the thermal image data 36 into an approximately rectangular shape; a right shoulder image 38b obtained by cropping the right shoulder of the working face F into an approximately rectangular shape; and a left shoulder image 38c obtained by cropping the left shoulder of the working face F into an approximately rectangular shape.

[0085] Such a second machine learning model 34 is constructed by repeatedly learning thermal image cut data 38 (top image 38a, right shoulder image 38b, left shoulder image 38c) and processed image data (not shown) that has been processed by adding noise to the thermal image cut data 38, correcting the exposure, inverting, or rotating it, as training data D2 for the input information.

[0086] At this time, the second machine learning model 34 identifies the characteristics of the face F in the evaluation item based on the RGB values indicating the thermal distribution of the input information, the thermal image clipping data 38, and learns by repeatedly adjusting the weighting and judgment threshold based on the training data D2 so that the output information for the input information has a probability of falling into the evaluation category.

[0087] Next, the processing operations in the face evaluation support system 1 configured as described above will be explained with reference to FIGS. 8 to 15. FIG. Note that Figure 8 shows a schematic diagram outlining the support screen 200, Figure 9 shows a flowchart of the evaluation support process, Figures 10 and 11 show explanatory diagrams explaining the setting of the cutout area E, and Figure 12 shows a flowchart of the face evaluation process.

[0088] Furthermore, Figure 13 shows an explanatory diagram explaining the calculation of the face evaluation value, Figure 14 shows an explanatory diagram outlining the evaluation result screen 210, and Figure 15 shows a sequence diagram of the face evaluation support system 1.

[0089] First, the user photographs the face F inside the tunnel T after excavation using the visible light camera 22 and thermal camera 23 of the mobile terminal 20, and stores multiple visible image data 35 and thermal image data 36 in the terminal memory unit 24 of the mobile terminal 20. At this time, it is desirable that the imaging of the working face F by the visible light camera 22 and the imaging of the working face F by the thermal camera 23 are carried out at approximately the same position and at approximately the same timing.

[0090] Thereafter, when the user performs an operation to execute the evaluation support program 31, the terminal control unit 26 of the mobile terminal 20 starts the evaluation support process and displays a guide screen (not shown) on the operation display unit 21 prompting the user to enter various registration information such as the project name, date, and the position of the face F.

[0091] At this time, the user inputs the project name and other information according to the instructions on the guide screen, and then presses a transition button (not shown) to transition to evaluation using image data. When detecting that the user has pressed the transition button, the terminal control unit 26 displays on the operation display unit 21 a support screen 200 that guides the user through evaluation using image data.

[0092] For example, as shown in Figure 8, the support screen 200 displays the title "Evaluation using face images" at the top of the screen, and below that, along with the guidance message "(1) Import image data," a visible image selection button 201 for selecting visible image data 35 and a thermal image selection button 202 for selecting thermal image data 36 are displayed.

[0093] Furthermore, the support screen 200 displays a guidance message saying "(2) Specify three points on the arc of the face," as well as a visible image display button 204 that switches the image data in the image display field 203 to visible image data 35, and a thermal image display button 205 that switches the image data in the image display field 203 to thermal image data 36.

[0094] In addition, the support screen 200 displays an image save button 206 below the image display field 203, along with a guidance message saying "(3) Save selected image and cropped image," which stores the visible image data 35, thermal image data 36, visible image cropped data 37, and thermal image cropped data 38 in the project data 32, and displays an evaluation start button 207 at the bottom of the screen, which starts the evaluation of the working face F.

[0095] When the support screen 200 is displayed on the operation display unit 21, the terminal control unit 26 determines whether the user has pressed the visible image selection button 201 or the thermal image selection button 202, as shown in FIG. 9 (step S101).

[0096] If the terminal control unit 26 has not detected the user pressing the visible image selection button 201 or the thermal image selection button 202 (step S101: No), the terminal control unit 26 waits for processing until it detects that the visible image selection button 201 or the thermal image selection button 202 has been pressed.

[0097] On the other hand, when it is detected that the user has pressed the visible image selection button 201 or the thermal image selection button 202 (step S101: Yes), the terminal control unit 26 displays an image selection screen (not shown) on the operation display unit 21 to prompt the user to select image data corresponding to the pressed visible image selection button 201 or thermal image selection button 202 (step S102).

[0098] At this time, the user follows the instructions on the image selection screen to select one of the visible image data 35 or thermal image data 36 corresponding to the image selection button pressed from the multiple image data displayed on the image selection screen, and then presses the decision button.

[0099] When the user presses the OK button on the image selection screen, the terminal control unit 26 temporarily stores the selected visible image data 35 or thermal image data 36 as image data to be used in subsequent processing.

[0100] The user may, for example, press the visible image selection button 201 to select one of the visible image data 35, and then press the thermal image selection button 202 to select one of the thermal image data 36, thereby selecting one of the visible image data 35 and one of the thermal image data 36.

[0101] In this case, as described above, it is desirable for the user to select one visible image data 35 and one thermal image data 36, but if there is no image data suitable for evaluation, the user may select only one of the visible image data 35 or the thermal image data 36.

[0102] After selecting the image data to be used for subsequent processing in this manner, the user presses either the visible image display button 204 or the thermal image display button 205 to switch the image displayed in the image display field 203, and then presses any three points on the contour of the face F in the image display field 203. Specifically, the user designates the arc portions of the contour of the face F by pressing any three points of the arc portion of the contour of the face F displayed in the image display field 203.

[0103] On the other hand, when the terminal control unit 26, which has temporarily stored image data to be used for subsequent processing, detects that either the visible image display button 204 or the thermal image display button 205 has been pressed, it displays the visible image data 35 and thermal image data 36 corresponding to the pressed image button in the image display field 203.

[0104] Thereafter, the terminal control unit 26 determines whether or not the user has pressed any three positions in the image display field 203, as shown in FIG. 9 (step S103). If the user has not pressed any three positions in the image display field 203 (step S103: No), the terminal control unit 26 waits for processing until it detects a press by the user.

[0105] On the other hand, if it is detected that the user has pressed any three points in the image display field 203 (step S103: Yes), the terminal control unit 26 sets the cutout area E for extracting the visible image cutout data 37 and the thermal image cutout data 38 based on the position information of the three pressed points (step S104).

[0106] For example, when the visible image data 35 is displayed in the image display field 203, if any three points on the arc portion on the contour of the face F are pressed as shown by the white circles in Figure 10(a), the terminal control unit 26 generates an arc passing through the three pressed points, as shown in Figure 10(b), and generates an approximately semicircular contour line L based on the generated arc.

[0107] Furthermore, as shown in Figure 11(a), the terminal control unit 26 regards the area surrounded by the contour line L as the face F and generates approximately rectangular cutout areas E at the top end, right shoulder, and left shoulder of the face F. Thereafter, the terminal control unit 26 determines whether or not the user has pressed the image save button 206, as shown in FIG. 9 (step S105).

[0108] If pressing of the image save button 206 is not detected (step S105: No), the terminal control unit 26 returns the process to step S103 and repeats steps S103 to S105 until pressing of the image save button 206 is detected.

[0109] On the other hand, if pressing of the image save button 206 is detected (step S105: Yes), the terminal control unit 26 generates and saves visible image cut data 37 and thermal image cut data 38 based on the three cut areas E (step S106).

[0110] Specifically, the terminal control unit 26 cuts out three cut-out areas E from one of the visible image data 35 and thermal image data 36 displayed in the image display field 203 to generate a cut-out image, and then stores the generated cut-out image in the terminal memory unit 24.

[0111] Furthermore, the terminal control unit 26 overlays the three cutout areas E on the other image data, cuts out the three cutout areas E from the other image data to generate a cutout image, and then stores the generated cutout image in the terminal memory unit 24.

[0112] For example, in step S104, if a cutout area E is set in the visible image data 35, the terminal control unit 26 cuts out three cutout areas E from the visible image data 35, and then saves the cutout image obtained by cutting out the cutout area E at the top end as top end image 37a, the cutout image obtained by cutting out the cutout area E at the right shoulder as right shoulder image 37b, and the cutout image obtained by cutting out the cutout area E at the left shoulder as left shoulder image 37c.

[0113] Furthermore, as shown in Figure 11 (b), the terminal control unit 26 overlaps and cuts out three cutout areas E from the thermal imaging data 36, and then saves the cutout image obtained by cutting out the cutout area E at the top end as top end image 38a, the cutout image obtained by cutting out the cutout area E at the right shoulder as right shoulder image 38b, and the cutout image obtained by cutting out the cutout area E at the left shoulder as left shoulder image 38c.

[0114] After saving the cut visible image data 37 and the cut thermal image data 38, the terminal control unit 26 determines whether or not the user has pressed the evaluation start button 207, as shown in FIG. 9 (step S107). If the terminal control unit 26 has not detected that the user has pressed the evaluation start button 207 (step S107: No), the terminal control unit 26 puts the process on hold until it detects that the user has pressed the evaluation start button 207.

[0115] On the other hand, if it is detected that the user has pressed the evaluation start button 207 (step S107: Yes), the terminal control unit 26 starts a face evaluation process to evaluate the condition of the face F based on the visible image clipping data 37 and the thermal image clipping data 38 (step S108).

[0116] Specifically, when the face evaluation process is started, the terminal control unit 26 reads in the visible image clipping data 37 and the thermal image clipping data 38 corresponding to either the top end, right shoulder or left shoulder of the face F as the image to be evaluated (step S121), as shown in Figure 12, and then initializes the count value n, which indicates the number of the evaluation item, to n = 1 (step S122).

[0117] Thereafter, the terminal control unit 26 reads out the first machine learning model 33 and the second machine learning model 34 corresponding to the n-th evaluation item from the terminal storage unit 24, as shown in FIG. 12 (step S123).

[0118] Then, the terminal control unit 26 starts an image analysis process for analyzing and evaluating the state of the working face F in the cut visible image data 37 and the cut thermal image data 38 using a machine learning model (step S124).

[0119] At this time, the terminal control unit 26 performs in parallel a visible image evaluation process (step S125) that evaluates the visible image cut data 37 of the visible image data 35 using the first machine learning model 33 and outputs a visible information evaluation value, and a thermal image evaluation process (step S126) that evaluates the thermal image cut data 38 of the thermal image data 36 using the second machine learning model 34 and outputs a thermal information evaluation value.

[0120] Specifically, in the visible image evaluation (step S125 in FIG. 12), the terminal control unit 26 inputs one of the cut-out visible image data 37 to the input layer 33a of the first machine learning model 33 as input information.

[0121] At this time, the first machine learning model 33 passes input information from the input layer 33a to the intermediate layer 33b, and from the intermediate layer 33b to the output layer 33c, while comparing and judging with pre-learned weights and thresholds to evaluate the condition of the face F in the visible image cut data 37 based on the evaluation items.

[0122] Then, the first machine learning model 33 calculates the probability that the evaluation category will be "nA", "nB", "nC", and "nD" for the nth evaluation item, and outputs them as visible information evaluation values (see Figure 5).

[0123] More specifically, the first machine learning model 33 identifies the characteristics of the face F in the evaluation item based on the RGB values of the visible image clipping data 37, which is also the training data D1, and compares and judges the characteristics of the face F with the four evaluation categories to calculate the probability of each evaluation category being an evaluation category.

[0124] On the other hand, in the thermal image evaluation (step S126 in FIG. 12), the terminal control unit 26 inputs one of the thermal image cut data 38 to the input layer 34a of the second machine learning model 34 as input information.

[0125] At this time, the second machine learning model 34 passes the input information from the input layer 34a to the intermediate layer 34b, and from the intermediate layer 34b to the output layer 34c, while comparing and judging with pre-learned weights and thresholds to evaluate the condition of the face F in the thermal image cut data 38 based on the evaluation items.

[0126] Then, the second machine learning model 34 outputs the probability that the evaluation category will be "nA", "nB", "nC", and "nD" for the nth evaluation item as thermal information evaluation values (see Figure 6).

[0127] More specifically, the second machine learning model 34 identifies the characteristics of the face F in the evaluation item based on the RGB values indicating the thermal distribution of the thermal image clip data 38, which is also the training data D2, and compares and judges the characteristics of the face F with the four evaluation categories to calculate the probability of each evaluation category being an evaluation category.

[0128] When the image analysis process is completed, the terminal control unit 26 determines whether or not both the visible information evaluation value and the thermal information evaluation value have been acquired (step S127). If both the visible information evaluation value and the thermal information evaluation value have been acquired (step S127: Yes), the terminal control unit 26 calculates the face evaluation value based on the visible information evaluation value output by the first machine learning model 33 and the thermal information evaluation value output by the second machine learning model 34 (step S128).

[0129] Specifically, the terminal control unit 26 calculates the average value of the probability that the visible information evaluation value corresponds to the evaluation category indicated by the visual information evaluation value and the probability that the thermal information evaluation value corresponds to the evaluation category indicated by the thermal information evaluation value for each evaluation category, and sets the average value of the calculated probabilities for each evaluation category as the face evaluation value.

[0130] For example, as shown in Figure 13, if the probability that the first evaluation item at the top end falls into the evaluation category "1-A" in the visible information evaluation value output by the first machine learning model 33 is "0.54" and the probability of the evaluation category "1-A" in the thermal information evaluation value output by the second machine learning model 34 is "0.37", the terminal control unit 26 calculates the average value of the probability "0.54" of the evaluation category "1-A" in the visible information evaluation value and the probability "0.37" of the evaluation category "1-A" in the thermal information evaluation value, and temporarily stores the calculated average value "0.455" as the probability that the top end falls into the evaluation category "1-A".

[0131] Similarly, the terminal control unit 26 calculates the average value of the probability "0.27" of the evaluation category "1-B" in the visible information evaluation value and the probability "0.40" of the evaluation category "1-B" in the thermal information evaluation value, and temporarily stores the calculated average value "0.335" as the probability that the top end falls into the evaluation category "1-B".

[0132] Furthermore, the terminal control unit 26 calculates the average value of the probability "0.10" of the evaluation category "1-C" in the visible information evaluation value and the probability "0.12" of the evaluation category "1-C" in the thermal information evaluation value, and temporarily stores the calculated average value "0.110" as the probability that the top end falls into the evaluation category "1-C".

[0133] Furthermore, the terminal control unit 26 calculates the average value of the probability "0.09" of the evaluation category "1-D" in the visible information evaluation value and the probability "0.11" of the evaluation category "1-D" in the thermal information evaluation value, and temporarily stores the calculated average value "0.100" as the probability that the top end falls into the evaluation category "1-D".

[0134] The terminal control unit 26 then temporarily stores the probability that the top end falls into evaluation category "1-A", the probability that it falls into evaluation category "1-B", the probability that it falls into evaluation category "1-C", and the probability that it falls into evaluation category "1-D" calculated in this manner as the face evaluation value of the top end.

[0135] After calculating the face evaluation value, the terminal control unit 26 determines the evaluation result for the cutout region E of the face F based on the average value of the probability of falling into the evaluation category, as shown in FIG. 12 (step S129).

[0136] Specifically, the terminal control unit 26 determines the evaluation result of the cutout area E to be the evaluation category with the highest probability among the probabilities calculated in step S128 that the evaluation category corresponds to. In this case, if there are two evaluation sections with the highest probability, the terminal control section 26 determines the evaluation section with the worst condition of the working face F as the evaluation result of the cutting area E.

[0137] For example, in the example shown in Figure 13, the probability that the evaluation category "1-A" applies is "0.455", the average value of the probability that the evaluation category "1-B" applies is "0.335", the average value of the probability that the evaluation category "1-C" applies is "0.110", and the average value of the probability that the evaluation category "1-D" applies is "0.100", so the terminal control unit 26 determines the evaluation result for the top end to be evaluation category "1-A".

[0138] In addition, in step S127 of Figure 12, if either the visible information evaluation value or the thermal information evaluation value has not been obtained (step S127: No), the terminal control unit 26 proceeds to step S129 and determines the evaluation category with the highest probability for the other evaluation value that has been obtained as the evaluation result. After determining the evaluation result, the terminal control unit 26 determines whether the count value n is 10 (step S130).

[0139] If the count value n is not n=10 (step S130: No), the terminal control unit 26 determines that evaluation based on all 10 evaluation items has not been completed for the visible image clipping data 37 and thermal image clipping data 38 to be evaluated, and updates the count value n by adding "1" to the current count value n (step S131). Thereafter, the terminal control unit 26 returns the process to step S123 and repeats the processes from step S123 to step S130 until the count value n becomes n=10.

[0140] On the other hand, if the count value n is n=10 (step S130: Yes), the terminal control unit 26 determines that evaluation based on all 10 evaluation items has been completed for the visible image clipping data 37 and thermal image clipping data 38 to be evaluated, and determines whether evaluation has been completed for all visible image clipping data 37 and thermal image clipping data 38 (step S132).

[0141] If evaluation of all visible image clipping data 37 and thermal image clipping data 38 has not been completed (step S132: No), the terminal control unit 26 reads the visible image clipping data 37 and thermal image clipping data 38 corresponding to one of the unevaluated areas, among the top end, right shoulder, or left shoulder of the working face F, and changes the visible image clipping data 37 and thermal image clipping data 38 to be evaluated (step S133).

[0142] Thereafter, the terminal control unit 26 returns the process to step S122 and repeats the processes from step S122 to step S132 until evaluation of all the cut visible image data 37 and cut thermal image data 38 is completed.

[0143] On the other hand, if the evaluation of all visible image cut data 37 and thermal image cut data 38 has been completed (step S132: Yes), the terminal control unit 26 displays the evaluation result screen 210 showing the evaluation result of the face F on the operation display unit 21 (step S134).

[0144] For example, as shown in FIG. 14, the evaluation result screen 210 displays the title "Evaluate with face image" at the top of the screen, and a save button 211 for saving the evaluation results at the bottom.

[0145] Furthermore, on the evaluation result screen 210, a top end button 212 that displays the evaluation results of the top end, a right shoulder button 213 that displays the evaluation results of the right shoulder, and a left shoulder button 214 that displays the evaluation results of the left shoulder are displayed above the save button 211.

[0146] In addition, the evaluation result screen 210 displays the evaluation items and evaluation categories corresponding to the parts selected by the top button 212, right shoulder button 213, and left shoulder button 214, and a double circle indicating the evaluation result determined in step S129 is displayed in the corresponding evaluation category.

[0147] When the evaluation result screen 210 is displayed on the operation display unit 21, the terminal control unit 26 ends the face evaluation process and advances the process to step S109 in FIG. Returning to FIG. 9, when the face evaluation process is completed, the terminal control unit 26 determines whether or not the user has pressed the save button 211 on the evaluation result screen 210 (step S109). If the terminal control unit 26 has not detected that the user has pressed the save button 211 (step S109: No), the terminal control unit 26 puts the process on hold until it detects that the save button 211 has been pressed.

[0148] On the other hand, when detecting that the user has pressed the save button 211 (step S109: Yes), the terminal control unit 26 associates the visible image data 35 and thermal image data 36, the visible image crop data 37 and thermal image crop data 38, and the evaluation results for each evaluation item, and stores them in the project data 32 of the terminal memory unit 24, as shown in FIG. 9.

[0149] Furthermore, the terminal control unit 26 transmits the visible image data 35, the thermal image data 36, the visible image crop data 37, the thermal image crop data 38, and the evaluation results for each evaluation item to the server 40 as transmission information (step S110), and then terminates the face evaluation support process.

[0150] Thus, in step S110 of the evaluation support process, when the server 40 receives the transmission information sent by the mobile terminal 20, the server control unit 43 of the server 40 starts the data registration process (step S151), as shown in FIG. 15, and registers the transmission information in the project data 44 in the server memory unit 42.

[0151] At this time, the server control unit 43 stores the evaluation results for each evaluation item in the evaluation data 46, and stores the visible image data 35, the thermal image data 36, the visible image crop data 37, and the thermal image crop data 38 in the image data 45.

[0152] In addition, when a user who wishes to create a face observation record book operates the management terminal 10 at a management office or the like to start the record book creation support process (step S152), the control unit 15 of the management terminal 10 sends request information to the server 40 requesting the transmission of the evaluation results of the face F corresponding to the project name.

[0153] At this time, the server 40 that has received the request information transmits the visible image data 35 and the evaluation data 46 read from the project data 44 corresponding to the project name as response information to the management terminal 10, as shown in FIG. 15 (step S153).

[0154] The control unit 15 of the management terminal 10 that receives the response information displays the response information on the display unit 12, and also accepts various operations by the user to assist in creating a face observation record book.

[0155] Thereafter, as shown in FIG. 15, when the control unit 15 of the management terminal 10 receives an output operation from the user, it outputs the face observation record book to the storage unit 13 or a printer (not shown) (step S154).

[0156] Furthermore, when the server 40 receives a user operation to start learning, for example, a machine learning model (step S115), the server control unit 43 inputs the teacher data D1 and the teacher data D2 into the first learning machine model (not shown) and the second machine learning model (not shown) stored in the server memory unit 42, respectively, as shown in FIG. 15, and starts learning (step S156).

[0157] At this time, the server control unit 43 accepts user operations and generates processed image data by applying processing such as adding noise, correcting exposure, inverting, or rotating to the visible image crop data 37 and the thermal image crop data 38 of the image data 45.

[0158] Thereafter, the server control unit 43 inputs the visible image cut data 37 and the processed image data as training data D1 into the first learning machine model, and inputs the thermal image cut data 38 and the processed image data as training data D2 into the second learning machine model to train it.

[0159] When the learning of the first machine learning model and the second machine learning model is completed, the server control unit 43 transmits the first machine learning model and the second machine learning model to the mobile terminal 20 as update information.

[0160] At this time, the terminal control unit 26 of the mobile terminal 20 that received the update information stores and updates the first machine learning model and the second machine learning model acquired from the server 40 as new first machine learning model 33 and second machine learning model 34 in the terminal memory unit 24, as shown in Figure 15 (step S157). In this way, the face evaluation support system 1 of this embodiment supports the evaluation of the face F by outputting the evaluation results of the face F for the evaluation items.

[0161] As described above, the face evaluation support system 1 of this embodiment is a system that supports evaluation of the face F of a tunnel T based on evaluation items. This face evaluation support system 1 is equipped with a visible image acquisition means (terminal control unit 26) that acquires visible image data 35 including visible information of the face F, and an invisible image acquisition means (terminal control unit 26) that acquires thermal image data 36 including the heat distribution of the face F.

[0162] Furthermore, the face evaluation support system 1 includes a visible information evaluation means (first machine learning model 33) that evaluates the RGB values indicating the visible information of the visible image data 35 based on the evaluation items and calculates a visible information evaluation value.

[0163] In addition, the face evaluation support system 1 is equipped with an invisible information evaluation means (second machine learning model 34) that evaluates the RGB values indicating the thermal distribution of the thermal imaging data 36 based on the evaluation items and calculates a thermal information evaluation value. The face evaluation support system 1 is provided with an output means (terminal control unit 26) that outputs the evaluation result of the face F for the evaluation items based on the visible information evaluation value and the thermal information evaluation value.

[0164] In addition, the face evaluation support method in this embodiment includes a visible image acquisition process in which a visible image acquisition means (terminal control unit 26) acquires visible image data 35 including visible information of the face F, and an invisible image acquisition process in which an invisible image acquisition means (terminal control unit 26) acquires thermal image data 36 including the heat distribution of the face F.

[0165] Furthermore, the face evaluation support method performs a visible information evaluation step in which the visible information evaluation means (first machine learning model 33) evaluates the RGB values indicating the visible information of the visible image data 35 based on evaluation items and calculates a visible information evaluation value.

[0166] In addition, the face evaluation support method performs an invisible information evaluation process in which an invisible information evaluation means (second machine learning model 34) evaluates the RGB values indicating the thermal distribution of the thermal imaging data 36 based on evaluation items and calculates a thermal information evaluation value.

[0167] The face evaluation support method then performs an output step in which output means (terminal control unit 26) outputs the evaluation results of the face F for the evaluation items based on the visible information evaluation value and the thermal information evaluation value.

[0168] According to this configuration, the evaluation results of the face F for the evaluation items can be output using the visible image data 35 and the thermal image data 36, so that the condition of the face F can be known in detail without relying on the observer observing the face F.

[0169] Furthermore, since the evaluation of the face F for the evaluation items is performed based on the visible information evaluation value obtained by evaluating the visible image data 35 and the thermal information evaluation value obtained by evaluating the thermal image data 36, the face evaluation support system 1 can reduce the variation in the evaluation results compared to when an observer evaluates the face F, and can output evaluation results with high evaluation accuracy. Therefore, the face evaluation support system 1 and the face evaluation support method can reduce the burden on the user and can evaluate the face F with high accuracy.

[0170] Furthermore, since the first machine learning model 33 and the second machine learning model 34 are provided for each evaluation item, the face evaluation support system 1 can optimize the first machine learning model 33 and the second machine learning model 34 for each evaluation item.

[0171] As a result, the face evaluation support system 1 can evaluate the face F using the first machine learning model 33 and the second machine learning model 34 optimized for each evaluation item. Therefore, the face evaluation support system 1 can improve the evaluation accuracy for each evaluation item compared to when evaluation of the face F based on multiple evaluation items is performed using one first machine learning model and one second machine learning model.

[0172] The first machine learning model 33 is configured to calculate the probability for each evaluation category as a visible information evaluation value, while the second machine learning model 34 is configured to calculate the probability for each evaluation category as a thermal information evaluation value.

[0173] This configuration allows for detailed evaluation of the visible image data 35 and the thermal image data 36. Therefore, the face evaluation support system 1 can improve the evaluation accuracy of the evaluation results output by the output means.

[0174] In addition, the output means (terminal control unit 26) is configured to calculate the probability for each evaluation category based on the visible information evaluation value and the thermal information evaluation value, and to select the evaluation category with the highest probability from the calculated probabilities for each evaluation category as the evaluation result. According to this configuration, the evaluation category with the highest probability is output as the evaluation result, eliminating the need for the user to determine the evaluation category, thereby improving convenience for the user.

[0175] The first machine learning model 33, which is a visible information evaluation means, is a machine learning model that receives visible image data 35 as input information, evaluates the visible image data 35 based on evaluation items, and outputs a visible information evaluation value. On the other hand, the second machine learning model 34, which is the invisible information evaluation means, is a machine learning model that receives the thermal imaging data 36 as input information, evaluates the thermal imaging data 36 based on the evaluation items, and outputs a thermal information evaluation value.

[0176] According to this configuration, the first machine learning model 33 and the second machine learning model 34 can accurately evaluate the RGB values of the visible image data 35 and the RGB values of the thermal imaging data 36. As a result, the face evaluation support system 1 can further improve the evaluation accuracy of the evaluation result output by the output means, and can therefore evaluate the face F with higher accuracy.

[0177] Furthermore, since the output means (terminal control unit 26) is configured to output an evaluation result based on the average value of the visible information evaluation value and the thermal information evaluation value, the output means can construct an ensemble learning model using the first machine learning model 33 and the second machine learning model 34.

[0178] Therefore, the face evaluation support system 1 can improve the evaluation accuracy of the face F even when the evaluation accuracy of the first machine learning model 33 or the evaluation accuracy of the second machine learning model 34 is low.

[0179] In addition, the face evaluation support system 1 is equipped with a teacher data storage means (server memory unit 42) that stores the visible image data 35 as teacher data D1 for the first machine learning model 33 and stores the thermal image data 36 as teacher data D2 for the second machine learning model 34.

[0180] According to this configuration, the teacher data D1 and D2 can be diversified using visible image data 35 and thermal image data 36 obtained at the construction site, thereby further improving the evaluation accuracy of the first machine learning model 33 and the second machine learning model 34.

[0181] The face evaluation support system 1 also includes a processing means (server control unit 43) for performing predetermined processing on the visible image data 35 and the thermal image data 36 stored in the teacher data storage means.

[0182] According to this configuration, the diversity of the teacher data D1 and D2 can be further increased, and the evaluation accuracy of the first machine learning model 33 and the second machine learning model 34 can be further improved.

[0183] As a result, the face evaluation support system 1 improves the calculation accuracy of the visible information evaluation value calculated by the first machine learning model 33 and the calculation accuracy of the thermal information evaluation value calculated by the second machine learning model 34, and is therefore able to output more accurate evaluation results of the face F.

[0184] In addition, the face evaluation support system 1 is equipped with a display means (operation display unit 21) that displays visible image data 35 or thermal image data 36, and an operation reception means (operation display unit 21) that receives user operations to specify any point that forms the contour of the face F in the visible image data 35 or thermal image data 36 displayed on the display means.

[0185] Furthermore, the face evaluation support system 1 includes face contour generating means (terminal control unit 26) that generates a contour line L of the face F based on three or more arbitrary points designated by the operation receiving means.

[0186] According to this configuration, the face F, which is the evaluation target, can be easily extracted from the visible image data 35 and the thermal image data 36. As a result, the face evaluation support system 1 can reduce the user's effort in extracting the face F, thereby improving both user convenience and evaluation efficiency.

[0187] In addition, the face evaluation support system 1 is equipped with an area setting means (terminal control unit 26) that sets multiple cut-out areas E indicating parts of the face F for the visible image data 35 and thermal image data 36 based on the contour line L of the face F generated by the face contour generating means.

[0188] The first machine learning model 33 calculates a visible information evaluation value for each cutout area E of the visible image data 35, and the second machine learning model 34 calculates a thermal information evaluation value for each cutout area E of the thermal image data 36.

[0189] According to this configuration, the evaluation of the face F can be performed more efficiently than when the entire face F is the evaluation target. Furthermore, since it becomes possible to use the evaluation results according to the importance and priority of the cutting area E, the face evaluation support system 1 can improve convenience for users.

[0190] Furthermore, since the output means (terminal control unit 26) is configured to use the other as the evaluation result if it does not acquire one of the visible information evaluation value and the thermal information evaluation value, the face evaluation support system 1 can output the evaluation result of the face F even if it is unable to acquire, for example, either the visible image data 35 or the thermal image data 36.

[0191] Furthermore, the thermal imaging data 36 capturing the heat distribution of the working face F allows the working face evaluation support system 1 to more easily generate invisible image data (thermal imaging data 36) than invisible image data based on measurement results such as laser measurement or sonic measurement. Therefore, the working face evaluation support system 1 can reduce the effort required to acquire the thermal imaging data 36 and efficiently evaluate the working face F.

[0192] The face evaluation support system 1 also includes a mobile terminal 20 used by a user, and a server 40 connected to the mobile terminal 20 via a communication line 2. The mobile terminal 20 is equipped with a visible light camera 22 that captures visible information of the face F, a thermal camera 23 that captures invisible information of the face F, the above-mentioned visible image acquisition means (terminal control unit 26), invisible image acquisition means (terminal control unit 26), first machine learning model 33, second machine learning model 34 and output means (terminal control unit 26).

[0193] According to this configuration, the mobile terminal 20 used by the user is equipped with a visible light camera 22 and a thermal camera 23, so that visible image data 35 and thermal image data 36 can be easily obtained using the highly portable mobile terminal 20.

[0194] Furthermore, since the entire process from capturing the image of the face F to outputting the evaluation results of the face F can be performed on the mobile terminal 20, the face evaluation support system 1 can evaluate the face F even inside the tunnel T where communication conditions are unstable. As a result, the face evaluation support system 1 can reduce the burden on the user and can evaluate the face F more efficiently.

[0195] The mobile terminal 20 also includes a transmission means (terminal control unit 26) that transmits to the server 40 the evaluation result of the working face F output by the output means (terminal control unit 26). On the other hand, the server 40 includes a server storage means (server storage unit 42) that stores the evaluation results of the face F acquired from the mobile terminal 20.

[0196] According to this configuration, the evaluation results of the face F can be easily shared with a terminal other than the mobile terminal 20, for example, a management terminal 10 in a management office away from the construction site. Therefore, the face evaluation support system 1 can easily create a face observation record book using the evaluation results of the face F in the management terminal 10. This allows the face evaluation support system 1 to improve convenience for users.

[0197] In correspondence between the configuration of this invention and the above-mentioned embodiment, The visible image acquisition means, invisible image acquisition means, output means, face contour generation means, area setting means and transmission means of the present invention correspond to the terminal control unit 26 of the embodiment, Similarly, The invisible information corresponds to the RGB values that indicate the heat distribution. The invisible image data corresponds to the thermal image data 36; The visible information evaluation means corresponds to a first machine learning model 33; The invisible information evaluation value corresponds to the thermal information evaluation value, The invisible information evaluation means corresponds to a second machine learning model 34; The teacher data storage means and the server storage means correspond to the server storage unit 42, The processing means corresponds to the server control unit 43, The predetermined area corresponds to the cutout area E, The display means and operation reception means correspond to the operation display unit 21, The first imaging means corresponds to the visible light camera 22, The second imaging means corresponds to the thermal camera 23, The visible image acquisition process and the invisible image acquisition process correspond to steps S101 and S102. The visible information evaluation step corresponds to step S125. The invisible information evaluation step corresponds to step S126. The output process corresponds to steps S127 to S129. The present invention is not limited to the configurations of the above-described embodiments, and many other embodiments can be obtained.

[0198] For example, in the above-described embodiment, the client terminal used by the user M at the excavation site is the mobile terminal 20, but this is not limiting, and a tablet terminal or a laptop computer may also be used as the client terminal.

[0199] Furthermore, the appearance of the working face F was photographed using the visible light camera 22 of the mobile terminal 20, but this is not limited to this, and photographs may also be taken using a tablet terminal with a camera, a laptop computer with a camera, a digital camera, a video camera, or a wearable camera, as long as visible image data 35 can be obtained.

[0200] Furthermore, the heat distribution on the working face F was photographed using the thermal camera 23 of the mobile terminal 20, but this is not limited to this, and the photograph may be taken using a camera other than the mobile terminal 20 as long as the thermal imaging data 36 can be acquired.

[0201] Furthermore, the visible image data 35 and the thermal image data 36 are not limited to still images captured by the visible light camera 22 and the thermal camera 23, but may be still images cut out from a moving image. Furthermore, the evaluation items and evaluation categories in Table 1, the support screen 200 in FIG. 8, and the evaluation result screen 210 in FIG. 14 are each merely examples, and are not limited to these, and may have any appropriate configuration.

[0202] Furthermore, the visible image data 35 and thermal image data 36 used for evaluation are selected from the visible image data 35 and thermal image data 36 stored in advance in the terminal storage unit 24, but are not limited to this.

[0203] For example, instead of steps S101 and S102 of the evaluation support process, a guidance screen may be displayed prompting the user to photograph the working face F, and the working face F may be photographed by the user's operation to obtain visible image data 35 and thermal image data 36. Alternatively, the visible image data 35 and the thermal image data 36 captured by a camera other than the mobile terminal 20 may be acquired via the communication line 2 or a storage medium.

[0204] Furthermore, although the heat distribution of the face F is used as invisible information of the face F, this is not limited to this, and the density distribution or intensity distribution of the face F measured by laser measurement or ultrasonic measurement may also be used as invisible information of the face F.

[0205] In this case, the mobile terminal 20 acquires density distribution image data showing the density distribution of the face F generated based on the measurement results and intensity distribution image data showing the intensity distribution via the communication line 2 or a storage medium, and a cropped image obtained by cutting out a portion of the density distribution image data or intensity distribution image data is used as input information for the second machine learning model 34 in place of the thermal image crop data 38.

[0206] Furthermore, the visible image data 35 and the thermal image data 36 were each cut out at the cutout area E to evaluate the face F, but this is not limited to this, and the area surrounded by the contour line L generated in step S104 may be cut out and used as the image to be evaluated in the face evaluation process in step S108.

[0207] Furthermore, the first machine learning model 33 uses a convolutional neural network, but is not limited to this, and any appropriate estimation algorithm may be used as long as it is capable of evaluating the visible image data 35 based on the evaluation items and outputting a visible information evaluation value.

[0208] Similarly, the second machine learning model 34 uses a convolutional neural network, but is not limited to this, and any appropriate estimation algorithm may be used as long as it can evaluate the thermal imaging data 36 based on the evaluation items and output a thermal information evaluation value.

[0209] Furthermore, the first machine learning model 33 and the second machine learning model 34 are convolutional neural networks, but this is not limited to this, and the first machine learning model 33 and the second machine learning model 34 may be different machine learning models. In addition, in step S129, the evaluation category with the highest probability is determined as the evaluation result of the face F, but this is not limitative, and two evaluation categories may be determined as the evaluation result of the face F.

[0210] For example, if the evaluation category with the highest probability and the evaluation category with the next highest probability are adjacent evaluation categories and the difference in probability of falling into the evaluation categories is within a threshold range, both the evaluation category with the highest probability and the evaluation category with the next highest probability may be determined as the evaluation result of the face F. This allows the face evaluation support system 1 to evaluate the condition of the face F in more detail, thereby improving the evaluation accuracy of the face.

[0211] Furthermore, in the above-described embodiment, as shown in Figure 16(a), the evaluation result of the face F was calculated by averaging the visible information evaluation value output by the first machine learning model 33 and the thermal information evaluation value output by the second machine learning model 34, but this is not limited to this.

[0212] For example, in step S128, the weighted average of the probability that the visible information evaluation value corresponds to the evaluation category indicated by the visual information evaluation value and the probability that the thermal information evaluation value corresponds to the evaluation category may be calculated for each evaluation category and used as the face evaluation value. In this case, the first machine learning model 33, the second machine learning model 34, and the process of step S128 performed by the terminal control unit 26 can construct an ensemble learning model.

[0213] Alternatively, as shown in Figure 16(b), a third machine learning model 39 that receives as input the visible information evaluation value output by the first machine learning model 33 and the thermal information evaluation value output by the second machine learning model 34 may calculate a face evaluation value and determine the evaluation result of the face F.

[0214] In this case, the first machine learning model 33, the second machine learning model 34, and the third machine learning model 39 can be used to construct an ensemble learning model. According to the above-described configuration, even if the evaluation accuracy of the first machine learning model 33 or the evaluation accuracy of the second machine learning model 34 is low, the evaluation accuracy of the face F can be improved by the ensemble learning model.

[0215] Furthermore, in step S156, the visible image crop data 37 and the thermal image crop data 38 are processed by adding noise, correcting exposure, inverting, or rotating, but this is not limited to this, and the visible image crop data 37 and the thermal image crop data 38 may be processed in any suitable manner.

[0216] Furthermore, although the mobile terminal 20 is configured to perform the evaluation support processing (steps S101 to S110 in FIG. 9), this is not limited to this, and the management terminal 10 may be configured to perform the evaluation support processing after acquiring the visible image data 35 and thermal image data 36 from the mobile terminal 20 via the communication line 2 or a storage medium.

[0217] Alternatively, for example, the visible image data 35 and thermal image data 36 selected in step S102 may be sent to the server 40, and the server 40 may set the cut-out area E in step S104, save the cut-out image in step S106, and perform the face evaluation process in step S108. Even with this configuration, the same effects as those of the above-described embodiment can be achieved. [Explanation of symbols]

[0218] 1... Face evaluation support system 2. Communication lines 20...Mobile device 21...Operation display section 22...Visible light camera 23...Thermal camera 26...Terminal control unit 33…First machine learning model 34…Second machine learning model 35...Visible image data 36...Thermal imaging data 39…Third machine learning model 40…Server 42...Server memory section 43...Server control unit D1, D2...teaching data E...Cut area F...face T...tunnel

Claims

1. A tunnel face evaluation support system that supports evaluation based on evaluation items of a tunnel face, a visible image acquisition means for acquiring visible image data including visible information of the face; an invisible image acquisition means for acquiring invisible image data including invisible information of the face; a visible information evaluation means for evaluating the visible information of the visible image data based on the evaluation items and calculating a visible information evaluation value; an invisible information evaluation means for evaluating the invisible information of the invisible image data based on the evaluation items and calculating an invisible information evaluation value; and an output means for outputting an evaluation result of the face for the evaluation item based on the visible information evaluation value and the invisible information evaluation value. Face evaluation support system.

2. A plurality of the evaluation items are set for the face, The visible information evaluation means and the invisible information evaluation means are provided for each of the evaluation items. The face evaluation support system according to claim 1.

3. The evaluation items are set with a plurality of evaluation categories indicating the state of the face, The visible information evaluation means The appearance frequency for each evaluation category is calculated as the visible information evaluation value, The invisible information evaluation means The appearance frequency for each evaluation category is calculated as the invisible information evaluation value. The face evaluation support system according to claim 1.

4. The output means The appearance frequency for each evaluation category is calculated based on the visible information evaluation value and the invisible information evaluation value, and the evaluation category with the highest appearance frequency among the calculated appearance frequencies for each evaluation category is set as the evaluation result. The face evaluation support system according to claim 3.

5. The visible information evaluation means a machine learning model is constructed using the visible image data as input information, evaluating the visible image data based on the evaluation items, and outputting the visible information evaluation value; The invisible information evaluation means The invisible image data is input, and the invisible image data is evaluated based on the evaluation items. The invisible information evaluation value is output. The face evaluation support system according to claim 1.

6. The output means The evaluation result is output based on the average or weighted average of the visible information evaluation value and the invisible information evaluation value. The face evaluation support system according to claim 5.

7. The output means The visible information evaluation value and the invisible information evaluation value are input as information, and the machine learning model outputs the evaluation result based on the visible information evaluation value and the invisible information evaluation value. The face evaluation support system according to claim 5.

8. The visible image data is stored as training data for the visible information evaluation means, and the invisible image data is stored as training data for the invisible information evaluation means. The face evaluation support system according to claim 5.

9. A processing means is provided for performing a predetermined processing process on the visible image data and the invisible image data stored in the teacher data storage means. The face evaluation support system according to claim 8.

10. a display means for displaying the visible image data or the invisible image data; an operation receiving means for receiving an operation by a user to specify an arbitrary point that is a contour of the face in the visible image data or the invisible image data displayed on the display means; and a face contour generating means for generating a contour of the face based on the three or more arbitrary points designated by the operation receiving means. The face evaluation support system according to claim 1.

11. an area setting means for setting a plurality of predetermined areas indicating parts of the face for the visible image data and the invisible image data based on the face contour generated by the face contour generating means; The visible information evaluation means calculating the visible information evaluation value for each of the predetermined regions of the visible image data; The invisible information evaluation means The invisible information evaluation value is calculated for each of the predetermined regions of the invisible image data. The face evaluation support system according to claim 10.

12. The output means When one of the visible information evaluation value and the invisible information evaluation value is not obtained, the other evaluation value is used as the evaluation result. The face evaluation support system according to claim 1.

13. The invisible image data is thermal image data obtained by capturing an image of the heat distribution on the working face. The face evaluation support system according to claim 1.

14. A mobile device used by a user; a server connected to the mobile terminal via a communication line; The mobile terminal a first imaging means for imaging the visible information of the working face; a second imaging means for imaging the invisible information of the working face; The visible image acquisition means, the invisible image acquisition means, the visible information evaluation means, the invisible information evaluation means, and the output means are provided. The face evaluation support system according to claim 1.

15. The mobile terminal a transmitting means for transmitting the evaluation result of the face output by the output means to the server; The server A server storage means is provided for storing the evaluation results of the face acquired from the mobile terminal. The face evaluation support system according to claim 14.

16. A tunnel face evaluation support method for supporting evaluation based on evaluation items of a tunnel face, a visible image acquiring step of acquiring visible image data including visible information of the face by a visible image acquiring means; an invisible image acquisition step in which invisible image acquisition means acquires invisible image data including invisible information of the face; a visible information evaluation step in which a visible information evaluation means evaluates the visible information of the visible image data based on the evaluation items and calculates a visible information evaluation value; an invisible information evaluation step in which invisible information evaluation means evaluates the invisible information of the invisible image data based on the evaluation items and calculates an invisible information evaluation value; and an output step in which an output means outputs an evaluation result of the face for the evaluation item based on the visible information evaluation value and the invisible information evaluation value. A method for supporting face evaluation.

Images