Information processing device, information processing method, and program

Abstract

To efficiently achieve both maintenance and inspection work by using image processing technology and confirmation and correction work by a user.SOLUTION: In a system for determining deterioration and damage of members and structures from images, a determination result of the degree of damage to be determined is compared with known prior information regarding determination of the degree of damage so as to determine whether or not a user is required to confirm the determination result. In the case of the determination that the user is required to confirm the determination result, information on comparison contents is presented to the user as auxiliary information for confirmation work.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, social infrastructure such as tunnels and bridges have been aging, increasing the burden of maintenance and inspection work. Traditionally, maintenance and inspection work has been carried out by inspectors visiting tunnels, bridges, and other sites, visually inspecting the site and checking it against their expert knowledge. In recent years, there has been a significant shortage of workers with the expert knowledge to carry out inspections compared to the number of inspection targets, which is cited as one of the reasons for the increased burden of maintenance and inspection work.

[0003] Meanwhile, there is an image processing technology that uses a large amount of correct image data to train (optimize) the parameters of a multi-layer convolutional neural network (CNN) to accurately determine the type and condition of objects in an image. It is known that sufficient training can achieve accuracy in determination that exceeds that of humans, and this technology is becoming widely used in various fields that use images. Attempts to apply such image processing technology to the maintenance and inspection of social infrastructure are also underway. Specifically, it is considered that by photographing the exterior of structures such as bridges and tunnels and applying the image processing described above to the photographed images, it is possible to detect abnormalities such as cracks or water leaks on the exterior and to determine the degree of damage. Patent Document 1 discloses a technology for detecting abnormalities. As long as workers on-site can take images, they will no longer need the special skills they had in the past, and the assessment process can be automated using image processing, so if it is used properly, maintenance and inspection work on social infrastructure can be carried out efficiently. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-200512 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in social infrastructure inspections, the ways in which structures deteriorate and are damaged vary widely depending on the components, installation environment, purpose of use, etc., making it difficult to prepare a large amount of comprehensive correct answer data, which is important when applying the image processing technology mentioned above. For images that deviate significantly from the training data, it is not possible to output correct results, which can result in incorrect judgments, and users often need to check and correct the automatic judgment results.

[0006] Furthermore, computer performance is improving day by day, making it possible to process a large number of images, but this also means that the amount of checking and correction work required by users increases, as mentioned above. As a result, the increase in checking and correction work can become a bottleneck, and there is a possibility that the throughput will not improve sufficiently when using image processing technology for the maintenance and inspection of social infrastructure.

[0007] In view of the above-mentioned problems, the present invention has an object to efficiently perform maintenance and inspection work using image processing technology and confirmation and correction work by the user at the same time. [Means for solving the problem]

[0008] The information processing device according to the present invention comprises image acquisition means for acquiring an image of an object to be determined, damage degree determination means for determining a degree of damage to the object to be determined from the image of the object to be determined, prior information acquisition means for acquiring prior information regarding the object to be determined or the determination by the damage degree determination means, result determination means for comparing a determination result by the damage degree determination means with the prior information and determining whether or not confirmation of the determination result by a user is required, and output generation means for generating and outputting auxiliary information to assist the user in confirmation based on the compared information when it is determined by the result determination means that confirmation by the user is required.The damage level determination means outputs likelihoods of multiple damage levels as the determination result, and the prior information acquisition means acquires information indicating a threshold value for the likelihood difference between two of the multiple damage levels as prior information regarding the determination by the damage level determination means, and if one of the two damage levels has the highest likelihood and the likelihood difference between the two damage levels is compared and the likelihood difference is equal to or less than the threshold, the result determination means determines that confirmation of the determination result by a user is required. It is characterized by: [Effects of the Invention]

[0009] According to the present invention, maintenance and inspection work using image processing technology can be efficiently performed at the same time as confirmation and correction work by the user. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device. [Figure 2] FIG. 2 is a block diagram illustrating an example of a functional configuration of an information processing device. [Figure 3] 10 is a flowchart illustrating an example of a processing procedure of the information processing device. [Figure 4] 10A and 10B are diagrams for explaining an overview of damage level determination results and important class boundaries; [Figure 5] FIG. 10 is a diagram illustrating an example of presenting auxiliary information. [Figure 6] FIG. 10 is a diagram illustrating an example of presenting auxiliary information. [Figure 7] FIG. 10 is a diagram illustrating an example of presenting auxiliary information. [Figure 8] FIG. 10 is a diagram illustrating an example of presenting patches that have an inconsistency. [Figure 9] FIG. 10 is a diagram showing an example of presenting an initial value of the damage level. [Figure 10] FIG. 10 is a diagram showing a display example in which a past determination target is compared with a current determination target. [Figure 11] FIG. 10 is a diagram showing an example of an image captured so as to include a determination target. [Figure 12] FIG. 10 is a diagram showing an example of an image in which a determination target is cut out with the same position and size. [Figure 13] FIG. 10 is a block diagram illustrating an example of a functional configuration of an information processing device according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, a description will be given of an embodiment of the present invention with reference to the accompanying drawings. Note that the embodiment described below shows an example of a specific implementation of the present invention, and is one of the specific examples of the configuration described in the claims. (First embodiment) The hardware configuration of the information processing device according to this embodiment will be described with reference to the block diagram in Fig. 1. The information processing device according to this embodiment is realized by a single computer device, but it may also be realized by distributing each function among multiple computers as needed. When it is made up of multiple computers, they are connected by a LAN (Local Area Network) or the like so that they can communicate with each other.

[0012] In FIG. 1, information processing device 100 is realized by a single computer device, and has a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, an external storage device 104, an input device interface 105, an output device interface 106, a communication interface 107, and a system bus 108. The CPU 101 controls the entire information processing device 100. The ROM 102 stores programs and parameters that do not require modification. The RAM 103 temporarily stores programs and data supplied from external devices, etc.

[0013] The external storage device 104 is a storage device including a hard disk or memory card fixedly installed in the information processing device 100, or an optical disk such as a CD that is detachable from the information processing device 100, a magnetic or optical card, an IC card, a memory card, or the like. The input device interface 105 is an interface with an input device 109 such as a pointing device or keyboard that inputs data in response to a user's operation. The output device interface 106 is an interface with an output device 110 such as a monitor that outputs data held by the information processing device 100, supplied data, and program execution results. The communication interface 107 is a communication interface for connecting to a network 111 such as a WAN or LAN. The system bus 108 connects the components of the information processing device 100 so that they can communicate with each other.

[0014] Next, the functional configuration of the information processing device according to this embodiment will be described with reference to the block diagram of Fig. 2. The information processing device 100 has an image acquisition unit 201, a damage level determination unit 202, a prior information acquisition unit 203, a result determination unit 204, and an output generation unit 205. Here, the object of damage assessment in this embodiment (hereinafter also simply referred to as the assessment object) will be described with reference to Figs. 11 and 12. In this embodiment, the case where the object of damage assessment is a bolt on a bridge, tunnel, or the like that is being inspected will be described as an example. Many bolts are usually used on bridges, tunnels, etc., and Fig. 11 is an example of an image of a portion of them. On-site workers will photograph the entire bridge or tunnel that is being inspected, gradually shifting the imaging range so as to cover all the bolts. As a result, many images like the one in Fig. 11 will be taken.

[0015] In this embodiment, images of the individual bolts to be determined are further extracted in advance from the image obtained in this manner, such as that shown in FIG. 11, to create images. FIG. 12 is an example of an image extracted from the bolt 1101 in the image of FIG. 11. As shown in FIG. 12, the bolt to be determined is roughly centered, and the image is extracted with a roughly uniform size. It is generally known that the accuracy of determination by image processing is improved by using an input image whose position and size have been normalized, as shown in FIG. 12. Therefore, in this embodiment, the images are extracted in advance, and the extracted images are successively saved and managed in the external storage device 104 as target images for damage level determination (hereinafter also referred to as determination images).

[0016] In this way, it is assumed that a large number of images for determination are stored and managed in the external storage device 104, and that the damage level of each of the objects to be determined is determined in the functional configuration shown in FIG. Next, each functional unit will be described. The image acquisition unit 201 acquires an image for determination from the external storage device 104. As described above, in this embodiment, the object for determining the degree of damage is a component such as a bolt, and an image of the object for determination is acquired, which has been cut out in advance. However, this is not limited to this, and a configuration may also be adopted in which an image showing multiple components is acquired as shown in FIG. 11, and the damage degree determination unit 202, which will be described later, internally performs processing to detect and cut out each component such as a bolt 1101, and determine the degree of damage for each component. In this case, the image acquired here does not need to be an image that has been cut out in advance.

[0017] The damage level determination unit 202 determines the damage level of the determination image acquired by the image acquisition unit 201, and outputs a damage level determination result (hereinafter simply referred to as a determination result). The damage level determination can be performed, for example, by preparing a large amount of training image data to which correct damage levels have been assigned, and training in advance a multi-class classifier using a model configured with a multi-layered neural network. Note that the multi-layered neural network model may be a known network in which CNNs are multi-layered, such as VGG or ResNet. However, the model is not limited to CNNs, and is not particularly limited as long as it can receive an image as input and output a damage level determination result to be used by the result determination unit 204, which will be described later.

[0018] In this embodiment, the damage level is classified into three classes, A to C. A is the highest damage level, and C is the lowest damage level. When training a CNN model such as the one described above, a method is known in which the output layer is trained to output a likelihood distribution for each class. This method, which uses cross-entropy loss as a loss function, is a common method for multi-class classification using CNN, and this embodiment also adopts a configuration that follows this method.

[0019] The prior information acquisition unit 203 acquires known prior information regarding the target of damage level assessment or the assessment result. In this embodiment, the prior information includes information regarding the location of important class boundaries when multi-class discrimination is performed by the damage level assessment unit 202, and threshold information on the likelihood difference regarding the class boundaries (hereinafter, both will be collectively referred to as important class boundary information). As will be described later, the threshold information is used to determine whether there is a difference equal to or greater than a threshold between two classes at the important class boundary. In classes with an order such as damage level, the likelihood difference tends to be small in cases where it is difficult to judge between adjacent classes. It can be said that it is unclear which of the adjacent classes is correct, which increases the need for user confirmation.

[0020] The important class boundary information may change each time depending on the purpose of use of the damage level assessment results by the user. For example, it may be a class boundary or threshold for determining whether or not to perform a re-inspection of maintenance or inspection, or a class boundary or threshold for determining whether or not there is an obligation to include it in a report, and therefore may be set sequentially according to the purpose of use. The advance information may be stored in advance in a storage device such as ROM 102, RAM 103, or external storage device 104, or may be input by the user using input device 109 at the time of use and sequentially acquired via input device interface 105.

[0021] The result determining unit 204 compares the damage level determination result from the damage level determination unit 202 with the prior information acquired by the prior information acquisition unit 203, and determines whether or not the determination result needs to be confirmed by the user. In this embodiment, the result determining unit 204 compares the damage level determination result with important class boundary information acquired as prior information. Then, if the determination result is one of two classes adjacent to the important class boundary and the likelihood difference between the two classes is equal to or less than a threshold, the result determining unit 204 determines that the determination result needs to be confirmed by the user.

[0022] The determination process of the result determination unit 204 will be specifically described using the example of Fig. 4. Fig. 4 is a diagram showing an example of the output from the damage level determination unit 202. As described above, in this embodiment, the damage level determination unit 202 outputs the damage level using the likelihood of three classes A to C. In this embodiment, the important class boundary is boundary 401, and classes A and B are adjacent to this boundary. The likelihood difference set as the threshold is 0.2. In the example of Figure 4(a), class A, which is one end of the important class boundary, has the highest likelihood of 0.9, so class A is the damage level determination result. Since class B, which is the other end of the important class boundary, has a likelihood of 0.08, the likelihood difference between class A and class B is 0.82, which is greater than the threshold likelihood difference of 0.2, so the result determination unit 204 determines that the user does not need to confirm the determination result.

[0023] 4(c), the likelihood of class A is 0.5, which is the maximum likelihood, and therefore class A is the damage level determination result. The likelihood of class B is 0.4, and therefore the likelihood difference between class A and class B is 0.1, which is smaller than the threshold likelihood difference of 0.2, and therefore the result determination unit 204 determines that the user needs to confirm the determination result. On the other hand, in the example of Figure 4(b), the likelihood of class B is 0.5, which is the maximum likelihood, and therefore class B is the damage level determination result. Here, although the likelihood difference between class B and class C is only 0.1, class C is not a class adjacent to the important class boundary, so the result determination unit 204 determines that the user does not need to confirm the determination result. In the case of Figure 4(b), even if class C is actually the correct determination result, it is considered to have little impact on the user because it is not a class adjacent to the important class boundary.

[0024] The output generation unit 205 generates auxiliary information (hereinafter also referred to as auxiliary information) to assist the user in checking and correcting the determination result according to the determination result of the result determination unit 204, and presents the auxiliary information to the user. Here, the auxiliary information is information on the content of a comparison between the damage level determination result and the prior information, and in this embodiment, it is information that shows the likelihood difference of the important class boundary, for example, as shown in Figure 4(c).

[0025] Here, an example of a method for presenting auxiliary information will be described with reference to FIGS. Fig. 5 is a list of damage level determination results. Damage levels 501 and 502 are the damage levels corresponding to the determination results shown in Fig. 4(a) and Fig. 4(b), respectively, and damage level 503 is the damage level corresponding to the determination result shown in Fig. 4(c). The output generation unit 205 can present auxiliary information to the user by displaying a list such as that shown in Fig. 5 on the output device 110.

[0026] That is, when the result determining unit 204 determines that confirmation of the determination result by the user is not required, the damage degree determination result is presented as is, such as damage degree 501 and damage degree 502. In contrast, in this embodiment, when the likelihood difference at the important class boundary is equal to or less than the threshold and the result determining unit 204 determines that confirmation by the user is required, the likelihood of two classes adjacent to the important class boundary is presented, such as damage degree 503. The information shown in damage degree 503 is auxiliary information. This allows the user to proceed with the confirmation of the judgment result for the damage level judgment target displayed as shown in damage level 503, taking into consideration the possibility that the other class is correct.

[0027] As shown in FIG. 5, a judgment image used by the damage level judgment unit 202, which corresponds to the above-described FIG. 12, may also be presented so that the user can view the judgment target and confirm the judgment result. In this way, the user can efficiently check the judgment results of a large number of judgment targets. To further improve the efficiency of the checking work, judgment results that require user confirmation may be highlighted so that the user can focus on them from among the many judgment results. For example, the judgment results can be highlighted by using different colors, such as black for normal judgment results and red for judgment results that require confirmation, by using bold or large letters, or by highlighting them with graphics or animations. In addition, in a list such as that shown in FIG. 5, the determination results that are determined to require confirmation by the user may be given priority and displayed in order from the top.

[0028] 5 shows a list of the determination results, it may be difficult for the user to confirm the determination results unless the user looks at the image to be determined more closely, etc. Therefore, for example, details of each determination result as shown in FIG. 6 may be displayed on the output device 110 to provide the user with supplementary information. In the detailed judgment result screen 601, image 602 is a cropped image of the judgment target. Pane 603 displays the likelihood of each damage level class as auxiliary information for the judgment target, along with check boxes for correction. Image 604 is an image including the periphery of the judgment target, with the judgment target surrounded by a dashed frame. This allows the user to confirm and correct the judgment results while viewing the situation around the judgment target.

[0029] If a user checks the damage level and determines that the damage level assessment result is incorrect, the damage level assessment result needs to be corrected. In this case, providing a mechanism that allows the user to select the content of the correction, as shown in pane 603, can reduce the overall burden of the checking and correction work. In particular, as shown in FIG. 4(c), when the likelihood difference is small at the important class boundary of the damage level assessment result, the user may correct the damage level to another class, for example, class B in FIGS. 5 and 6. Therefore, presenting these as candidate options can assist in the work when correction is necessary. In this way, any information that can improve the efficiency of the checking and correction work depending on the content compared by the result determination unit 204 may be presented as auxiliary information.

[0030] Also, damage level assessment results that are determined to require confirmation may be presented with priority. For example, the damage level assessment results for individual assessment targets as shown in Fig. 6 are displayed in order starting with the assessment results determined by result assessment unit 204 to require user confirmation. Then, when the user selects previous button 605 or next button 606, the display of image 602, pane 603, and image 604 is switched, allowing the user to confirm or correct another assessment result. Note that the individual assessment results as shown in Fig. 6 may be displayed, for example, by the user selecting the assessment result that the user wants to confirm or correct in the list of Fig. 5. The above description does not limit the method of presenting auxiliary information, but is merely an example of a method of presenting the comparison result of the damage level and the prior information to the user as auxiliary information. Any display may be used as long as it conceptually includes these display contents.

[0031] Next, the processing procedure in the information processing device according to this embodiment will be described with reference to the flowchart in Fig. 3. As mentioned above, in this embodiment, there are many images cut out from which damage level determination is to be performed, and this flow is a flow for performing damage level determination on one of these images. The flow shown in Fig. 3 is repeated for the number of images to be subjected to damage level determination, and the results obtained are integrated to produce a final display such as that shown in Fig. 5 or Fig. 6. First, in step S301, the image acquisition unit 201 acquires an image of a target for damage level determination from the external storage device 104. In this embodiment, as described above, the target for damage level determination is a member such as a bolt, and the image acquisition unit 201 acquires an image of the target for determination cut out.

[0032] In step S302, the damage level determination unit 202 receives the determination image acquired in step S301 as an input, determines the damage level of the object to be determined, and outputs the determination result. In this embodiment, as described above, it is assumed that the damage level determination unit 202 has already learned to determine the likelihood that the object to be determined belongs to one of the three classes A to C representing the damage level, and to output the likelihood as the damage level determination result. In step S303, the prior information acquisition unit 203 acquires known prior information related to damage level determination of the object to be determined. In this embodiment, as described above, important class boundary information is acquired. Here, as in the example shown above, the important class boundary is between class A and class B, and the likelihood difference threshold is 0.2.

[0033] In step S304, the result determination unit 204 receives the damage level determination result and the prior information obtained in steps S302 and S303 as input, and compares the two to determine whether or not confirmation of the damage level determination result is necessary. In this embodiment, as described above, when the output of the damage level determination unit 202 is as shown in FIG. 4(c), the likelihood of class A, which is one end of the important class boundary, is 0.5, the maximum likelihood, and the likelihood of class B, which is the other adjacent class at the important class boundary, is 0.4. Since the likelihood difference between class A and class B is 0.1, which is smaller than the likelihood difference threshold of 0.2 obtained as important class boundary information, the result determination unit 204 determines that confirmation of the damage level determination result by the user is necessary.

[0034] If the result determination unit 204 determines in step S305 that the user needs to confirm the damage level determination result, the process proceeds to step S306. On the other hand, if the result determination unit 204 determines that confirmation is not needed, the process of this flow ends, the damage level determination result is output, and the process of this flow is repeated for the remaining images as described above. In step S306, the output generation unit 205 acquires the results of the comparison in step S304 and outputs them as auxiliary information to assist the user in the confirmation work. In this embodiment, as described above, information such as the output likelihood at the important class boundary is displayed on the screen where the user confirms the damage level determination result, for example, as shown in damage level 503 in Fig. 5.

[0035] As described above, the information processing device compares the damage level determination result with the important class boundary information acquired as prior information, and if the likelihood difference at the important class boundary is equal to or less than the threshold, determines that confirmation by the user is necessary and presents information such as the likelihood to the user as auxiliary information. This is expected to make it easier to avoid serious problems that could be caused by incorrect determination of the damage level at the important class boundary and that could adversely affect the intended use.

[0036] (Second embodiment) In the following embodiments, the description of the parts that overlap with the first embodiment will be omitted, and only the parts that are different from the first embodiment will be described. Furthermore, the hardware configuration of the information processing device according to the following embodiments is the same as the hardware configuration of the information processing device according to the first embodiment. In the first embodiment, the prior information acquisition unit 203 acquires important class boundary information as prior information. In contrast to this, in the present embodiment, for example, damage level assessment information for each assessment target is stored and managed in the external storage device 104. Then, in step S303 of FIG. 3, the prior information acquisition unit 203 acquires past damage level assessment information for the same assessment target as prior information. Here, the past damage level assessment information is, for example, information such as the class representing the damage level in the past assessment and the date and time when the damage level assessment was performed. Note that if damage level assessments have been performed multiple times in the past, only the damage level assessment information performed most recently may be acquired as prior information. At this time, in steps S304 and S305, the result determination unit 204 compares the current damage level determination result with the prior information, and determines that confirmation is necessary if there is an inconsistency in aging as shown below. For example, if the current damage level class is improved compared to the previous damage level determination result even though no repairs or the like have been performed, the result determination unit 204 may determine that there is an inconsistency and that confirmation of the determination result by the user is necessary.

[0037] In addition, in the case of damage level classes that have an order relationship, it is possible for changes to occur to adjacent classes, so it may be possible to determine that there is an inconsistency if improvements have been made to classes that are more than a certain number of classes away. Furthermore, since there is a possibility that repairs may result in improvement, when the results of damage assessment targets are saved and managed, it is possible to also manage whether or not repairs have been performed, and when acquiring the results as preliminary information, it is also possible to acquire whether or not the repairs have been performed. In this case, the result assessment unit 204 may determine that there is an inconsistency when comparing with the preliminary information if the damage class has improved even though there is no record of repairs being performed. This allows the user to limit the targets to be checked, which is expected to further improve the user's work efficiency.

[0038] Note that the inconsistency with past damage level assessment results is not limited to the above inconsistency. For example, the general progression of damage level deterioration and its period of time may be set in advance in the result assessment unit 204. The result assessment unit 204 may determine that the change in damage level class and the elapsed time are inconsistent with the normal deterioration and damage process and that confirmation by the user is required if, for example, the deterioration progresses in a shorter period than the set period between the change in damage level class and the elapsed time.

[0039] If a comparison of the damage level assessment result and the prior information reveals an inconsistency, in step S306, the output generation unit 205 presents the information from the previous damage level assessment used in the comparison as auxiliary information. For example, for a list like that shown in FIG. 5 described in the first embodiment, the output shown in FIG. 7 can be generated. That is, in addition to the current damage level assessment result class "C," the damage level 701 presents the date and time of the previous damage level assessment (2020.09.30) and the previous assessment result class "A." This allows the user to compare the previous assessment result with the current assessment result and efficiently learn that the current damage level assessment class has naturally improved compared to the previous damage level assessment, i.e., that the damage level class "A" has become "C."

[0040] The output method is not limited to the list display shown in Figure 5, but may be an individual display as shown in Figure 6 described above. Alternatively, if the inconsistency is with the results of past damage assessments and the assessment images used in those past assessments can be obtained, a comparison display as shown in Figure 10 may be used. Since changes over time are generally important in inspection work, images used in past damage assessments are usually saved and managed together with those results as part of those assessments. Therefore, it is sufficient to acquire those saved and managed images. Pane 1001 in Figure 10(a) displays the target image and assessment results for the current damage assessment, while pane 1002 displays the image from a past damage assessment of the same target, the assessment results, the date the assessment was performed, whether repairs were performed, etc.

[0041] Although the pane 1001 presents class "C" as the current damage assessment result, this assessment result may be incorrect and the user may wish to correct it. FIG. 10(b) illustrates a pull-down list 1003 for correcting the damage assessment result. This pull-down list 1003 presents potential damage assessment results to the user, allowing the user to easily correct the damage level by selecting one. The potential damage assessment results in the pull-down list may be displayed in the top position of the damage level class that the user is likely to select, allowing the user to easily select one. For example, in this embodiment, auxiliary information is provided to allow the user to confirm and correct inconsistencies in the results, such as improvement over time despite no repairs or other modifications. Therefore, at least the class from the previous assessment is presented at the top of the pull-down list. The pull-down list 1003 in FIG. 10(b) illustrates the previous damage level class "A" displayed at the top of the pull-down list.

[0042] (Third embodiment) In the first embodiment, an example has been described in which an image showing a bolt, which is a component for which damage level determination is to be performed, is extracted in advance by a human operation and stored and managed in the external storage device 104. In this case, it can be said that the component for which damage level determination is to be performed is always included in the image. Therefore, in this embodiment, the prior information acquisition unit 203 acquires, as prior information, information that the component for determination is always present in the image (hereinafter referred to as presence information). There are other cases where it can be said that a target for damage assessment is necessarily present in an image, such as when there are work constraints imposed when acquiring an image, such as when a photographer photographing a component on-site always photographs the component so that it is at the center. In cases where an image of a target for assessment is extracted in advance by a human operation or when there are work constraints, the presence information is known in advance, and therefore, in this embodiment, the image acquired is associated with the presence information in advance and stored and managed in the external storage device 104. This allows the prior information acquisition unit 203 to acquire the presence information as prior information in step S303 of FIG. 3 .

[0043] In this embodiment, the result determination unit 204 acquires existence information as prior information, but there are cases where the damage level determination result of the determination target is not obtained from the damage level determination unit 202. In such cases, an inconsistency occurs in which no determination result is obtained even though the determination target exists. Therefore, in step S304, the result determination unit 204 determines that the user needs to confirm the determination result when such an inconsistency occurs.

[0044] The damage level determination unit 202 described in the first embodiment outputs a determination result assuming that the damage level of the determination target belongs to some class. In this embodiment, as shown in FIG. 13 , the damage level determination unit 202 further includes a detection unit 1301 that identifies the position, etc., containing the determination target from the input image. If the detection unit 1301 cannot detect the determination target from the determination image, the damage level determination unit 202 determines that a damage level determination result has not been obtained. Like the damage level determination unit 202, the detection unit 1301 is also generally well known as one that uses the above-mentioned CNN (such as Faster R-CNN, SSD, or YOLO), and any such general detector will suffice. As mentioned above, damage to infrastructure and other structures may occur in ways that could not be predicted during learning, depending on the type of components and the environment. In such cases, the detection unit 1301 fails to detect the damage, and no damage level determination result can be obtained.

[0045] In step S306, the output generation unit 205 outputs, as auxiliary information, a damage level assessment result not obtained despite the input of a judgment image in which a target should be present. The display may be similar to the example in FIG. 5 described in the first embodiment in that the target is distinguished from others by highlighting it to make it easier for the user to find the target among the many damage level assessment results. In contrast, in this embodiment, the damage level assessment result is not yet determined and the user must confirm and enter one of the classes. Therefore, a display that allows the user to input is provided, such as the pull-down list 901 in FIG. 9(a). Note that because the image was too damaged to detect a target even though it should have been present, the target may be in an unexpected state. Therefore, the highest damage level may be presented as an initial value. In FIG. 9(a), class A, which has the highest damage level, is presented in advance. Note that, when the pull-down list 901 is expanded, the results may be displayed in descending order of damage level, as shown in FIG. 9(b).

[0046] As described above, when there are a large number of damage level assessment results for the items being checked by the user, the task of checking and correcting the damage level can be a heavy burden for the user. Therefore, by presenting an initial value, it is expected that the burden on the user will be further reduced.

[0047] (Fourth embodiment) In the third embodiment, an example was described in which presence information was obtained as prior information by using an image previously extracted by a human operation as the target image for damage assessment. In contrast, in the present embodiment, for example, by extracting an image range corresponding to a component part shown on a drawing of the structure to be inspected from an image associated with the drawing and using it as a judgment image, presence information indicating that the target object should be captured in the image is obtained as prior information. Subsequent processing can be performed in the same manner as in the third embodiment. The method for associating the drawing and image is not particularly limited, and may be performed manually in advance, or may be performed by capturing an image that includes identifiable landmarks, or may be performed based on the coordinates of the photographing position and the camera orientation.

[0048] (Fifth embodiment) In the first embodiment, an example of determining the damage level of a bolt as a component to be determined for damage level has been described, but the determination target is not limited to parts such as bolts. For example, the damage level determination unit 202 may determine the damage level of a concrete wall surface. In this embodiment, an example of determining the damage level of a concrete wall surface will be described with reference to the flowchart in FIG. 3 and FIG. 8.

[0049] Image 801 in FIG. 8(a) represents the entire concrete wall surface, and division lines are displayed to divide image 801 into portions of a size that allows for damage assessment. Hereinafter, the divided images will be referred to as patch images. In the example of FIG. 8(a), patch images are obtained that are divided into 4 vertical and 6 horizontal images. In this embodiment, deformed portions due to cracks, water leakage, etc. are detected in advance from the patch images of the wall surface. Note that detection of deformed portions is performed in advance before starting the processing of FIG. 3, and the detection process may be performed by the information processing device 100 or by another device. The detection results of deformed portions (hereinafter, "deformation information") are stored and managed in the external storage device 104.

[0050] In step S301, the image acquisition unit 201 acquires a patch image as a determination target from the external storage device 104 and also acquires deformation information. Then, in step S302, the damage degree determination unit 202 determines the damage degree of the wall surface based on the deformation information. In FIG. 8(a), deformation 802 is a detected crack deformation, and in step S303, the prior information acquisition unit 203 acquires the detected deformation information separately as prior information. Then, in step S304, the result determination unit 204 acquires the damage degree determination result and the deformation information and compares them.

[0051] In this embodiment, in step S305, the result determination unit 204 determines that the user needs to confirm the determination result if the damage level is below the threshold, such as when the output of the damage level determination unit 202 indicates "no damage" despite the detection of a defect. The auxiliary information presented by the output generation unit 205 at this time is the result of the comparison. For example, as shown in FIG. 8(b), the damage level determination results for each patch on the entire wall surface are superimposed on the defect information. Patches determined to be "damaged" by the damage level determination unit 202, such as patch 803, are highlighted by coloring. Patch 804 does not have the same coloring as patch 803, despite the presence of cracking or deformation 802. This indicates that the determination was "no damage," and the user need only check the affected area.

[0052] To make it easier to find the relevant part among many patches, a highlighting display may be added to distinguish between patches that require confirmation and those that do not. For example, like patch 804 in Figure 8(b), patches that require confirmation may be highlighted by surrounding them with a thick frame. This makes it easier for the user to check whether there has been an error in the assessment of the damage level of the wall surface.

[0053] Although the example of the inconsistency where a "no damage" judgment was made even though a deformity was detected was described above, the present invention is not limited to this. In the case where a "damage" judgment is output even though no deformity was detected, this is also something that the user should check, so the same highlighting as above may be performed.

[0054] As described above, according to this embodiment, even for damage to a concrete wall, auxiliary information for the user to check and correct can be obtained based on the damage degree determination result and the prior information. Note that although this embodiment has been described using a concrete wall as an example, it can also be applied to other wall surfaces, such as mortar, as long as the wall surface is susceptible to deformation such as cracks and water leakage.

[0055] (Sixth embodiment) In the first embodiment, the likelihood difference between two classes adjacent to an important class boundary is used to determine whether or not the user needs to confirm the damage level determination result. As described above, if the likelihoods of adjacent damage level classes in an ordered relationship are close, the determination result by the damage level determination unit 202 may be ambiguous. On the other hand, even if the classes are not adjacent but distant, if the likelihood difference between the classes is small, the ordered damage level determination may not have been performed correctly. Therefore, a threshold value for the likelihood difference between non-adjacent classes may be acquired as prior information. For example, the likelihood of the highest likelihood class, which is the determination result, and the likelihood of non-adjacent classes may be calculated, and if the likelihood difference between the classes is equal to or less than the threshold, the result determination unit 204 may determine that the user needs to confirm the result.

[0056] (Seventh embodiment) In the first embodiment, an example was described in which the likelihood difference of the important class boundary is presented as auxiliary information. By presenting the likelihood difference, the user can know how ambiguous the judgment result is, and this is useful as auxiliary information when checking the judgment result. However, some users may not need to be presented with the likelihood difference, and knowing that the judgment result is ambiguous may be sufficient as auxiliary information. In such cases, for example, it may be possible to present only the two ambiguous classes, or to present a message such as "the damage level judgment result is ambiguous."

[0057] (Eighth embodiment) In the above-described embodiment, the process of presenting each type of auxiliary information for each type of advance information has been described, but the auxiliary information may be presented in a mixed form. Hereinafter, an example of the process of presenting multiple types of auxiliary information will be described with reference to the flowchart in FIG. 3, the prior information acquisition unit 203 acquires multiple types of prior information. For example, important class boundary information and existence information are acquired as the prior information. Then, in steps S304 and S305, the result determination unit 204 compares each of the multiple types of prior information with the determination results and determines whether the user needs to confirm the determination results. In step S306, the output generation unit 205 creates auxiliary information for a determination result that is determined to require user confirmation. When generating multiple types of auxiliary information, the output generation unit 205 may include information in the auxiliary information that allows the user to distinguish how the auxiliary information was obtained. For example, the output generation unit 205 includes information such as "The damage level determination result is vague" for an object that requires confirmation when the prior information is important class boundary information, and "The existence of the object to be determined must be confirmed" for an object that requires confirmation when the prior information is existence information, and presents the information in a distinguishable manner.

[0058] The method of presentation is not limited, and the above-mentioned text information may be displayed for differentiation, or different colors may be assigned to the text information for differentiation. Also, different icons may be assigned to the information to be presented, and the icons may be used for differentiation. This allows the user to know from what perspective they should check the target, which is expected to improve work efficiency.

[0059] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0060] 201 Judgment image acquisition unit, 202 Damage degree determination unit, 203 Prior information acquisition unit, 204 Result determination unit, 205 Output generation unit

Claims

1. image acquisition means for acquiring an image of a determination target; a damage level determination means for determining a damage level of the object to be determined from an image of the object to be determined; a prior information acquisition means for acquiring prior information regarding the object to be determined or the determination by the damage level determination means; a result determining means for comparing the result of the damage level determining means with the prior information and determining whether or not confirmation of the result of the damage level determining means by a user is necessary; an output generating means for generating and outputting auxiliary information for assisting the user in making a confirmation based on the compared information when the result determining means determines that confirmation by the user is necessary; and the damage level determination means outputs a plurality of likelihoods of damage levels as the determination results; the prior information acquisition means acquires, as prior information related to the determination by the damage level determination means, information indicating a threshold value of a likelihood difference between two of the plurality of damage levels; An information processing device characterized in that if either of the two damage levels has the highest likelihood and, by comparing the likelihood difference between the two damage levels, the likelihood difference is less than the threshold value, the result determination means determines that the user needs to confirm the determination result.

2. The information processing device described in Claim 1, characterized in that the prior information acquisition means acquires, as prior information regarding the judgment by the damage degree judgment means, information indicating a predetermined boundary among the boundaries of the multiple damage degrees and a threshold value of the likelihood difference between two damage degrees adjacent to the predetermined boundary.

3. 3. The information processing apparatus according to claim 2, wherein the output generating means generates and outputs information indicating two damage levels adjacent to the boundary and their likelihoods.

4. 4. The information processing apparatus according to claim 1, wherein the output generating means generates and outputs information for presenting a message to the effect that the determination result is ambiguous.

5. The image acquisition means acquires images of a plurality of determination targets, the damage level determination means determines a damage level of each of the plurality of determination targets from an image of the plurality of determination targets; the result determination means determines whether or not confirmation of the determination result by a user is required for each of the plurality of determination targets; The information processing device according to any one of claims 1 to 4, characterized in that the output generation means outputs auxiliary information that presents, in priority, the judgment results that the result judgment means has determined require user confirmation.

6. The image acquisition means acquires images of a plurality of determination targets, the damage level determination means determines a damage level of each of the plurality of determination targets from an image of the plurality of determination targets; the result determination means determines whether or not confirmation of the determination result by a user is required for each of the plurality of determination targets; The information processing device described in any one of claims 1 to 4, characterized in that the output generation means outputs auxiliary information that highlights and presents the judgment result that the result judgment means has determined requires user confirmation.

7. an image acquisition step of acquiring an image of a determination target; a damage degree determination step of determining a damage degree of the object to be determined from an image of the object to be determined; a prior information acquisition step of acquiring prior information regarding the object to be determined or the determination by the damage degree determination step; a result determination step of comparing a determination result from the damage degree determination step with the prior information and determining whether or not confirmation of the determination result by a user is necessary; an output generating step of generating and outputting auxiliary information for assisting the user in making a confirmation based on the compared information when it is determined in the result determining step that confirmation by the user is necessary; and the damage degree determination step outputs a plurality of likelihoods of damage degrees as the determination results; the prior information acquisition step acquires, as prior information related to determination in the damage level determination step, information indicating a threshold value of a likelihood difference between two damage levels among the plurality of damage levels; An information processing method characterized in that if either of the two damage levels has the highest likelihood and the likelihood difference between the two damage levels is compared and the likelihood difference is less than the threshold, the result determination process determines that the user needs to confirm the determination result.

8. an image acquisition step of acquiring an image of a determination target; a damage degree determination step of determining a damage degree of the object to be determined from an image of the object to be determined; a prior information acquisition step of acquiring prior information regarding the object to be determined or the determination by the damage degree determination step; a result determination step of comparing a determination result from the damage degree determination step with the prior information and determining whether or not confirmation of the determination result by a user is necessary; an output generating step of generating and outputting auxiliary information for assisting the user in making a confirmation based on the compared information when it is determined in the result determining step that confirmation by the user is necessary; on the computer, the damage degree determination step outputs a plurality of likelihoods of damage degrees as the determination results; the prior information acquisition step acquires, as prior information related to determination in the damage level determination step, information indicating a threshold value of a likelihood difference between two damage levels among the plurality of damage levels; A program characterized in that, when either of the two damage levels has the highest likelihood and the likelihood difference between the two damage levels is compared and the likelihood difference is less than the threshold, the result determination process determines that the user needs to confirm the determination result.

Images