Learning data collection device, learning data collection method, and program

Abstract

To provide a learning data collection device, a learning data collection method, and a program for collecting learning data that enables efficient re-learning.SOLUTION: The learning data collection device includes: an inspection image acquisition unit for acquiring inspection images; an area detection result acquisition unit for acquiring area detection results detected by an area detector that has been subjected to learning; a correction history acquisition unit for acquiring correction history of the area detection results; a calculation unit for calculating correction quantification information obtained by quantifying the correction history; a database for storing the inspection images, area detection results, and the correction history in association with one another; an image extraction condition setting unit for setting a threshold value of the correction quantification information as an extraction condition for extracting inspection images to be used for re-learning from the database; and a first learning data extraction unit for extracting inspection images that satisfy the extraction conditions as learning data for causing the area detector to be subjected to re-learning.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a learning data collection device, a learning data collection method, and a program, and particularly to a learning data collection device, a learning data collection method, and a program for extracting learning data for re-learning a region detector.

Background Art

[0002] In recent years, an object to be inspected may be inspected using a photographed image of the object. The inspection of the object is performed by detecting and specifying a predetermined region from the photographed image of the object. For example, when performing a damage inspection of a structure, a photographed image of a structure such as a bridge, a road, or a building, which is the object to be inspected, is acquired. Then, the damage of the structure is detected and specified by image processing from the photographed image.

[0003] Here, as one method of detecting a region from a photographed image by image processing, a method of detecting a region by a region detector (for example, a damage detector) that has been machine-learned is known. It is also known that the accuracy of region detection can be improved by newly re-learning (or additional learning) a region detector that has been machine-learned once. However, if all the huge number of images accumulated in a database or the like are used as learning data and learned by the region detector, the learning time will become extremely long. Further, if appropriate data is not learned as learning data, a high learning effect cannot be expected. That is, simply randomly selecting learning data cannot cause the region detector to perform efficient learning.

[0004] Therefore, conventionally, for the purpose of performing efficient learning when performing machine learning, proposals have been made regarding a method for extracting learning data.

[0005] For example, Patent Document 1 describes a technique aimed at reliably and rapidly improving the identification accuracy of images. Specifically, it describes a technique of using, for machine learning, an image other than the images used in past machine learning and having a low similarity to the images used in past machine learning.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] Here, when performing area detection of an object by an area detector that has been machine-learned, in order to perform efficient machine learning, it is necessary to perform re-learning using inspection images that the user was not satisfied with in the area detection results obtained by the area detector.

[0008] On the other hand, the user may make minor corrections to the area detection results. Even such minor corrections require time for learning.

[0009] Therefore, in order to perform efficient re-learning, it is desired to extract inspection images that the user was not satisfied with and exclude minor corrections from the target of re-learning.

[0010] The present invention has been made in view of such circumstances, and an object thereof is to provide a learning data collection device, a learning data collection method, and a program that can collect learning data for performing efficient re-learning.

Means for Solving the Problems

[0011] To achieve the above object, a learning data collection device according to one aspect of the present invention includes an inspection image acquisition unit that acquires an inspection image obtained by imaging an object to be inspected, a region detection result acquisition unit that acquires a region detection result detected by a learned region detector based on the inspection image, a correction history acquisition unit that acquires a correction history of the region detection result, a calculation unit that calculates correction quantification information obtained by quantifying the correction history, a database that stores the inspection image, the region detection result, and the correction history in association with each other, an image extraction condition setting unit that sets, as an extraction condition, a threshold value of the correction quantification information for extracting an inspection image to be used for re-learning from the database, and a first learning data extraction unit that extracts, from the database, an inspection image that satisfies the extraction condition and the region detection result and the correction history associated with the inspection image as learning data for re-learning the region detector.

[0012] According to this aspect, correction quantification information obtained by quantifying the correction history is calculated, and an inspection image having a certain correction quantification information is extracted as learning data. Therefore, it is possible to collect, as learning data, an inspection image in which the user corrects the region detection result and the correction is not a minor correction, and by using the collected learning data, efficient re-learning can be performed on the region detector.

[0013] Preferably, the learning data collection device includes an actual size information acquisition unit that acquires actual size information in the inspection image, and the calculation unit calculates the correction quantification information converted into an actual size based on the actual size information.

[0014] Preferably, the image extraction condition setting unit receives image information of the object, and the first learning data extraction unit extracts an inspection image from the database based on the image information and the threshold value of the correction quantification information.

[0015] Preferably, the image information of the object is information including at least one of information on the size of the region, information on the position of the region, information on the direction of the region, information on the type of the region, and meta information of the inspection image.

[0016] Preferably, the learning data collection device includes an image quality determination unit that determines the image quality of the inspection image. The image extraction condition setting unit receives information regarding the image quality of the inspection image, and the first learning data extraction unit extracts the inspection image from the database based on the information regarding the image quality and the threshold of the corrected quantification information.

[0017] Preferably, the learning data collection device includes an additional information acquisition unit that acquires additional information including at least one of identification information, member information, location information, environment information, material information, inspection information, and management information of the inspection image. The image extraction condition setting unit receives the additional information, and the first learning data extraction unit extracts the inspection image based on the additional information included in the inspection image and the additional information received by the image extraction condition setting unit.

[0018] Preferably, the learning data collection device includes a learning history acquisition unit that acquires history information regarding the history of use of the inspection image for learning, and a statistical information generation unit that generates statistical information of the images used for learning based on the acquired history information. The first learning data extraction unit extracts the inspection image from the database based on the generated statistical information.

[0019] Preferably, the inspection image acquisition unit includes a divided image obtained by dividing and photographing the object and a panoramic composite image obtained by synthesizing the divided images. The correction history acquisition unit acquires the correction history on the panoramic composite image, and the first learning data extraction unit extracts a region on the panoramic composite image based on the correction history on the panoramic composite image and extracts at least one of the divided images constituting the region.

[0020] Preferably, it includes a second learning data extraction unit that extracts, as learning data, the inspection images not extracted by the first learning data extraction unit to relearn the region detector.

[0021] Preferably, it includes an image confirmation display unit that displays the extracted inspection images and information related to the inspection images.

[0022] Another aspect of the present invention, a learning data collection method, includes steps of: obtaining an inspection image obtained by imaging an object to be inspected; obtaining a region detection result detected by a learned region detector based on the inspection image; obtaining a correction history of the region detection result; calculating correction quantification information obtained by quantifying the correction history; setting, as an extraction condition, a threshold value of the correction quantification information for extracting, from a database storing the inspection image, the region detection result, and the correction history in an associated manner, an inspection image to be used for re-learning; and extracting, from the database, an inspection image that satisfies the extraction condition, and the region detection result and the correction history associated with the inspection image as learning data for re-learning the region detector.

[0023] Another aspect of the present invention, a program, causes a computer to execute a learning data collection process including steps of: obtaining an inspection image obtained by imaging an object to be inspected; obtaining a region detection result detected by a learned region detector based on the inspection image; obtaining a correction history of the region detection result; calculating correction quantification information obtained by quantifying the correction history; setting, as an extraction condition, a threshold value of the correction quantification information for extracting, from a database storing the inspection image, the region detection result, and the correction history in an associated manner, an inspection image to be used for re-learning; and extracting, from the database, an inspection image that satisfies the extraction condition, and the region detection result and the correction history associated with the inspection image as learning data for re-learning the region detector.

Advantages of the Invention

[0024] According to the present invention, correction quantification information obtained by quantifying the correction history is calculated, and an inspection image having a certain amount of correction quantification information is extracted as learning data. Thus, it is possible to collect inspection images in which a user makes a correction to the region detection result and the correction is not a minor correction, and by using the collected learning data, it is possible to efficiently perform re-learning on the region detector.

Brief Description of the Drawings

[0025]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, preferred embodiments of the learning data collection device, learning data collection method, and program according to the present invention will be described with reference to the accompanying drawings. In the following description, an example of damage inspection of a structure will be described. That is, the object of inspection is a structure, and a case where an inspection for detecting damage (damage area) as an area from an inspection image of the structure is performed will be described.

[0027] FIG. 1 is a conceptual diagram showing an example of the learning data collection device of the present invention and a damage detector which is an area detector connected thereto. FIG. 1 shows a computer A equipped with the learning data collection device of the present invention, and computers B1 and B2 equipped with a learned damage detector. The computer A, the computers B1 and B2 are connected to a network and can communicate with each other.

[0028] The computer A functions as a server and receives, via the network, the inspection images, damage detection results (area detection results), and upload data C1 and C2 of the correction history uploaded from the computers B1 and B2, respectively. The computer A also functions as a learning computer capable of re-training the damage detectors installed in the computers B1 and B2. Then, the computer A distributes the re-trained damage detector or the detection parameter file (denoted by reference numeral D in the figure) obtained by re-training the damage detector to the computers B1 and B2.

[0029] The computers B1 and B2 upload, as upload data C1 and C2, the damage detection results output by the installed damage detectors for the input inspection images E1 and E2, and the correction history made by the user for the damage detection results to the computer A. The computers B1 and B2 perform the upload for the upload data C1 and C2 of all or some of the inspection images processed by the damage detector. The damage detectors installed in the computers B1 and B2 are damage detectors that have already been subjected to machine learning and are damage detectors that have been learned by a known technique.

[0030] Computer A stores the received upload data C1 and C2 in the database 23 (Figure 2) installed therein. Also, since Computer A extracts learning data by means of the learning data collection device 10 (Figure 2) installed therein and performs relearning using the extracted learning data, the detection performance of the damage detector can be improved efficiently. Here, relearning is a concept including the case where learning data is added and all the learning data that has already been learned is learned again, and the case where only the learning data is additionally learned. Note that the number of computers (Computers B1 and B2 in Figure 1) equipped with the damage detector connected to Computer A is not limited and may be single or plural.

[0031] Computer A, Computer B1, and Computer B2 are each connected to a monitor 9, and the user inputs commands via a keyboard 5 and a mouse 7. Note that the form of the computers shown is an example. For example, it is also possible to use a tablet terminal instead of the computers shown.

[0032] <First Embodiment> Next, the first embodiment of the present invention will be described.

[0033] Figure 2 is a block diagram showing a main functional configuration example of the learning data collection device 10 installed in the computer A of the present embodiment. The hardware structure for executing various controls of the learning data collection device 10 shown in Figure 2 is various processors as follows. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (program) and functions as various control units, a programmable logic device (PLD) such as an FPGA (Field Programmable Gate Array), which is a processor whose circuit configuration can be changed after manufacture, and a dedicated electric circuit, which is a processor having a circuit configuration specifically designed to execute specific processing such as an ASIC (Application Specific Integrated Circuit).

[0034] One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types (for example, a plurality of FPGAs, or a combination of a CPU and an FPGA). Also, a plurality of control units may be composed of one processor. As an example of configuring a plurality of control units with one processor, first, as represented by a computer such as a client or a server, one processor is configured by a combination of one or more CPUs and software, and this processor functions as a plurality of control units. Second, as represented by a system on chip (SoC), there is a form in which a processor that realizes the functions of the entire system including a plurality of control units with one IC (Integrated Circuit) chip is used. Thus, the various control units are configured using one or more of the above various processors as a hardware structure.

[0035] The learning data collection device 10 includes an inspection image acquisition unit 11, a damage detection result acquisition unit (area detection result acquisition unit) 13, a correction history acquisition unit 15, a calculation unit 17, an actual size information acquisition unit 18, an image extraction condition setting unit 19, a first learning data extraction unit 21, a database 23, a display control unit 25, and a storage unit 26. The storage unit 26 stores programs and information related to various controls of the learning data collection device 10. Further, the display control unit 25 controls the display on the monitor 9.

[0036] The inspection image acquisition unit 11 acquires an inspection image obtained by imaging a structure to be inspected. The inspection image acquired by the inspection image acquisition unit 11 is an image input to a damage detector (area detector) mounted on the computer B1 or B2, and the damage (area) in the inspection image is detected by the damage detector.

[0037] The damage detection result acquisition unit 13 acquires the damage detection result of the inspection image acquired by the inspection image acquisition unit 11. Here, the damage detection result is output by a damage detector mounted on the computer B1 or B2 and uploaded to the computer A.

[0038] The correction history acquisition unit 15 acquires the correction history of the damage detection result. The correction history acquired by the correction history acquisition unit 15 is the correction history for the damage detection result acquired by the damage detection result acquisition unit 13. Note that when no correction is made, the correction history acquisition unit 15 acquires information indicating that no correction has been made as the correction history.

[0039] The calculation unit 17 calculates correction quantification information obtained by quantifying the correction history. The calculation unit 17 may include an actual size information acquisition unit 18, and the actual size information acquisition unit 18 acquires the actual size information in the inspection image. In this case, the calculation unit 17 calculates the correction quantification information converted to the actual size based on the actual size information. Note that the calculation of the correction quantification information performed by the calculation unit 17 will be described later.

[0040] The database 23 stores by associating inspection images, damage detection results, and correction histories. The database 23 stores by associating the inspection images acquired by the inspection image acquisition unit 11, the damage detection results acquired by the damage detection result acquisition unit 13, and the correction histories acquired by the correction history acquisition unit 15.

[0041] The image extraction condition setting unit 19 sets, as extraction conditions, conditions for extracting inspection images to be used for relearning from the database 23, with the threshold value of the correction quantification information as the extraction condition. The threshold value of the correction quantification information is determined according to the correction quantification information stored in the database 23. Also, the user can extract, as learning data, inspection images in which the damage detection results have been corrected and which are not minor corrections, by changing this threshold value. For example, the user can set a predetermined threshold value and collect inspection images having correction quantification information equal to or greater than that threshold value.

[0042] The first learning data extraction unit 21 extracts, from the database 23, inspection images that satisfy the extraction conditions, and the damage detection results and correction histories associated with the inspection images, as learning data for relearning the damage detector.

[0043] <<Data Stored in the Database>> Next, a specific example of the data stored in the database 23 will be described. The database 23 stores at least inspection images, damage detection results, and correction histories uploaded from computers B1 and B2.

[0044] FIG. 3 is a diagram showing an example of an inspection image acquired by the inspection image acquisition unit 11. The inspection image 31 is an image of a part of the floor slab of a bridge, which is an example of a structure to be inspected. The inspection image 31 has cracks 33 and 35. The inspection image 31 is input to computer B1 or B2, and a damage detection result is output by the damage detector installed in computer B1. Note that the structure is not limited to a bridge, and may be other structures such as a tunnel, a box culvert, a dam, a seawall, a building (such as a wall surface or a floor).

[0045] Figure 4 is a diagram showing the inspection image shown in Figure 3 input into the damage detector and the resulting damage detection result. The damage detection result 41 has a damage detection result 37 for the crack 33 and a damage detection result 39 for the crack 35. The damage detection result 39 is good and sufficiently detects the crack 35. On the other hand, the damage detection result 37 does not sufficiently detect the crack 33 and is a poor damage detection result. Therefore, the user makes a correction regarding the damage detection result 37.

[0046] Figure 5 is a conceptual diagram showing the correction history made by the user for the damage detection result 41 shown in Figure 4. In the correction history 52, an additional vector 54 is added to the damage detection result 37 of the crack 33. That is, since the damage detection result 37 cannot sufficiently detect the crack 33, the user manually adds the additional vector 54.

[0047] The inspection image 31, the damage detection result 41, and the correction history 52 described in the above specific examples are uploaded from the computer B1 or B2 to the computer A.

[0048] <<Quantified correction information>> Next, the quantified correction information will be described. The quantified correction information is calculated by the calculation unit 17 quantifying the correction history. Here, the quantification (correction amount) of the correction history is, for example, in the case of a crack, information indicating the length of the vector added or deleted by the correction, or the amount of change in the coordinates of the vector moved by the correction. Also, for example, in the case of water leakage, free lime, peeling, exposed reinforcement, etc., it is the area of the region added or deleted by the correction, or the amount of change in the area of the region. Then, in the image extraction condition setting unit 19, a threshold value corresponding to the quantified correction information stored in the database 23 is set, and the first learning data extraction unit 21 extracts, for example, an inspection image having quantified correction information equal to or greater than the threshold value.

[0049] The calculation unit 17 may or may not include the actual size information acquisition unit 18. When the calculation unit 17 does not include the actual size information acquisition unit 18, the correction amount in terms of pixel size is used as the correction quantification information. Also, when the calculation unit 17 includes the actual size information acquisition unit 18, the calculation unit 17 calculates the correction amount converted into the actual size using the actual size information (the subject resolution (mm / pixel) or the size of members in the image, etc.). By including the actual size information acquisition unit 18, the calculation unit 17 can calculate the correction amount more accurately.

[0050] FIG. 6, FIG. 7, and FIG. 8 are diagrams showing specific examples of the correction history acquired by the correction history acquisition unit 15. Note that in FIGS. 6, 7, and 8, the vector (detection vector) detected and output by the damage detector for damage (crack) is shown by a dotted line, and the vector after the user makes a correction (corrected vector) is shown by a solid line. Also, the locations where the dotted line and the solid line are drawn almost parallel are the locations where the detection vector and the corrected vector actually overlap, indicating that the detection by the damage detector is good.

[0051] The correction history 56 shown in FIG. 6 shows the case where additional corrections are made by the user. Specifically, in the region 61 of the correction history 56, a corrected vector that was not output by the damage detector is added by the user, and damage with undetected leakage is added by the user's correction.

[0052] The correction history 58 shown in FIG. 7 shows the case where a correction for false detection is made by the user. Specifically, damage is detected in the region 63 of the correction history 58, but only the detection vector is shown in the region 63, and the corrected vector is not drawn. Therefore, the detection vector in the region 63 is due to false detection and is deleted and corrected.

[0053] The correction history 60 shown in Fig. 8, in (A) of the same figure, although in some areas the detection vector and part of the corrected vector overlap, in some areas corrections are made where vectors are deleted, and in some areas vectors are added. Specifically, in area 65, since only the detection vector exists, corrections for deleting false detections are made, and in area 67, since only the corrected vector exists, damage due to detection omission is added. Also, in (B) of the same figure, a correction history is described in which some coordinate values of the vector are changed and corrected. Specifically, the four coordinate values in the figure are different between the detection vector and the corrected vector, and the detection of damage is partially changed.

[0054] As described above, specific examples of the correction history have been shown, but the correction history is not limited to this, and various forms of correction history are adopted.

[0055] <<Learning data collection process>> Next, the learning data collection process (learning data collection method) using the learning data collection device 10 will be described. Fig. 9 is a flowchart showing the learning data collection process (learning data collection method) using the learning data collection device 10.

[0056] First, the inspection image acquisition unit 11 acquires inspection images uploaded from computers B1 and B2 (step S10). Also, the damage detection result acquisition unit 13 acquires damage detection results uploaded from computers B1 and B2 (step S11). Also, the correction history acquisition unit 15 acquires correction histories uploaded from computers B1 and B2 (step S12). Then, the calculation unit 17 calculates correction quantification information based on the acquired correction history (step S13). Then, the image extraction condition setting unit 19 sets the threshold value of the correction quantification information as the extraction condition (step S14). Then, the first learning data extraction unit 21 extracts learning data for re-learning (step S15).

[0057] Each of the above configurations and functions can be appropriately implemented by any hardware, software, or a combination of both. For example, the present invention can also be applied to a program that causes a computer to execute the above-described processing steps (processing procedures), a computer-readable recording medium (non-transitory recording medium) that records such a program, or a computer to which such a program can be installed.

[0058] As described above, in this embodiment, the modification history is quantified, and learning data is extracted according to the quantified extraction conditions. Therefore, inspected images that have been modified and whose modifications are not minor are extracted as learning data, enabling efficient learning of the damage detector.

[0059] <Second Embodiment> Next, the second embodiment will be described. In the second embodiment, inspected images are extracted based on, in addition to the modification quantification information, information obtained from the inspected images (image information of the structure and information regarding the image quality of the inspected images). As a result, it is possible to collect learning data that enables more efficient relearning.

[0060] FIG. 10 is a block diagram showing a functional configuration example of the learning data collection device 10 according to this embodiment. Parts that have already been described in FIG. 2 are denoted by the same reference numerals, and the description thereof is omitted.

[0061] The learning data collection device 10 includes an inspected image acquisition unit 11, a damage detection result acquisition unit 13, a modification history acquisition unit 15, a calculation unit 17, an actual size information acquisition unit 18, an image extraction condition setting unit 19, a first learning data extraction unit 21, an image quality determination unit 51, a database 23, a display control unit 25, and a storage unit 26.

[0062] The image extraction condition setting unit 19 of the present embodiment receives the image information of the structure, which is one of the information obtained from the inspection image. Here, the image information of the structure is information including at least one of the damage size information (area size information), damage position information (area position information), damage direction information (area direction information), damage type information (area type information), and meta information of the inspection image. In the database 23, this image information is stored in relation to the inspection image. Note that the image information is acquired, for example, when damage is detected by a damage detector. Specifically, the damage size information, damage position information, damage direction information, damage type information, and meta information of the inspection image are acquired when damage is detected by the damage detector and uploaded in association with the inspection image.

[0063] The first learning data extraction unit 21 extracts the inspection image from the database 23 based on the image information and the corrected quantification information. That is, the extraction condition of the image information to be extracted (to be additionally learned) in the image extraction condition setting unit 19 and the threshold value of the corrected quantification information are set. Specific examples of the extraction condition of the image information to be set include the condition of extracting an image with a minimum crack width of 0.2 mm or more, or the condition of extracting an image with free lime in the central part. Note that only the extraction condition of the image information may be set in the image extraction condition setting unit 19, and in that case, the inspection image is extracted according to the extraction condition of the image information.

[0064] <<Image information of the structure>> Next, the image information of the structure will be described.

[0065] One of the image information of the structure is the damage size. The damage size is quantified in pixel size, or in actual size when the actual size information acquisition unit 18 is provided, and is used as the image information of the structure. Specific examples of the damage size include the total length of the damage in the inspection image (the length of the crack when the inspection object is a concrete structure, the length of the crack when the inspection object is a steel member). Also, the total length for each crack width (less than 0.1 mm, 0.1 mm or more and less than 0.2 mm, 0.2 mm or more, etc.) may be used as the image information of the structure.

[0066] In addition, as specific examples of the size of the damage, the total area of the damage in the inspection image (such as water leakage, free lime, peeling, steel bar exposure, etc. when the inspection object is a concrete structure, corrosion, deterioration of the anticorrosion function, etc. when the inspection object is a steel member) can be cited. In addition, the maximum width of the damage, the maximum area (such as in the case of water leakage, free lime, peeling, steel bar exposure), the minimum interval (the minimum distance from an adjacent damage of the same type), the density (the number of cracks per square meter, the length [cracks, fissures], the area [water leakage, free lime, peeling, steel bar exposure], etc.) can be cited.

[0067] As one of the image information of the structure, there is the position of the damage. The position of the damage is grasped from the inspection image in which the entire member is shown. Also, after panoramically synthesizing the images taken by dividing the member, the position of the damage may be grasped from the panoramically synthesized image. Specific examples of the position of the damage include the position (central part / end part, etc.), the directionality (the cracks in the slab are in the bridge axis direction / right angle direction of the bridge axis, the cracks in the pier are in the vertical direction / horizontal direction, etc.).

[0068] As one of the image information of the structure, there is the type of the damage. The type of the damage utilizes the detection results of the damage detector (for example, crack detection, water leakage detection, free lime detection, peeling detection, steel bar exposure detection, crack detection, corrosion detection, deterioration detection of the anticorrosion function, etc.). Also, the user may specify the type of the damage.

[0069] As one of the image information of the structure, there is meta information (Exif information). The meta information is, for example, the camera model, the lens type, the F value, the shutter speed, the focal length, the flash ON / OFF, the number of pixels, the ISO sensitivity, etc.

[0070] FIG. 11 is a diagram showing an example of a storage configuration of information (image information of a structure) obtained from an inspection image, which is stored in association with the inspection image in the database 23. Note that the information indicated by reference numeral 69 is information obtained from the learned inspection image, and the information indicated by reference numeral 71 is additional information to be described later. Examples of the information obtained from the inspection image include, for example, the maximum crack width (mm), the minimum crack interval (m), the presence or absence of damage other than cracks, and the name of the member imaged. For example, such information is obtained when the inspection image is input to the damage detector. In this way, the database 23 stores the image information of the structure in association with the inspection image.

[0071] <<Example of Image Quality Judgment>> As one of the information obtained from the inspection image, there is information regarding the image quality of the inspection image.

[0072] The image quality determination unit 51 determines the image quality of the inspection image acquired by the inspection image acquisition unit 11. In this case, the image extraction condition setting unit 19 receives information regarding the image quality of the inspection image. The first learning data extraction unit 21 extracts the inspection image from the database 23 based on the information regarding the image quality and the threshold of the correction quantification information.

[0073] Here, various methods can be adopted for the determination performed by the image quality determination unit 51. Specific examples of the determination method of the image quality determination unit 51 will be described below.

[0074] <<Determination of Image Quality by Machine Learning>> As an image quality determination method performed by the image quality determination unit 51, there is a determination method using an image quality determination device subjected to machine learning. That is, the image quality determination unit 51 is constituted by an image quality determination device (image quality determination AI) by machine learning, and the image quality of the inspection image is determined by the image quality determination device.

[0075] <<Judgment by Spatial Frequency Spectrum>> The image quality determination unit 51 may quantify and determine the image quality based on, for example, the maximum value, average value, or sum of the spectrum in the high-frequency region of the spatial frequency spectrum of the region in the inspection image. Specifically, the larger the maximum value, average value, or sum of the components within a specific pixel radius (r pixel radius) from the four corners of the spatial frequency spectrum image (obtained by performing a fast Fourier transform (FFT) on the captured image), the stronger (more) the high-frequency components, indicating less blur and better image quality.

[0076] <<Judgment by Histogram>> In the judgment by the histogram (an example of an index indicating image quality) performed by the image quality determination unit 51, the image quality determination unit 51 converts an individual image (a color image composed of R, G, and B components) into a grayscale image. For example, the grayscale (density) = R × 0.30 + G × 0.59 + B × 0.11 (where R, G, and B are the values of the red signal, green signal, and blue signal, respectively). The image quality determination unit 51 calculates the histogram (density histogram; refer to the example in Fig. 12) of the converted grayscale image. The calculation of the histogram and the following judgment may be performed on a part of the region rather than the entire individual image. The image quality determination unit 51 determines whether the individual image is too bright or too dark according to the following formulas (1) and (2), with G(i) {i = 0, 1, …, 255} being the histogram of each density value (the closer to 0, the darker; the closer to 255, the brighter). The judgment thresholds (kb, hb, kd, hd) may be default values (for example, kb = 205, hb = 0.5, kd = 50, hd = 0.5), or may be set by the image quality determination unit 51 according to the user input via the operation unit (keyboard 5 and mouse 7).

[0077]

Number

[0078]

Number

[0079] If the ratio of density values greater than or equal to kb in the above-described formula (1) is greater than or equal to hb of the whole, the image quality determination unit 51 determines that it is "too bright". In this case, the image quality determination unit 51 determines that "the image quality is low (because it is too bright)" and sets the individual image as the image to be confirmed. Similarly, if the ratio of density values less than or equal to kd in the formula (2) is greater than or equal to hd of the whole, the image quality determination unit 51 determines that "the image quality is low (because it is too dark)" and sets the individual image as the image to be confirmed.

[0080] Based on the histogram, it is also possible to determine whether the gradation is crushed. For example, the image quality determination unit 51 uses G(i) {i = 0, 1,..., 255} as the histogram of each density value. When G(0) > Td, it determines that "the gradation on the shadow side is crushed", and when G(255) > Tb, it determines that "the gradation on the highlight side is crushed". In these cases, the image quality determination unit 51 determines that "the image quality is low" and sets the individual image as the image to be confirmed. The determination thresholds (Td, Tb) may be default values (for example, Td = 0, Tb = 0), or may be set by the image quality determination unit 51 according to user input via the operation unit (keyboard 5 and mouse 7).

[0081] As described above, the image quality determination unit 51 can determine the image quality of the inspection image by various methods.

[0082] In this embodiment, in addition to the corrected quantification information, the inspection image is extracted based on the information obtained from the inspection image (image information of the structure and information regarding the image quality of the inspection image), so that learning data for efficient relearning can be collected.

[0083] <Third Embodiment> Next, the third embodiment will be described. In the third embodiment, in addition to the corrected quantification information, the inspection image is extracted based on the additional information of the inspection image. Thereby, learning data for more efficient relearning can be collected.

[0084] FIG. 13 is a block diagram showing a functional configuration example of the learning data collection device 10 of the present embodiment. Parts already described in FIG. 2 are denoted by the same reference numerals, and the description thereof is omitted.

[0085] The learning data collection device 10 includes an inspection image acquisition unit 11, a damage detection result acquisition unit 13, a correction history acquisition unit 15, a calculation unit 17, an actual size information acquisition unit 18, an image extraction condition setting unit 19, a first learning data extraction unit 21, an attached information acquisition unit 53, a database 23, a display control unit 25, and a storage unit 26.

[0086] The attached information acquisition unit 53 acquires attached information including at least one of identification information of the inspection image, member information, location information, environmental information, material information, inspection information, management information, and structure type information. And in this case, the image extraction condition setting unit 19 receives the attached information, and the first learning data extraction unit 21 extracts the inspection image based on the attached information included in the inspection image and the attached information received by the image extraction condition setting unit 19. That is, the extraction condition of the attached information to be extracted (to be additionally learned) is also set together with the threshold value of the correction quantification information in the image extraction condition setting unit 19. For example, as the extraction condition of the attached information, an image of a pier within 100 m from the coast is extracted, an image of a bridge with a traffic volume of 1000 vehicles / day or more and an age of 30 years or more is extracted, and the like. Note that only the extraction condition of the attached information may be set in the image extraction condition setting unit 19, and in that case, the inspection image is extracted according to the extraction condition of the attached information.

[0087] Specific examples of each piece of additional information are described below. The identification information is, for example, the name of a bridge or an ID (Identification) number. The member information is, for example, the member type (floor slab / bridge pier / girder, etc.) or the direction (bridge axis direction, vertical direction). The location information is, for example, the prefecture, city, town, region, latitude / longitude, and distance from the sea. The environmental information is, for example, the climate (temperature [average, maximum, minimum, etc.], humidity [average, maximum, minimum, etc.], rainfall, snowfall) and traffic volume. The material information is, for example, the concrete aggregate size and material strength (compressive strength, tensile strength). The inspection information (such as the results of core extraction tests on concrete) is the chloride ion concentration, the progress of carbonation, and the presence or absence of alkali-aggregate reaction. The management information is the number of years, construction conditions (such as the temperature or humidity during construction), and repair history. The structure type information is bridges, tunnels, box culverts, buildings, etc.

[0088] In this embodiment, in addition to the corrected quantification information, additional information is also used to extract learning data, so that learning data that can perform more efficient relearning can be extracted.

[0089] <Fourth Embodiment> Next, the fourth embodiment will be described.

[0090] FIG. 14 is a block diagram showing a functional configuration example of the learning data collection device 10 of this embodiment. Note that the parts already described in FIG. 2 are denoted by the same reference numerals and the description thereof is omitted.

[0091] The learning data collection device 10 includes an inspection image acquisition unit 11, a damage detection result acquisition unit 13, a correction history acquisition unit 15, a calculation unit 17, an actual size information acquisition unit 18, an image extraction condition setting unit 19, a first learning data extraction unit 21, a learning history acquisition unit 55, a statistical information generation unit 57, a second learning data extraction unit 59, a database 23, a display control unit 25, and a storage unit 26.

[0092] The learning history acquisition unit 55 acquires history information regarding the history of use of inspection images in learning. Specifically, the learning history acquisition unit 55 acquires information as to whether or not the inspection images acquired by the inspection image acquisition unit 11 have already been used in the machine learning of the damage detectors of computers B1 or B2.

[0093] The statistical information generation unit 57 generates statistical information on the images used in learning based on the acquired history information. The first learning data extraction unit 21 may extract inspection images based on the generated statistical information. That is, the extraction conditions for the statistical information to be extracted (to be additionally learned) are also set together with the threshold value of the corrected quantification information in the image extraction condition setting unit 19. Here, the extraction conditions for the statistical information are, for example, conditions with low frequencies in the image information or the attached information among the images already used in learning. Note that only the extraction conditions for the statistical information may be set in the image extraction condition setting unit 19, and in that case, the inspection images are extracted according to the extraction conditions for the statistical information.

[0094] Specific examples of the statistical information are shown below. Note that "○%" described below indicates the ratio of the number of images satisfying a predetermined condition to the total number of learned images.

[0095] For example, the statistical information on the maximum crack width can be described as ○% less than 0.05 mm, ○% 0.05 mm or more and less than 0.1 mm, ○% 0.1 mm or more and less than 0.2 mm, ○% 0.2 mm or more and less than 0.5 mm, ○% 0.5 mm or more and less than 1.0 mm, ○% 1.0 mm or more. Also, for example, the statistical information on the minimum crack interval can be described as ○% less than 0.2 m, ○% 0.2 m or more and less than 0.5 m, ○% 0.5 m or more and less than 1 m, ○% 1 m or more. Also, the statistical information on damage other than cracks can be described as ○% none, ○% water leakage, ○% free lime, ○% peeling, ○% reinforcement exposure, ○% water leakage and free lime. In addition to this, statistical information can be generated based on members, distance from the sea, traffic volume, and years of service. Furthermore, conditions for combinations of multiple statistical information, such as the maximum crack width × damage other than cracks, can be generated.

[0096] The second learning data extraction unit 59 extracts, as learning data for re-training the damage detector, from the inspection images not extracted by the first learning data extraction unit 21. For example, the second learning data extraction unit 59 extracts randomly or regularly (e.g., extracts at regular intervals after arranging in the order of file names) from the inspection images not extracted by the first learning data extraction unit 21. Also, the second learning data extraction unit 59 extracts randomly, excluding low-quality inspection images. The second learning data extraction unit 59 extracts the same number of inspection images as the number of inspection images extracted by the first learning data extraction unit 21. By training the learning data extracted by the second learning data extraction unit 59, it is possible to prevent being overly influenced by inspection images under biased learning conditions.

[0097] <Other examples> <<Panorama composite image>> Next, the case where there is a correction history in the panorama composite image will be described.

[0098] The inspection images acquired by the inspection image acquisition unit 11 include panorama composite images. That is, the inspection image acquisition unit 11 acquires the divided images obtained by dividing and photographing the structure and the panorama composite images obtained by synthesizing the divided images. And in this case, the correction history acquisition unit 15 acquires the correction history performed on the panorama composite image.

[0099] The first learning data extraction unit 21 extracts a region on the panorama composite image based on the correction history on the panorama composite image, and extracts at least one of the divided images constituting the region.

[0100] FIG. 15 is a diagram conceptually showing the correction history on the panorama composite image acquired by the inspection image acquisition unit 11.

[0101] On the panoramic composite image 71 shown in FIG. 15, the user is making corrections (addition or deletion). The corrections made by the user are indicated by vectors (1) to (5). The corrections made by the user exist across the divided image 73 and the divided image 75 that form part of the panoramic composite image 71. In such a case, the first learning data extraction unit 21 can automatically select inspection images in, for example, the following patterns. In this case, the corrections on the panoramic composite image 71 are corrections that satisfy the extraction conditions set by the image extraction condition setting unit 19.

[0102] The first learning data extraction unit 21 selects all the images including vectors (1) to (5) as pattern 1. That is, the first learning data extraction unit 21 selects the divided image 73 and the divided image 75 as learning data. Also, when the lengths of vectors (1) to (5) are equal to or greater than a predetermined threshold as pattern 2, the first learning data extraction unit 21 selects all the divided images including vectors (1) to (5). Further, the first learning data extraction unit 21 selects the divided image with the longest length including vectors (1) to (5) as pattern 3. In this case, since the divided image 75 has vectors (1) to (5) that are longer than those of the divided image 73, the divided image 75 is selected. Also, the first learning data extraction unit 21 compares the amount of correction included in the divided image 73 and the amount of correction included in the divided image 75 with predetermined thresholds respectively as pattern 4, and determines whether to extract each image or not. Further, the first learning data extraction unit 21 determines the image quality of the images including vectors (1) to (5) as pattern 5, and selects the image with the highest image quality.

[0103] <<Image confirmation display unit>> Next, the image confirmation display unit that displays the inspection images extracted by the first learning data extraction unit 21 or the second learning data extraction unit 59 will be described.

[0104] The learning data collection device 10 can include an image confirmation display unit configured by the monitor 9.

[0105] FIG. 16 is a diagram showing an example of a confirmation screen of an extracted inspection image displayed on the monitor 9. In the example shown in FIG. 16, the extracted inspection image 77 is shown. Further, on the confirmation screen, a selection button 81 for selecting whether to adopt or not adopt as learning data is provided for the extracted inspection image. Further, a button 79 for receiving a command to clarify detection omission locations, false detection locations, etc. based on the correction history is provided. Further, the confirmation screen has a display 83 of information related to the inspection image (information obtained from the image and / or attached information).

[0106] As described above, since the learning data collection device 10 has the image confirmation display unit, the user can confirm the extracted inspection image, and can collect learning data with a higher learning effect.

[0107] <<Other inspections>> In the above description, an example in which the present invention is applied when performing a damage inspection of a structure has been described. The example of the inspection to which the present invention is applied is not limited to this. The present invention is applied when detecting some region (including the region of an object) by the region detector. For example, when the object of inspection is a human body, the present invention is also applied when detecting blood vessels from a CT (Computed Tomography) image which is an inspection image of the human body. Further, the present invention is also applied when performing an inspection (such as appearance inspection) of surface scratches and defects using an image with products such as articles and drugs as the object. Further, the present invention is also applied to an inspection by an image of damage / defects inside a structure using X-rays.

[0108] Although the example of the present invention has been described above, it goes without saying that the present invention is not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present invention.

Explanation of reference numerals

[0109] 10: Learning data collection device 11: Inspection image acquisition unit 13: Damage detection result acquisition unit 15: Correction history acquisition unit 17: Calculation unit 18: Actual Size Information Acquisition Unit 19: Image Extraction Condition Setting Unit 21: First Learning Data Extraction Unit 23: Database 25: Display Control Unit 26: Memory Unit A: Computer B1: Computer B2: Computer C1, C2: Uploaded Data D: Retrained Damage Detector or Detection Parameter File E1, E2: Inspection Images

Claims

1. an acquisition unit that acquires an area detection result indicating an area detected by the area detector based on an inspection image obtained by capturing an image of an object to be inspected; a calculation unit that calculates modified quantification information by quantifying a modification history of the area detection result; an extractor for extracting training data from the inspection image for training the area detector based on at least a threshold value of the modified quantification information; A learning data collection device comprising:

2. the area detection result is a detection result of a damaged area detected by the area detector, The correction history is a correction of the detection result of the damaged area. The learning data collection device according to claim 1 .

3. The learning data collection device according to claim 1 , wherein the calculation unit calculates the correction quantification information by quantifying based on dimensions of the correction history.

4. The learning data collection device according to claim 1 , wherein the threshold value is a dimension corresponding to the corrected quantification information.

5. an image quality determination unit for determining an image quality of the inspection image; The extraction unit receives information regarding image quality of the inspection image, 5. The learning data collection device according to claim 1, wherein the inspection image is extracted based on the information regarding image quality and the threshold value.

6. obtaining an area detection result indicating an area detected by the area detector based on an inspection image obtained by capturing an image of an object to be inspected; calculating modified quantification information by quantifying a modification history of the area detection result; extracting training data from the inspection image for training the area detector based at least on a threshold value of the modified quantification information; A method for collecting data for learning, including:

7. obtaining an area detection result indicating an area detected by the area detector based on an inspection image obtained by capturing an image of an object to be inspected; calculating modified quantification information by quantifying a modification history of the area detection result; extracting training data from the inspection image for training the area detector based at least on a threshold value of the modified quantification information; A program for causing a computer to execute a learning data collection step including the steps of:

Images