LEARNING DATA COLLECTION DEVICE, LEARNING DATA COLLECTION METHOD, AND PROGRAM

Abstract

To provide a learning data collecting apparatus, a learning data collecting method, and a program which collects learning data which allows efficient re-learning.SOLUTION: A computer A mounted with a learning data collecting apparatus 10 has an inspection image acquiring unit 11 for acquiring an inspection image, a region detection result acquiring unit (damage detection result acquiring unit 13) for acquiring a region detection result detected by a learned region detector, a modification history acquiring unit 15 for acquiring a modification history of the region detection result, a calculation unit 17 for calculating a modification quantization information obtained by quantizing modification history, a data base for storing the inspection image, the region detection result, and the modification history in association with one another, an image extraction condition setting unit 19 for setting a threshold value of the modification quantization information as an extraction condition for use in extracting the inspection image for use in re-learning from the data base, and a first learning data extracting unit 21 for extracting the inspection image satisfying the extraction conditions as a learning data for use in making the region detector re-learning.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a learning data collection device, a learning data collection method, and a program, and more particularly to a learning data collection device, a learning data collection method, and a program for extracting learning data for re-learning an area detector. [Background technology]

[0002] In recent years, photographed images of an object to be inspected are sometimes used to inspect the object. The inspection of the object is performed by detecting and identifying a predetermined area from the photographed image of the object. For example, when inspecting a structure for damage, photographed images of the structure to be inspected, such as a bridge, road, or building, are obtained. Then, damage to the structure is detected and identified from the photographed images by image processing.

[0003] Here, as one method of detecting an area from a captured image by image processing, a method of detecting an area using an area detector (e.g., a damage detector) that has been subjected to machine learning is known. It is also known that the accuracy of area detection can be improved by re-learning (or additional learning) a region detector that has been subjected to machine learning once. However, if the area detector is made to learn all of the huge number of images stored in a database or the like as learning data, the learning time will be enormous. Furthermore, unless appropriate data is learned as learning data, a high learning effect cannot be expected. In other words, the area detector cannot be made to learn efficiently just by randomly selecting learning data.

[0004] Therefore, in the past, when machine learning is performed, methods for extracting learning data have been proposed with the aim of performing efficient learning.

[0005] For example, Patent Document 1 describes a technology that aims to reliably and quickly improve the classification accuracy of images. Specifically, the technology describes a technology that uses, for machine learning, images other than images used in past machine learning, which have low similarity to the images used in past machine learning. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2017-224184 A Summary of the Invention [Problem to be solved by the invention]

[0007] Here, when detecting the area of ​​an object using an area detector trained through machine learning, in order to perform efficient machine learning, it is necessary to have the area detector re-learn using an inspection image in which the user was not satisfied with the area detection results performed by the area detector.

[0008] On the other hand, users may make minor corrections to the region detection results, but even such minor corrections require time for learning.

[0009] Therefore, in order to perform efficient re-learning, it is desirable to extract inspection images that the user is not satisfied with and exclude minor corrections from the re-learning process.

[0010] The present invention has been made in consideration of the above circumstances, and aims to provide a learning data collection device, a learning data collection method, and a program for collecting learning data that enables efficient re-learning. [Means for solving the problem]

[0011] In order to achieve the above-mentioned object, a learning data collection device which is one embodiment of the present invention comprises an inspection image acquisition unit which acquires an inspection image of an object to be inspected, an area detection result acquisition unit which acquires an area detection result detected by a trained area detector based on the inspection image, a correction history acquisition unit which acquires a correction history of the area detection result, a calculation unit which calculates correction quantification information which quantifies the correction history, a database which stores the inspection image, the area detection result, and the correction history in association with each other, an image extraction condition setting unit which sets a threshold value of the correction quantification information as an extraction condition for extracting inspection images to be used for re-learning from the database, and a first learning data extraction unit which extracts inspection images which satisfy the extraction conditions and the area detection results and correction history associated with the inspection image from the database as learning data for re-learning the area detector.

[0012] According to the present embodiment, correction quantification information that quantifies the correction history is calculated, and inspection images having a certain amount of correction quantification information are extracted as learning data. Therefore, inspection images in which a user makes corrections to the area detection results and the corrections are not minor can be collected as learning data, and the collected learning data can be used to efficiently re-learn the area detector.

[0013] Preferably, the learning data collection device includes an actual size information acquisition unit that acquires actual size information in the inspection image, and the calculation unit calculates modified quantification information converted into actual size using the actual size information.

[0014] Preferably, the image extraction condition setting unit receives image information of the object, and the first learning data extraction unit extracts the inspection image from the database based on the image information and the threshold value of the modified quantification information.

[0015] Preferably, the image information of the object includes at least one of area size information, area position information, area direction information, area type information, and inspection image meta information.

[0016] Preferably, the learning data collection device includes an image quality judgment unit that judges the image quality of the inspection image, the image extraction condition setting unit accepts information regarding the image quality of the inspection image, and the first learning data extraction unit extracts the inspection image from the database based on the information regarding the image quality and a threshold value of the modified quantification information.

[0017] Preferably, the learning data collection device includes an additional information acquisition unit that acquires additional information including at least one of identification information, component information, location information, environmental information, material information, inspection information, and management information of the inspection image, the image extraction condition setting unit accepts the additional information, and the first learning data extraction unit extracts the inspection image based on the additional information possessed by the inspection image and the additional information accepted by the image extraction condition setting unit.

[0018] Preferably, the learning data collection device includes a learning history acquisition unit that acquires historical information regarding the history of the test image being used for learning, and a statistical information generation unit that generates statistical information of the images used for learning based on the acquired historical information, and the first learning data extraction unit extracts the test image from the database based on the generated statistical information.

[0019] Preferably, the inspection image acquisition unit includes split images obtained by photographing the object in sections and a panoramic composite image obtained by combining the split images, the correction history acquisition unit acquires the correction history on the panoramic composite image, and the first learning data extraction unit extracts an area on the panoramic composite image based on the correction history on the panoramic composite image and extracts at least one of the split images that constitute the area.

[0020] Preferably, the apparatus further comprises a second learning data extraction unit which extracts, from the inspection image not extracted by the first learning data extraction unit, learning data for re-training the area detector.

[0021] Preferably, the apparatus further comprises an image confirmation display section for displaying the extracted examination image and information related to the examination image.

[0022] Another aspect of the present invention, a method of collecting learning data, includes the steps of acquiring an inspection image of an object to be inspected, acquiring area detection results detected by a trained area detector based on the inspection image, acquiring a correction history of the area detection results, calculating correction quantification information that quantifies the correction history, setting extraction conditions for extracting inspection images to be used for re-learning from a database that stores the inspection images, area detection results, and correction history in association with each other, the extraction conditions being a threshold value of the correction quantification information, and extracting from the database inspection images that satisfy the extraction conditions and the area detection results and correction history associated with the inspection images, as learning data for re-learning the area detector.

[0023] Another aspect of the present invention is a program that causes a computer to execute a learning data collection process, including the steps of acquiring an inspection image of an object to be inspected, acquiring area detection results detected by a trained area detector based on the inspection image, acquiring a correction history of the area detection results, calculating correction quantification information that quantifies the correction history, setting a threshold value of the correction quantification information as an extraction condition for extracting inspection images to be used for re-learning from a database that stores the inspection images, area detection results, and correction history in association with each other, and extracting from the database inspection images that satisfy the extraction conditions and the area detection results and correction history associated with the inspection images as learning data for re-learning the area detector. Effect of the Invention

[0024] According to the present invention, correction quantification information that quantifies the correction history is calculated, and inspection images having a certain amount of correction quantification information are extracted as learning data. This makes it possible to collect inspection images in which the user has made corrections to the area detection results and the corrections are not minor, and by using the collected learning data, efficient re-learning can be performed on the area detector. [Brief description of the drawings]

[0025] [Figure 1] FIG. 1 is a conceptual diagram showing a learning data collection device and a damage detector connected thereto. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a main functional configuration of the learning data collection device. [Diagram 3] FIG. 3 is a diagram showing an example of an inspection image acquired by the inspection image acquisition unit. [Figure 4] FIG. 4 is a diagram showing the damage detection results. [Diagram 5] FIG. 5 is a conceptual diagram showing a revision history. [Figure 6] FIG. 6 is a diagram showing a specific example of a revision history. [Figure 7] FIG. 7 is a diagram showing a specific example of a revision history. [Figure 8] FIG. 8 is a diagram showing a specific example of a revision history. [Figure 9] FIG. 9 is a flowchart showing the learning data collection process. [Figure 10] FIG. 10 is a block diagram illustrating an example of a functional configuration of the learning data collection device. [Figure 11] FIG. 11 is a diagram showing an example of a storage configuration of information obtained from an inspection image, which is stored in a database in association with the inspection image. [Figure 12] FIG. 12 is a diagram showing an example of a density histogram. [Figure 13] FIG. 13 is a block diagram illustrating an example of a functional configuration of the learning data collection device. [Figure 14] FIG. 14 is a block diagram illustrating an example of a functional configuration of the learning data collection device. [Figure 15] FIG. 15 is a diagram conceptually showing a correction history on a panoramic composite image. [Figure 16] FIG. 16 is a diagram showing an example of a confirmation screen for an inspection image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Hereinafter, preferred embodiments of the learning data collection device, learning data collection method, and program according to the present invention will be described with reference to the attached drawings. In the following description, an example of damage inspection of a structure will be described. That is, the subject of inspection is a structure, and the inspection is performed to detect damage (damaged area) as an area from an inspection image of the structure.

[0027] Fig. 1 is a conceptual diagram showing a learning data collection device of the present invention and a damage detector, which is an example of an area detector, connected thereto. Fig. 1 shows a computer A equipped with the learning data collection device of the present invention, and computers B1 and B2 equipped with trained damage detectors. Computer A, computer B1, and computer B2 are connected to a network and can communicate information with each other.

[0028] Computer A functions as a server and receives upload data C1 and C2 of inspection images, damage detection results (area detection results), and correction history uploaded from computers B1 and B2, respectively, via the network. Computer A also functions as a learning computer that can retrain the damage detectors installed in computers B1 and B2. Computer A then distributes the retrained damage detectors or detection parameter files (indicated by symbol D in the figure) obtained by retraining the damage detectors to computers B1 and B2.

[0029] Computers B1 and B2 upload the damage detection results output by the damage detectors installed in them for the input inspection images E1 and E2, and the history of corrections made by the user to the damage detection results, as upload data C1 and C2, to computer A. Computers B1 and B2 upload the upload data C1 and C2 for all inspection images or some of the inspection images processed by the damage detectors. The damage detectors installed in computers B1 and B2 are damage detectors that have already been subjected to machine learning, and are damage detectors that have been trained using known technology.

[0030] Computer A stores the received upload data C1 and C2 in a database 23 (Fig. 2) mounted on it. Computer A also extracts learning data using a learning data collection device 10 (Fig. 2) mounted on it and performs re-learning using the extracted learning data, thereby efficiently improving the detection performance of the damage detector. Here, re-learning is a concept that includes a case where learning data is added and all learning data that has already been learned is re-learned, and a case where only learning data is additionally learned. Note that the number of computers mounted with damage detectors (computers B1 and B2 in Fig. 1) connected to computer A is not limited, and may be either one or multiple.

[0031] Computer A, computer B1, and computer B2 are each connected to a monitor 9, and a user inputs commands via a keyboard 5 and a mouse 7. Note that the illustrated form of the computer is an example. For example, a tablet terminal can be used instead of the illustrated computer.

[0032] <First embodiment> Next, a first embodiment of the present invention will be described.

[0033] Fig. 2 is a block diagram showing an example of a main functional configuration of the learning data collection device 10 mounted on the computer A of this embodiment. The hardware structure for executing various controls of the learning data collection device 10 shown in Fig. 2 is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various control units, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing specific processing such as an ASIC (Application Specific Integrated Circuit).

[0034] One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple control units may be composed of one processor. As an example of configuring multiple control units with one processor, first, as represented by a computer such as a client or server, one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple control units. Second, as represented by a system on chip (SoC), there is a form using a processor that realizes the functions of the entire system including multiple control units with one IC (Integrated Circuit) chip. In this way, various control units are configured using one or more of the above various processors as a hardware structure.

[0035] The learning data collection device 10 includes an inspection image acquisition unit 11, a damage detection result acquisition unit (area detection result acquisition unit) 13, a correction history acquisition unit 15, a calculation unit 17, an actual size information acquisition unit 18, an image extraction condition setting unit 19, a first learning data extraction unit 21, a database 23, a display control unit 25, and a storage unit 26. The storage unit 26 stores programs and information related to various controls of the learning data collection device 10. The display control unit 25 controls the display on the monitor 9.

[0036] The inspection image acquisition unit 11 acquires an inspection image of a structure to be inspected. The inspection image acquired by the inspection image acquisition unit 11 is an image input to a damage detector (area detector) mounted on the computer B1 or B2, and damage (area) in the inspection image is detected by the damage detector.

[0037] The damage detection result acquisition unit 13 acquires the damage detection results of the inspection images acquired by the inspection image acquisition unit 11. Here, the damage detection results are output by a damage detector mounted on the computer B1 or B2, and uploaded to the computer A.

[0038] Correction history acquisition unit 15 acquires the correction history of the damage detection result. The correction history acquired by correction history acquisition unit 15 is a correction history for the damage detection result acquired by damage detection result acquisition unit 13. When no correction has been made, correction history acquisition unit 15 acquires information indicating that no correction has been made as the correction history.

[0039] The calculation unit 17 calculates modified quantification information that quantifies the modification history. The calculation unit 17 may also include an actual size information acquisition unit 18, which acquires actual size information in the inspection image. In this case, the calculation unit 17 calculates modified quantification information converted into actual size using the actual size information. The calculation of the modified quantification information performed by the calculation unit 17 will be described later.

[0040] Database 23 stores the inspection image, the damage detection result, and the correction history in an associated manner. Database 23 stores the inspection image acquired by inspection image acquisition unit 11, the damage detection result acquired by damage detection result acquisition unit 13, and the correction history acquired by correction history acquisition unit 15 in an associated manner.

[0041] The image extraction condition setting unit 19 sets a threshold value of the modified quantification information as an extraction condition for extracting inspection images to be used for re-learning from the database 23. The threshold value of the modified quantification information is determined according to the modified quantification information stored in the database 23. Furthermore, the user can change this threshold value to extract, as learning data, inspection images whose damage detection results have been modified and which are not minor modifications. For example, the user can set a predetermined threshold value and collect inspection images having modified quantification information equal to or greater than the threshold value.

[0042] The first learning data extraction unit 21 extracts, from the database 23, inspection images that satisfy an extraction condition and damage detection results and correction histories associated with the inspection images as learning data for re-training the damage detector.

[0043] <<Data stored in the database>> Next, a description will be given of a specific example of data stored in the database 23. The database 23 stores at least the inspection images, damage detection results, and correction history uploaded from the computers B1 and B2.

[0044] FIG. 3 is a diagram showing an example of an inspection image acquired by the inspection image acquisition unit 11. The inspection image 31 is an image obtained by photographing a part of a bridge deck, which is an example of a structure to be inspected. The inspection image 31 has a crack 33 and a crack 35. The inspection image 31 is input to the computer B1 or B2, and a damage detection result is output by a damage detector mounted on the computer B1. Note that the structure is not limited to a bridge, and may be other structures such as a tunnel, a box culvert, a dam, a seawall, or a building (walls, floors, etc.).

[0045] Fig. 4 is a diagram showing the damage detection results obtained when the inspection image shown in Fig. 3 is input into the damage detector. Damage detection result 41 includes damage detection result 37 for crack 33 and damage detection result 39 for crack 35. Damage detection result 39 is good and adequately detects crack 35. On the other hand, damage detection result 37 does not adequately detect crack 33 and is a poor damage detection result. Therefore, the user makes corrections to damage detection result 37.

[0046] Fig. 5 is a conceptual diagram showing a revision history made by a user to the damage detection result 41 shown in Fig. 4. In revision history 52, an additional vector 54 has been added to the damage detection result 37 of the crack 33. That is, because the damage detection result 37 does not adequately detect the crack 33, the user manually added the additional vector 54.

[0047] The inspection image 31, damage detection result 41, and correction history 52 described in the above specific example are uploaded to computer A from computer B1 or B2.

[0048] <<Corrected quantification information>> Next, the modified quantification information will be described. The modified quantification information is calculated by quantifying the modification history by the calculation unit 17. Here, the quantification of the modification history (modification amount) is, for example, in the case of cracks, information indicating the length of a vector that has been modified and added or deleted, or the amount of change in the coordinates of a vector that has been modified and moved. Also, for example, in the case of water leakage, free lime, peeling, exposed rebar, etc., it is the area of ​​a region that has been modified and added or deleted, or the amount of change in the area of ​​a region. Then, in the image extraction condition setting unit 19, a threshold value corresponding to the modified quantification information stored in the database 23 is set, and the first learning data extraction unit 21 extracts, for example, inspection images having modified quantification information equal to or greater than the threshold value.

[0049] Calculation unit 17 may or may not include actual size information acquisition unit 18. When calculation unit 17 does not include actual size information acquisition unit 18, the correction amount in pixel size is used as correction quantification information. When calculation unit 17 includes actual size information acquisition unit 18, calculation unit 17 calculates the correction amount converted into actual size using actual size information (subject resolution (mm / pixel), or size of components in the image, etc.). By including actual size information acquisition unit 18, calculation unit 17 can calculate the correction amount more accurately.

[0050] Figures 6, 7, and 8 are diagrams showing specific examples of correction histories acquired by correction history acquisition unit 15. In Figures 6, 7, and 8, dotted lines indicate vectors (detection vectors) output by the damage detector after detecting damage (cracks), and solid lines indicate vectors after correction by the user (corrected vectors). Also, the areas where the dotted and solid lines are drawn almost parallel are areas where the detection vectors and corrected vectors actually overlap, indicating that the detection by the damage detector is good.

[0051] 6 shows a case where an additional correction has been made by the user. Specifically, in an area 61 of the correction history 56, a corrected vector that has not been output by the damage detector has been added by the user, and undetected damage has been added by the user's correction.

[0052] The correction history 58 shown in Fig. 7 shows a case where a user has corrected a false positive. Specifically, damage is detected in an area 63 of the correction history 58, but only a detection vector is drawn in the area 63, and no corrected vector is drawn. Therefore, the detection vector in the area 63 is due to a false positive, and is deleted and corrected.

[0053] In the correction history 60 shown in Fig. 8, (A) shows that the detected vector and the corrected vector overlap in some areas, but in some areas vectors have been deleted and vectors have been added. Specifically, in area 65, since only the detected vector exists, a correction has been made to delete a false positive, and in area 67, since only the corrected vector exists, undetected damage has been added. Also, (B) shows a correction history in which some of the coordinate values ​​of the vectors have been changed. Specifically, the four coordinate values ​​in the figure are different between the detected vector and the corrected vector, and the detection of damage has been partially changed.

[0054] Although specific examples of revision histories have been shown above, the revision histories are not limited to these, and various forms of revision histories may be adopted.

[0055] <<Learning data collection process>> Next, a description will be given of a learning data collection step (learning data collection method) using the learning data collection device 10. Fig. 9 is a flowchart showing the learning data collection step (learning data collection method) using the learning data collection device 10.

[0056] First, the inspection image acquisition unit 11 acquires the inspection images uploaded from the computers B1 and B2 (step S10). In addition, the damage detection result acquisition unit 13 acquires the damage detection results uploaded from the computers B1 and B2 (step S11). In addition, the correction history acquisition unit 15 acquires the correction history uploaded from the computers B1 and B2 (step S12). Thereafter, the calculation unit 17 calculates the corrected quantification information based on the acquired correction history (step S13). Thereafter, the image extraction condition setting unit 19 sets a threshold value of the corrected quantification information as an extraction condition (step S14). Thereafter, the first learning data extraction unit 21 extracts learning data to be re-learned (step S15).

[0057] The above-mentioned configurations and functions can be realized by any hardware, software, or a combination of both. For example, the present invention can be applied to a program that causes a computer to execute the above-mentioned processing steps (processing procedures), a computer-readable recording medium (non-transitory recording medium) on which such a program is recorded, or a computer on which such a program can be installed.

[0058] As described above, in this embodiment, the correction history is quantified and learning data is extracted based on the quantified extraction conditions, so that inspection images that have been corrected and where the correction is not minor are extracted as learning data, allowing the damage detector to perform efficient learning.

[0059] <Second embodiment> Next, a second embodiment will be described. In the second embodiment, the inspection image is extracted based on information obtained from the inspection image (image information of the structure and information on the image quality of the inspection image) in addition to the corrected quantification information. This makes it possible to collect learning data that allows more efficient re-learning.

[0060] 10 is a block diagram showing an example of a functional configuration of the learning data collection device 10 of this embodiment. Note that the same reference numerals are used to denote the same parts as those already described in FIG.

[0061] The learning data collection device 10 includes an inspection image acquisition unit 11, a damage detection result acquisition unit 13, a correction history acquisition unit 15, a calculation unit 17, an actual size information acquisition unit 18, an image extraction condition setting unit 19, a first learning data extraction unit 21, an image quality judgment unit 51, a database 23, a display control unit 25, and a memory unit 26.

[0062] The image extraction condition setting unit 19 of this embodiment accepts image information of a structure, which is one piece of information obtained from an inspection image. Here, the image information of a structure is information including at least one of damage size information (area size information), damage position information (area position information), damage direction information (area direction information), damage type information (area type information), and inspection image meta information. The database 23 stores this image information in association with the inspection image. Note that the image information is acquired, for example, when damage is detected by a damage detector. Specifically, the damage size information, damage position information, damage direction information, damage type information, and inspection image meta information are acquired when damage is detected by the damage detector and uploaded in association with the inspection image.

[0063] The first learning data extraction unit 21 extracts inspection images from the database 23 based on the image information and the modified quantification information. That is, the image extraction condition setting unit 19 sets the extraction conditions for the image information to be extracted (to be additionally learned) and the threshold value for the modified quantification information. Specific examples of the image information extraction conditions that may be set include a condition for extracting an image with a minimum crack width of 0.2 mm or more, or a condition for extracting an image with free lime in the center. Note that only the extraction conditions for image information may be set in the image extraction condition setting unit 19, in which case the inspection images are extracted according to the extraction conditions for the image information.

[0064] <<Image information of structure>> Next, image information of a structure will be described.

[0065] One type of image information of a structure is the size of the damage. The size of the damage is quantified by pixel size, or by actual size if an actual size information acquisition unit 18 is provided, and is used as image information of the structure. A specific example of the size of the damage is the total length of the damage in the inspection image (the length of a crack when the inspection target is a concrete structure, or the length of a crack when the inspection target is a steel member). In addition, the total length of each crack width (less than 0.1 mm, 0.1 mm to less than 0.2 mm, 0.2 mm or more, etc.) may be used as image information of the structure.

[0066] Specific examples of the size of damage include the total area of ​​damage in the inspection image (water leakage, free lime, peeling, exposed rebar, etc. when the inspection target is a concrete structure, and corrosion and deterioration of anti-corrosion function when the inspection target is a steel member), the maximum width and maximum area of ​​damage (water leakage, free lime, peeling, exposed rebar, etc.), minimum spacing (minimum distance between adjacent damage of the same type), and density (number of cracks per square meter, length [cracks, fissures], area [water leakage, free lime, peeling, exposed rebar], etc.).

[0067] One type of image information for a structure is the location of damage. The location of damage can be determined from an inspection image that captures the entire component. Alternatively, images of a component can be divided and then panoramic-combined, and the location of damage can be determined from the composite panoramic image. Specific examples of the location of damage include position (center / end, etc.) and directionality (cracks in the deck in the direction of the bridge axis / perpendicular to the bridge axis, cracks in the pier in the vertical / horizontal direction, etc.).

[0068] One of the image information of a structure is the type of damage. The type of damage is determined by the detection results of a damage detector (e.g., crack detection, water leakage detection, free lime detection, peeling detection, exposed rebar detection, crack detection, corrosion detection, deterioration of anti-corrosion function detection, etc.). The user may also specify the type of damage.

[0069] Meta information (Exif information) is one type of image information of a structure, such as the camera model, lens type, F-number, shutter speed, focal length, flash ON / OFF, number of pixels, ISO sensitivity, etc.

[0070] FIG. 11 is a diagram showing an example of a storage configuration of information obtained from an inspection image (image information of a structure) that is stored in association with the inspection image in the database 23. Note that the information indicated by the reference numeral 69 is information obtained from the learned inspection image, and the information indicated by the reference numeral 71 is supplementary information that will be described later. Information obtained from an inspection image includes, for example, the maximum crack width (mm), the minimum crack spacing (m), the presence or absence of damage other than cracks, and the name of the imaged component. For example, this information is obtained by inputting the inspection image into a damage detector. In this way, the database 23 stores image information of a structure in association with the inspection image.

[0071] <<Example of image quality judgment>> One type of information that can be obtained from an inspection image is information regarding the image quality of the inspection image.

[0072] The image quality determination unit 51 determines the image quality of the inspection image acquired by the inspection image acquisition unit 11. In this case, the image extraction condition setting unit 19 accepts information regarding the image quality of the inspection image. The first learning data extraction unit 21 extracts the inspection image from the database 23 based on the information regarding the image quality and the threshold value of the modified quantification information.

[0073] Here, various methods can be adopted for the judgment made by the image quality judging section 51. A specific example of the judgment method of the image quality judging section 51 will be described below.

[0074] <<Image quality assessment using machine learning>> The image quality judgment method performed by the image quality judgment unit 51 includes a judgment method using an image quality judger that has been subjected to machine learning. That is, the image quality judgment unit 51 is configured with an image quality judger (image quality judgment AI) using machine learning, and judges the image quality of the inspection image by the image quality judger.

[0075] <<Judgment based on spatial frequency spectrum>> The image quality determination unit 51 may quantify and determine the image quality by the maximum spectrum value, average value, or sum of the spectrum in the high frequency region in the spatial frequency spectrum of the region in the inspection image. Specifically, the greater the maximum value, average value, or sum of the components within a specific pixel number radius (r pixel radius) from the four corners of the spatial frequency spectrum image (obtained by performing a fast Fourier transform (FFT) on the captured image), the stronger (more) the high frequency components are, so there is less blurring and shaking and the image quality is better.

[0076] <<Judgment based on histogram>> In the judgment by the image quality judgment unit 51 based on a histogram (one example of an index indicating image quality), the image quality judgment unit 51 converts an individual image (a color image composed of R, G, and B components) into a grayscale image. For example, grayscale (density) = R x 0.30 + G x 0.59 + B x 0.11 (R, G, and B are the values ​​of red, green, and blue signals, respectively). The image quality judgment unit 51 calculates a histogram (density histogram; see the example in FIG. 12) of the converted grayscale image. The calculation of the histogram and the following judgment may be performed for a part of the area of ​​the individual image, not for the entire image. The image quality judgment unit 51 uses G(i) {i = 0, 1, ..., 255} as a histogram of each density value (the closer to 0, the darker, and the closer to 255, the brighter), and judges whether the individual image is too bright or too dark using the following formulas (1) and (2). The judgment thresholds (kb, hb, kd, hd) may be default values ​​(e.g., kb=205, hb=0.5, kd=50, hd=0.5) or may be set by the image quality judgment unit 51 in response to user input via the operation unit (keyboard 5 and mouse 7).

[0077]

number

[0078]

number

[0079] The image quality judgment unit 51 judges the image to be "too bright" if the ratio of density values ​​equal to or greater than kb in the above-mentioned formula (1) is equal to or greater than the overall hb. In this case, the image quality judgment unit 51 judges the image to be "low image quality (because it is too bright)" and sets the individual image to be checked. Similarly, the image quality judgment unit 51 judges the image to be "low image quality (because it is too dark)" if the ratio of density values ​​equal to or less than kd in the above-mentioned formula (2) is equal to or greater than the overall hd, and sets the individual image to be checked.

[0080] It is also possible to judge whether the gradation is crushed or not based on the histogram. For example, the image quality judgment unit 51 uses G(i){i=0,1,...,255} as a histogram of each density value and judges that "the gradation on the shadow side is crushed" when G(0)>Td, and judges that "the gradation on the highlight side is crushed" when G(255)>Tb. In these cases, the image quality judgment unit 51 judges that the image quality is low, and sets the individual image as the image to be checked. The judgment thresholds (Td, Tb) may be default values ​​(for example, Td=0, Tb=0), or may be set by the image quality judgment unit 51 according to the user's input via the operation unit (keyboard 5 and mouse 7).

[0081] As described above, the image quality determining unit 51 can determine the image quality of the inspection image by various methods.

[0082] In this embodiment, the inspection image is extracted based on not only the modified quantification information but also information obtained from the inspection image (image information of the structure and information related to the image quality of the inspection image), so that learning data that enables efficient re-learning can be collected.

[0083] <Third embodiment> Next, a third embodiment will be described. In the third embodiment, the test image is extracted based on the supplementary information of the test image in addition to the corrected quantification information. This makes it possible to collect learning data that allows more efficient re-learning.

[0084] 13 is a block diagram showing an example of a functional configuration of the learning data collection device 10 of this embodiment. Note that the same reference numerals are used to denote the same parts as those already described in FIG.

[0085] The learning data collection device 10 includes an inspection image acquisition unit 11, a damage detection result acquisition unit 13, a correction history acquisition unit 15, a calculation unit 17, an actual size information acquisition unit 18, an image extraction condition setting unit 19, a first learning data extraction unit 21, ancillary information acquisition unit 53, a database 23, a display control unit 25, and a memory unit 26.

[0086] The incidental information acquisition unit 53 acquires incidental information including at least one of identification information, component information, location information, environmental information, material information, inspection information, management information, and structure type information of the inspection image. In this case, the image extraction condition setting unit 19 accepts the incidental information, and the first learning data extraction unit 21 extracts the inspection image based on the incidental information of the inspection image and the incidental information accepted by the image extraction condition setting unit 19. That is, the extraction condition of the incidental information to be extracted (to be additionally learned) in the image extraction condition setting unit 19 is also set together with the threshold value of the corrected quantification information. For example, the extraction condition of the incidental information may be to extract an image of a bridge pier within 100 m from the coast, or to extract an image of a bridge with a traffic volume of 1000 vehicles / day or more and an age of 30 years or more. Note that only the extraction condition of the incidental information may be set in the image extraction condition setting unit 19, in which case the inspection image is extracted according to the extraction condition of the incidental information.

[0087] Specific examples of each type of additional information are described below. Identification information is, for example, the name of the bridge and ID (Identification) number. Component information is, for example, the component type (deck slab / pier / girder, etc.) or direction (bridge axis direction, vertical direction). Location information is, for example, prefecture, city, town, village, region, latitude and longitude, and distance from the sea. Environmental information is, for example, climate (temperature [average, maximum, minimum, etc.], humidity [average, maximum, minimum, etc.], rainfall, snowfall), and traffic volume. Material information is, for example, concrete aggregate size and material strength (compressive strength, tensile strength). Inspection information (concrete core test results, etc.) is chloride ion concentration, progress of neutralization, and presence or absence of alkali-aggregate reaction. Management information is age, construction conditions (temperature or humidity at the time of construction, etc.), and repair history. Structure type information is, for example, bridge, tunnel, box culvert, building, etc.

[0088] In this embodiment, by extracting learning data using the auxiliary information in addition to the corrected quantification information, it is possible to extract learning data that allows more efficient re-learning.

[0089] <Fourth embodiment> Next, a fourth embodiment will be described.

[0090] 14 is a block diagram showing an example of a functional configuration of the learning data collection device 10 of this embodiment. Note that the same reference numerals are used to denote the same parts as those already described in FIG.

[0091] The learning data collection device 10 includes an inspection image acquisition unit 11, a damage detection result acquisition unit 13, a correction history acquisition unit 15, a calculation unit 17, an actual size information acquisition unit 18, an image extraction condition setting unit 19, a first learning data extraction unit 21, a learning history acquisition unit 55, a statistical information generation unit 57, a second learning data extraction unit 59, a database 23, a display control unit 25, and a memory unit 26.

[0092] The learning history acquisition unit 55 acquires history information regarding the history of the inspection image being used for learning. Specifically, the learning history acquisition unit 55 acquires information as to whether the inspection image acquired by the inspection image acquisition unit 11 has already been used for machine learning of the damage detector of the computer B1 or B2.

[0093] The statistical information generating unit 57 generates statistical information of the image used for learning based on the acquired history information. The first learning data extracting unit 21 may extract the test image based on the generated statistical information. That is, the extraction conditions of the statistical information to be extracted (to be additionally learned) in the image extraction condition setting unit 19 are also set together with the threshold value of the corrected quantification information. Here, the extraction conditions of the statistical information are, for example, conditions that occur less frequently in the image information or the incidental information among the images already used for learning. Note that only the extraction conditions of the statistical information may be set in the image extraction condition setting unit 19, in which case the test image is extracted according to the extraction conditions of the statistical information.

[0094] The following are examples of statistical information. Note that the "%" below indicates the percentage of images that meet a certain condition out of the total number of images that have been trained.

[0095] For example, statistical information on the maximum crack width can be described as less than 0.05 mm, 0.05 mm or more and less than 0.1 mm, 0.1 mm or more and less than 0.2 mm, 0.2 mm or more and less than 0.5 mm, 0.5 mm or more and less than 1.0 mm, and 1.0 mm or more. For example, statistical information on the minimum crack spacing can be described as less than 0.2 m, 0.2 m or more and less than 0.5 m, 0.5 m or more and less than 1 m, and 1 m or more. Statistical information on damage other than cracks can be described as none, water leakage, free lime, peeling, exposed rebar, and water leakage and free lime. In addition to these, statistical information can be generated for components, distance from the sea, traffic volume, and age. Furthermore, conditions for combinations of multiple statistical information can be generated, such as maximum crack width x damage other than cracks.

[0096] The second learning data extraction unit 59 extracts learning data for retraining the damage detector from the inspection images not extracted by the first learning data extraction unit 21. For example, the second learning data extraction unit 59 extracts randomly or regularly (for example, sorting by file name and extracting every predetermined number of images) from the inspection images not extracted by the first learning data extraction unit 21. The second learning data extraction unit 59 also extracts randomly, excluding inspection images of low image quality. The second learning data extraction unit 59 extracts the same number of inspection images as the number of inspection images extracted by the first learning data extraction unit 21. By training the learning data extracted by the second learning data extraction unit 59, it is possible to prevent the learning from being too influenced by inspection images under biased conditions.

[0097] <Other examples> <<Panorama composite image>> Next, a case where a panoramic composite image has a correction history will be described.

[0098] The inspection images acquired by the inspection image acquisition unit 11 include a panoramic composite image. That is, the inspection image acquisition unit 11 acquires divided images obtained by dividing and photographing a structure, and a panoramic composite image obtained by synthesizing the divided images. In this case, the correction history acquisition unit 15 acquires the correction history made to the panoramic composite image.

[0099] The first learning data extraction unit 21 extracts an area on the panorama composite image based on the correction history on the panorama composite image, and extracts at least one divided image that constitutes the area.

[0100] FIG. 15 is a diagram conceptually showing a correction history on a panoramic composite image acquired by the inspection image acquisition unit 11. As shown in FIG.

[0101] The user has made modifications (additions or deletions) on the panoramic composite image 71 shown in FIG. 15. The modifications made by the user are indicated by vectors (1) to (5). The modifications made by the user exist across divided images 73 and 75 which form part of the panoramic composite image 71. In such a case, the first learning data extraction unit 21 can automatically select the inspection images, for example, in the following pattern. Note that in this case, the modifications made to the panoramic composite image 71 are modifications which satisfy the extraction conditions set by the image extraction condition setting unit 19.

[0102] The first learning data extraction unit 21 selects all images including vectors (1) to (5) as pattern 1. That is, the first learning data extraction unit 21 selects divided image 73 and divided image 75 as learning data. Furthermore, the first learning data extraction unit 21 selects all divided images including vectors (1) to (5) when the length of vectors (1) to (5) is equal to or greater than a predetermined threshold as pattern 2. Furthermore, the first learning data extraction unit 21 selects a divided image including vectors (1) to (5) with the longest length as pattern 3. In this case, since divided image 75 has vectors (1) to (5) longer than divided image 73, divided image 75 is selected. Furthermore, the first learning data extraction unit 21 compares the amount of correction included in divided image 73 and the amount of correction included in divided image 75 with a predetermined threshold as pattern 4, and determines whether to extract each image. Furthermore, the first learning data extraction unit 21 determines the image quality of images that include vectors (1) to (5) as pattern 5, and selects the image with the highest image quality.

[0103] <<Image confirmation display section>> Next, the image confirmation display unit that displays the test image extracted by the first learning data extraction unit 21 or the second learning data extraction unit 59 will be described.

[0104] The learning data collection device 10 can be equipped with an image confirmation display unit constituted by a monitor 9.

[0105] FIG. 16 is a diagram showing an example of a confirmation screen for an extracted inspection image displayed on the monitor 9. In the example shown in FIG. 16, an extracted inspection image 77 is shown. The confirmation screen also has a selection button 81 for selecting whether or not to adopt the extracted inspection image as learning data. Also, a command button 81 is provided to clearly indicate missed detection points, erroneous detection points, etc. based on the correction history. A button 79 for accepting a command is provided. The confirmation screen also has a display 83 of information related to the examination image (information obtained from the image and / or additional information).

[0106] In this way, since the learning data collection device 10 has an image confirmation display unit, the user can confirm the extracted test image, and learning data with a higher learning effect can be collected.

[0107] <<Other tests>> In the above description, an example in which the present invention is applied to inspecting a structure for damage has been described. The example of the inspection to which the present invention is applied is not limited to this. The present invention is applied to the case in which a region (including the region of an object) is detected by a region detector. For example, when the object of inspection is a human body, the present invention is also applied to the case in which blood vessels are detected from a CT (Computed Tomography) image, which is an inspection image of the human body. The present invention is also applied to the case in which products such as articles and medicines are used as objects and images are used to inspect for surface scratches and defects (visual inspection, etc.). The present invention is also applied to the inspection of images of damage and defects inside a structure using X-rays.

[0108] Although the present invention has been described above with reference to examples, it goes without saying that the present invention is not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present invention. [Explanation of symbols]

[0109] 10: Learning data collection device 11: Inspection image acquisition section 13: Damage detection result acquisition unit 15: Revision history acquisition section 17: Calculation section 18: Actual size information acquisition section 19: Image extraction condition setting section 21: First learning data extraction unit 23: Database 25: Display control unit 26: Storage section A: Computer B1: Computer B2: Computer C1, C2: Upload data D: Retrained damage detector or detection parameter file E1, E2: Inspection images

Claims

1. an acquisition unit that acquires an area detection result indicating an area detected by the area detector based on an inspection image obtained by capturing an image of an object to be inspected; a calculation unit that calculates correction quantification information of the inspection image by quantifying a correction history of the area detection result; an extracting unit that extracts, from the inspection image, the inspection image for training the area detector as training data based on at least a threshold value of the modified quantification information; A learning data collection device comprising:

2. the area detection result is a detection result of a damaged area detected by the area detector, The correction history is a correction of the detection result of the damaged area. The learning data collection device according to claim 1 .

3. The learning data collection device according to claim 1 , wherein the calculation unit calculates the correction quantification information by quantifying based on dimensions of the correction history.

4. The learning data collection device according to claim 1 , wherein the threshold value is a dimension corresponding to the corrected quantification information.

5. an image quality determination unit for determining an image quality of the inspection image; The extraction unit receives information regarding image quality of the inspection image, 5. The learning data collection device according to claim 1, wherein the inspection image is extracted based on the information regarding image quality and the threshold value.

6. obtaining an area detection result indicating an area detected by the area detector based on an inspection image obtained by capturing an image of an object to be inspected; calculating modified quantified information of the inspection image by quantifying a modification history of the region detection result; extracting, from the test images, test images for training the area detector based on at least a threshold value of the modified quantification information; A method for collecting data for learning, including:

7. obtaining an area detection result indicating an area detected by the area detector based on an inspection image obtained by capturing an image of an object to be inspected; calculating modified quantified information of the inspection image by quantifying a modification history of the region detection result; extracting, from the test images, test images for training the area detector based on at least a threshold value of the modified quantification information; A program for causing a computer to execute a learning data collection step including the steps of:

Images