Information processing device, method for controlling information processing device, and storage medium

Abstract

The present disclosure discloses an information processing device, a method for controlling the information processing device, and a storage medium. The information processing device includes: a receiving unit configured to receive an input of a position of a detection target included in a specified image; an acquisition unit configured to acquire a storage amount of training data including a pair of information indicating the image and information indicating the position specified by the input; a training unit configured to train a training model for detecting the detection target from the image based on the stored training data; and a display control unit configured to control the display unit to display the storage amount and a reference amount of the training data in a comparable manner, the reference amount being set to the amount of training data necessary for the training unit to train the training model.

Description

Technical Field

[0001] The present disclosure relates to a technique for detecting objects from images by using machine learning. Background Art

[0002] When inspecting the wall surface of a structure such as a bridge and inspecting the appearance of the exterior of parts and products, an inspector checks for defects such as cracks by visual observation. Such inspection operations require high operating costs. Thus, Japanese Patent No. 5384429 discusses an inspection method for automatically detecting defects from a captured image of an inspection target by using a pattern recognition technique.

[0003] In the pattern recognition technology discussed in Japanese Patent No. 5384429, in order to train a classifier to achieve target detection accuracy, a sufficient number of multiple types of training data corresponding to the characteristics of the detection target defects are required. In the method for generating training data, a user manually inputs defect data visually inspected by the user for a portion of a captured image of an inspection target to use the data as training data. When a sufficient amount of training data has been stored, a classifier is trained by using the training data, and a process for detecting defects is performed by using the trained classifier on the remaining images. This improves the efficiency of the operation for detecting defects occurring in the inspection target structure. However, since the process for generating training data takes time and effort, it is necessary to efficiently store the required amount of training data. Summary of the invention

[0004] According to one aspect of some embodiments, an information processing device includes: a receiving unit configured to receive an input of a position of a detection target included in a specified image; an acquisition unit configured to acquire a storage amount of training data including pairs of information indicating the image and information indicating the position specified by the input; a training unit configured to train a training model for detecting the detection target from the image based on the stored training data; and a display control unit configured to control a display unit to display the storage amount and a reference amount of the training data in a comparable manner, the reference amount being set to an amount of training data required for the training unit to train the training model.

[0005] Further features of different embodiments will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1A , Figure 1B and Figure 1C is a diagram illustrating an example of defect data to be stored as training data and the defect data superimposed on a captured image.

[0007] Figure 2is a diagram schematically illustrating screen transition when defect data is input to an inspection target image.

[0008] Figure 3A and Figure 3B is a block diagram illustrating an example of the configuration of an information processing apparatus.

[0009] Figure 4 is a table illustrating an example of a data classification table.

[0010] Figure 5 is a flowchart illustrating an example of processing performed by the information processing apparatus.

[0011] Fig. 6A and Figure 6B is a diagram schematically illustrating a process for storing training data.

[0012] Fig. 7A and Figure 7B is a diagram illustrating an example of display data displayed by the information processing device.

[0013] Fig. 8A and Figure 8B It is a diagram illustrating another example of display data displayed by the information processing device.

[0014] Fig. 9A and Fig. 9B is a block diagram illustrating another example of the configuration of the information processing apparatus.

[0015] Fig.10 is a flowchart illustrating another example of the processing performed by the information processing apparatus.

[0016] Fig.11A , Fig. 11B , Fig. 11C and Fig.11D is a diagram illustrating an example of a verification target image group and corresponding defect data.

[0017] Fig. 12A , Fig. 12B and Fig. 12C 1 is a diagram schematically illustrating a process for prominently displaying a local area for which input of defect data is to be preferentially presented.

[0018] Fig.13A , Fig. 13B and Fig. 13C is a diagram illustrating an example of defect detection results before and after performing a correction operation.

[0019] Fig.14 is a diagram illustrating an example of display data displayed by the information processing device.

[0020] Fig.15is a flowchart illustrating still another example of the processing executed by the information processing apparatus.

[0021] Fig.16A , Fig. 16B and Fig. 16C It is a diagram illustrating another example of display data displayed by the information processing device. DETAILED DESCRIPTION

[0022] The exemplary embodiments will be described in detail below with reference to the accompanying drawings. The configurations described below in the exemplary embodiments are representative examples, and the scope of some embodiments is not limited to specific configurations.

[0023] The first exemplary embodiment will be described below using an example of a process for displaying the storage status of training data to a user who visually inspects an inspection target image to identify defects and prompting the user to issue an instruction for performing training by using the stored training data depending on the storage status. In particular, the present exemplary embodiment will be described below using an example of an information processing system for performing so-called inspection on infrastructure to check for aging deterioration of structures such as bridges.

[0024] The terms used in the description of the present exemplary embodiment are defined below. In the description of the information processing system for performing inspections on infrastructure, the term "inspection target" refers to a concrete structure. A user of the information processing system according to the present exemplary embodiment intends to inspect a defective area to determine whether defects such as cracks exist on the surface based on a captured image of the inspection target. For example, in the case of a concrete structure, the term "defective area" refers to cracks, delamination, and spalling of concrete. The term also refers to an area with defects such as weathering, exposure of reinforcing iron rods, rust, water leakage, water dripping, corrosion, damage, cold joints, sediments, and rock pockets.

[0025] The term "defect data" refers to a combination of defect type and position information about the location where the defect area exists. For example, a user visually inspects a captured image of an inspection target and identifies a defect. The defect data includes position information of the identified defect represented by a line and information about the defect type. The defect data may also include detailed information such as the width of a crack. The defect data may include defect type and position information obtained as a result of defect detection processing performed by an information processing device. However, in the case where the defect type is identified as one type or in the case where it is not necessary to distinguish the defect type, the position data is handled as defect data. The term "training data" refers to a pair of a captured image of an inspection target as a subject and defect data. In the present exemplary embodiment, in particular, a captured image of the inspection target is used as input data, and a classifier of a defect detection method is trained by using training data in which the defect data is used as correct answer data (supervisory data). In the present exemplary embodiment, the defect data input by the user is used as defect data to be used as training data. However, the result of the defect detection method can also be used as defect data after being corrected by the user.

[0026] The term "data classification" refers to classification information for classifying characteristics and properties of training data. In the present exemplary embodiment, the storage status of training data is displayed for each data classification, thereby allowing the user to determine whether the desired training data has been stored. In the present exemplary embodiment, data classification in the information processing system related to the inspection of infrastructure is performed, for example, by the type of concrete wall surface including defective areas. More specifically, data is classified into each type according to the color and surface roughness of the concrete wall surface.

[0027] First, we will refer to Figure 1A , Figure 1B , Figure 1C and Figure 2 An example of a screen displayed by the information processing apparatus according to the present exemplary embodiment is described. Figure 1A , Figure 1B and Figure 1C is a diagram illustrating a captured image of a concrete wall surface, defect data indicating an input result of a defect area in the application window 100 , and the defect data superimposed on the captured image. Figure 1AThe state in which only the captured image 110 of the concrete wall surface as the inspection target is displayed is illustrated. Hereinafter, the captured image of the concrete wall surface is referred to as the inspection target image or simply as the image. The image 110 includes cracks (defective areas) 111, 112, and 113. The data input operator or the inspection manager (hereinafter referred to as the user) visually checks the cracks 111 to 113 present in the image 110 and performs a defect data input operation. The defect data input operation refers to an operation of inputting linear data on the image, for example, by tracing the defect with the aid of a mouse or a handwriting tablet. This allows the user to input data indicating the positions of the cracks 111 to 113 as the detection target according to the present exemplary embodiment based on the coordinate system associated with the image. Figure 1B Illustrate the user based Figure 1A The image 110 shown in FIG. 1 is a state where defect data 121, 122, and 123 are input. Although the actual crack may be Figure 1A The areas shown in each have an area, but Figure 1B And the corresponding defect data 121 to 123 shown in the subsequent drawings are indicated by thin solid lines to simplify the description. However, the information processing apparatus may handle each piece of defect data as an area and receive an input operation in which the position and area of ​​the defect data are defined. Figure 1C is an example Figure 1B The defect data 121, 122 and 123 shown in FIG. 1 are superimposed on Figure 1A FIG. 110 is a graph of an image 110 shown in FIG.

[0028] In the case where there are a large number of inspection target images and in the case where the inspection target images have a large size, the defect data input operation is particularly time-consuming and laborious. In this case, in order to improve the efficiency of the operation, it is effective to acquire defect data by not only performing a manual tracing operation but also performing a defect detection method for detecting defects by using a classifier for identifying defects included in an image. In this case, the user can check whether the defect detection result is correct by comparing the inspection target image with the defect data, and then correct the defect detection result only when necessary. This reduces the operational burden compared to the case where defect data is input from scratch. The present exemplary embodiment will be described below on the premise of training a classifier to support processing for identifying defect types. However, in terms of the function of detecting defects (cracks) from an image, the classifier is equivalent to a training model (trained model) called a detector.

[0029] In order to train the classifier, it is necessary to prepare a sufficient amount of training data of various types. The user inputs the defect data, and Figure 1CThe pair of image and defect data is obtained as shown in . The pair can be used as training data. More specifically, at the same time as the user inputs the defect data, the pair of image and defect data is stored as training data. When the amount of training data of a predetermined type reaches a sufficient amount, training of a classifier is performed using the stored training data. Then, by using the trained classifier, the defect detection method is applied to the remaining inspection target images to obtain detection results. Therefore, the burden of the defect data input operation is reduced.

[0030] In this exemplary embodiment, while the user inputs the defect data, the storage status of each type of training data is displayed on a predetermined display unit. This allows the user to grasp the storage status of each type of training data, so that the defect data input operation can be performed efficiently to obtain the necessary amount of training data.

[0031] The following will refer to Figure 2 Describes methods for displaying the storage status of training data. Figure 2 is a diagram schematically illustrating screen transition when a user inputs defect data onto an inspection target image. Figure 2 The following state is illustrated, in which images in the inspection target image group 200 are sequentially displayed one after another, and the user inputs defect data while checking whether there is a defect in each image. The window 210 displays the selected image 201 and a progress bar for each type is displayed in the progress bar display area 220 indicating the storage status of the training data. Figure 2 , the bar of the progress bar 221 has not reached the upper limit, indicating a state in which the amount of training data is insufficient to perform training. The user performs a defect data input operation while visually checking the image 201. The window 211 indicates a display state when defect data is input. At this time, a pair of the image 201 and defect data (defects 231, 232, and 233) on the window 211 is stored as training data, and the bar of the progress bar 222 increases. In this way, while the user inputs defect data, training data is stored for each type.

[0032] The window 212 shows a state in which the defect data input operation has been performed for a predetermined number of images and a sufficient amount of training data has been stored for each type. In the progress bar display area 240 on the window 212, the bars of all the progress bars have reached the upper limit. At this time, a screen is displayed to prompt the user to issue an instruction for executing training. For example, an "Execute Training" button 241 is displayed in the window 212. The method for displaying the storage status of the training data and the method for determining whether to prompt the user to execute the training will be described in detail below.

[0033] In the following description, the image handled in the present exemplary embodiment is a visible light image (red, green, and blue (RGB) image) capturing the surface of a concrete wall as an inspection target, but some embodiments are not limited thereto. For example, some embodiments are also applicable to thermal images captured by infrared cameras and images captured by line sensor cameras.

[0034] Figure 3A and Figure 3B An example of the configuration of the information processing apparatus 300 according to the present exemplary embodiment is illustrated. Figure 3A 300 according to the present exemplary embodiment. The information processing apparatus 300 can be implemented by a computing machine including a central processing unit (CPU) 301, a main storage device 302, an auxiliary storage device 303, an operation unit 304, a display unit 305, a network interface (I / F) 306, and a system bus 307. The components are connected via the system bus 307 in a manner that enables mutual communication. The computing machine may be a general-purpose personal computer (PC) or another information processing apparatus (such as a server apparatus, a tablet apparatus, or a built-in computer, etc.). Hardware components optimally designed for processing according to one or more embodiments are also applicable.

[0035] The CPU 301 is a processor for controlling the information processing apparatus 300. The main storage device 302 is a storage device (such as a random access memory (RAM)) used as a work area and a temporary data storage area of ​​the CPU 301. The auxiliary storage device 303 is a storage device for storing various programs, various setting information, various display conditions, and view data. The auxiliary storage device 303 is, for example, a read-only memory (ROM), a hard disk drive (HDD), or a solid-state drive (SSD). Each of the main storage device 302 and the auxiliary storage device 303 is an example of a storage unit.

[0036] The operation unit 304 (such as a mouse, keyboard, touch pad, touch panel, and handwriting tablet) is an input device for inputting information to the information processing device 300. The display unit 305 is a display device such as a monitor, a display, and a touch panel display. The network I / F 306 is an interface for communicating with an external device via a network. According to the present exemplary embodiment, the operation unit 304 and the display unit 305 are different components. However, as another example, the operation unit 304 and the display unit 305 can be a single component having both the function of an input device (such as a touch panel and a tablet) and the function of a display device. These units can be external devices connected to the information processing device 300 via an input / output interface.

[0037] Figure 3B301 is an example of a block diagram illustrating a software configuration of the information processing apparatus 300. The information processing apparatus 300 includes a receiving unit 311, an acquiring unit 313, a setting unit 314, a display control unit 315, a training unit 317, and a detecting unit 318. Each functional unit is realized by the CPU 301 loading a program stored in the auxiliary storage device 303 into the main storage device 302 and then executing a process corresponding to each flowchart described below. Then, the CPU 301 stores the execution result of the process in the main storage device 302. For example, in the case where a hardware component is configured as a substitute for a software process by using the CPU 301, a calculation unit or a circuit corresponding to the process of each functional unit described below needs to be configured.

[0038] The receiving unit 311 receives operation information input by the user via the operating unit 304. In the present exemplary embodiment, the receiving unit 311 receives input for specifying the position of the detection target included in the captured image (detection target image) of the inspection target. In the present exemplary embodiment, the detection target is a crack. The receiving unit 311 receives a user operation on a graphical user interface (GUI) item displayed on the display unit 305, and notifies each relevant functional unit of the user operation. The acquisition unit 313 acquires the storage amount of the training data stored in the storage unit 312. The setting unit 314 sets the reference amount of the training data required for training based on the user input or the past training results. In the present exemplary embodiment, the reference amount refers to the amount of training data required for training in order to perform the detection process with the required accuracy. The reference amount is preset as a value indicating a lower limit. For example, in a certain setting method, in the case where at least X detection target images at a predetermined size require input of defect data, X is defined as the reference amount. The display control unit 315 is a functional unit for controlling the content to be displayed on the display unit 305. In the present exemplary embodiment, defect data representing the inspection target is superimposed on the inspection target image, and the reference amount and the storage amount of the training data are displayed in a comparable manner.

[0039] The determination unit 316 determines whether the storage amount of the training data has reached the set reference amount. In the present exemplary embodiment, in the case where the determination unit 316 determines that the storage amount of the training data has reached the set reference amount, the determination unit 316 notifies the display control unit 315 of the determination. Then, the display control unit 315 presents a GUI for issuing an instruction for executing training to the user. Displays such as Figure 2 A display item such as the “Execute Training” button 241 shown in FIG. 1 is displayed to prompt the user to determine whether to execute the training.

[0040] After the storage amount of the training data has reached the set reference amount, the training unit 317 trains a training model for detecting the detection target from the inspection target image based on the stored training data. In the present exemplary embodiment, the training is performed in response to receiving a user operation that specifies the "Execute Training" button 241, which is a GUI displayed as a result of the determination performed by the determination unit 316. The detection unit 318 detects the detection target from the inspection target image by using the classifier (training model) that has been trained by the training unit 317. However, in the case where the information processing device 300 is a learning device in an information processing system that includes a learning device for performing training and a detection device for performing detection as different devices, the detection unit 318 can be omitted.

[0041] The storage unit 312 is a functional unit of the auxiliary storage device 303 or an external storage device connected via the network I / F 306. The storage unit 312 stores the inspection target image, the defect data corresponding to the inspection target image, and the information about the characteristics such as texture and the information about the defect type based on the inspection target image as a data classification table. Among these data, the pair of the inspection target image and the defect data is called training data according to the present exemplary embodiment. In other words, the training data is stored in the storage unit 312. The inspection target image can be replaced with a feature quantity obtained by performing feature extraction. By configuring to store feature quantity data instead of image data, the amount of data to be stored in the storage unit 312 can be reduced. In any case, the information representing the inspection target image is stored as input data of the training data.

[0042] Figure 4 is a table illustrating an example of a data classification table stored in the storage unit 312. The data classification according to the present exemplary embodiment refers to classification information for classifying characteristics of training data, and is used to determine whether training data having necessary characteristics has been stored. Figure 4 An example of a data classification table for training data for inspection of infrastructure is illustrated. The data classification table includes a classification identifier (ID) for each item and other information. The classification ID is a unique number automatically assigned in the order of registration of data classification. The characteristics of the training data are classified in advance by the classification ID. Figure 4 In , each individual category associated with a category ID corresponds to a combination of background features and detected targets. Figure 4 In , a combination of the color and texture of the concrete wall surface (background features) and the defect pattern (detection target type) is used as an item for defining the classification, but another item may also be used.

[0043] The content of specific processing executed by the information processing apparatus 300 according to the present exemplary embodiment will be described below with reference to a flowchart. Figure 5 501 is a flowchart illustrating an example of main processing performed by the information processing apparatus 300 according to the present exemplary embodiment. In the following description, each operation number will be indicated with a leading S. In the present exemplary embodiment, when an application for executing a process of detecting a defect from an inspection target image is activated in the information processing apparatus 300, a work area of ​​the CPU 301 is initialized, and the process in S501 is started.

[0044] In S501, the setting unit 314 sets a reference amount of training data, which is the amount of data required to perform training. The basic reference amount according to the present exemplary embodiment is an amount predetermined for each image and each defect type. The reference amount of training data can be set using any application method. For example, obtaining a reference amount based on an experiment can be a possible method. As another method, the user can directly specify the reference amount of training data for each type.

[0045] In S502, the display control unit 315 reads at least one image of the inspection target image group 200 from the storage unit 312 and displays the image on the display unit 305. In S503, the receiving unit 311 receives an operation for specifying the position of the defect data in the displayed inspection target image. In the present exemplary embodiment, the form of the screen displayed on the display unit 305 in the processing in S502 to S503 is as follows: Figure 2 , as shown in windows 210 to 212 in . At this time, in addition to the position of the defect data, the user may also be able to specify the defect pattern by selecting a defect pattern (type) from the options. Each time an input of defect data is received, the receiving unit 311 stores the displayed pair of the inspection target image and the defect data in the storage unit 312 as training data. In S504, the acquisition unit 313 acquires the storage status of the training data from the storage unit 312. In the present exemplary embodiment, the acquisition unit 313 acquires the storage status of the training data consisting of the pair of defect data and the corresponding image input by the user in the storage unit 312.

[0046] The following will refer to Fig. 6A and Figure 6B A process for storing training data according to the present exemplary embodiment is described. Fig. 6A and Figure 6BThe process for storing training data according to the first exemplary embodiment is schematically illustrated. In the present exemplary embodiment, images of the inspection target image group 200 are displayed one by one, and on each of the images, the user performs input of defect data. Then, the CPU 301 stores the pair of the input defect data and the displayed image as training data. Therefore, before the defect data input operation is completed for all images in the inspection target image group 200, the results of the operation performed so far are stored as training data. When the storage amount of such data reaches the reference amount, it becomes possible to perform training to perform detection and recognition of cracks based on the classifier on the remaining images of the inspection target image group 200. Fig. 6A A state is illustrated in which defect data is input to an image group 602 which is a part of an inspection target image group 601 and defect data 611 is obtained. Fig. 6A In FIG. 6 , pairs of an image group 602 and defect data 611 enclosed by a dotted line form training data.

[0047] The information processing device 300 according to the present exemplary embodiment analyzes the training data to obtain the characteristics of the image and the defect type. Then, based on the acquired information and the data classification information registered in the data classification table, the training data is stored for each classification. In the present exemplary embodiment, the color of the concrete wall surface in the image and the texture of the image indicating the smoothness are acquired as information about the characteristics of the image (inspection target image). In the inspection of the infrastructure, a captured image of the concrete wall surface is used. Therefore, the color of the concrete wall surface can be determined based on a feature amount such as an average brightness value. In order to acquire the texture of the image, for example, textures such as unevenness and smoothness can be determined based on spatial frequency information obtained by fast Fourier transform (FFT). Since the defect type is information maintained by the defect data, the defect type can be acquired by analyzing the defect data. Based on the information acquired by the analysis and the information in the data classification table stored in the storage unit 312, the training data can be stored for each data classification. After classification, by calculating the amount of data for each data classification, the type of data included in the training data (data classification) and the amount of data for each classification can be acquired, such as Figure 6B In S504, the acquisition unit 313 acquires the storage amount of training data for each classification in this way.

[0048] Return to reference Figure 5 As described in the flowchart shown in , in S505, the display control unit 315 generates display data for displaying the storage status of the training data acquired by the acquisition unit 313 on the display unit 305, and outputs the display data to the display unit 305. Figure 2As shown in FIG. 5 , in the present exemplary embodiment, a progress bar display area 220 indicating the storage amount relative to the reference amount of the training data set in S501 is displayed in the upper portion of the screen. Displaying the storage amount as a progress bar indicating the degree of progress relative to the reference amount allows the user to easily compare the storage amount with the reference amount and check the difference between these amounts.

[0049] In S506, the determination unit 316 determines whether the storage amount of the acquired training data has reached the set reference amount. In the present exemplary embodiment, unless otherwise specified, the determination unit 316 comprehensively determines whether the storage amount of the training data for each data classification has reached a predetermined amount. More specifically, the determination unit 316 determines whether the storage amount acquired in S504 has reached the reference amount of the training data set in S501. The determination method can be represented, for example, by the following inequality (1).

[0050] D p ≤V p (p=1, 2, ..., P) (1)

[0051] Parameters Dp and Vp represent the reference amount of training data and the storage amount of training data in the pth data classification, respectively. Using inequality (1), the determination unit 316 determines whether the storage amount of training data has reached the reference amount for each data classification.

[0052] When the determination unit 316 determines that the storage amount has reached the reference amount (S506 is "Yes"), the processing proceeds to S507. In S507, the display control unit 315 causes the display unit 305 to display the "Execute Training" button 241 to prompt the user to perform training. In S508, the determination unit 316 determines whether the user has issued an instruction for performing training. In the present exemplary embodiment, when the user operates the "Execute Training" button 241, the determination unit 316 determines that the user has issued an instruction for performing training. If the determination unit 316 determines that the user has issued an instruction for performing training (S508 is "Yes"), then the processing proceeds to S509. In S509, the training unit 317 performs training of the classifier (training model) based on the stored training data.

[0053] In S510, the determination unit 316 determines whether the CPU 301 has issued an instruction for terminating an application for executing defect detection processing from an inspection target image. If the determination unit 316 determines that an instruction for terminating the application has not been issued ("No" in S510), the process returns to S502. In S502, the receiving unit 311 receives a defect data input operation for the same inspection target image or another inspection target image. On the other hand, if the determination unit 316 determines that an instruction for terminating the application has been issued ("Yes" in S510), then Figure 5 The processing of the flowchart shown in is ended. If the determination unit 316 determines that the storage amount has not reached the reference amount ("No" in S506), the processing skips S507 to S509 and proceeds to S510. If the determination unit 316 determines that the user has not issued an instruction to perform training ("No" in S508), the processing returns to S502. In S502, the receiving unit 311 receives a defect data input operation for the next inspection target image. This completes the description of the example of the main processing performed by the information processing device 300 according to the present exemplary embodiment.

[0054] In the processing in S506, the method for determining whether the storage amount has reached the reference amount can be changed depending on the characteristics of the inspection target or the detection target and the accuracy of the detection result expected by the user. For example, the determination can be performed independently for each data classification. In this case, the determination unit 316 determines whether the storage amount of the training data has reached the reference amount for one or more data classifications among the five different data classifications based on inequality (1). By performing the determination based on the storage amount of some data classifications with large data amounts in the data classification by the determination unit 316, the storage amount of the training data of the defects that rarely occur can be ignored. As another determination method, the determination unit 316 can determine whether the storage amount of the training data has reached the reference amount for one or more data classifications pre-set among the data classifications. This determination method is effective in the case of pre-identifying the desired training data classification. In this case, the determination can be achieved by the user pre-setting the data classification to be stored. As another determination method, the determination unit 316 can determine whether the total storage amount of the training data has reached the reference amount, regardless of the type of data classification.

[0055] The following will refer to Fig. 7A , Figure 7B , Fig. 8A and Figure 8B Display data to be displayed on the display unit 305 by the display control unit 315 in the main process is described in detail. Fig. 7A700. A progress bar 702 and a progress rate 703 indicating the overall storage status are displayed together with an inspection target image 701 in an application window 700. When the input of defect data is received by the receiving unit 311 and new training data is stored, the bar of the progress bar 702 and the progress rate 703 increase to indicate the state of the storage amount increasing. The color and thickness of the bar of the progress bar 702 and the text of the progress rate 703 can change with the storage status. Changing the display with the storage status in this way makes it easier to intuitively grasp the storage status.

[0056] Figure 7B is a table showing an example of detailed display of the storage status of training data. Figure 7B , an application window 711 displays the storage status 712 for each data classification in a list form. Displaying detailed information in this way makes it easier for the user to grasp the level of storage capacity of training data for each data type. The displayed items include concrete wall surface, texture, defect pattern, storage capacity, and storage rate. However, other items can also be displayed. The storage status is initially displayed in descending order of storage rate. However, it is expected that the display order can be freely changed by the user. Possible methods for controlling the display order include selecting the display order from a drop-down menu 713 for determining the display order and clicking a sort control icon 714 for each item.

[0057] This screen for displaying the storage status of training data can be changed according to the user's requirements. Fig. 7A Click the progress bar 702 or the progress rate 703 in the screen shown, then Figure 7B The detailed screen shown in the figure appears. If the user clicks Figure 7B , then Fig. 7A The screen shown in appears. This screen transition enables the user to view information at the desired level each time as needed. Depending on the method for calculating the storage amount, a difference may occur between the timing when defect data is input into one of the inspection target images and training data is added, and the timing when the progress of the progress bar indicating the storage status of the training data and the numerical value of the progress rate are updated. Therefore, the method for calculating the storage status can be adjusted so that the timing when the progress rate of the progress bar reaches 100% is associated with the processing in S506 for comparing the storage amount with the reference amount. By making the timing when the bar of the progress bar reaches the upper limit (the timing when the progress rate reaches 100%) coincide with the timing when the storage amount reaches the reference amount, it becomes easier for the user to grasp the timing when the training becomes executable.

[0058] Fig. 8A exemplifies a screen displayed in the process in S507 due to the storage amount reaching the reference amount. Fig. 7A and Figure 7B , an inspection target image 801, a progress display area 802, and a progress rate 803 are displayed in an application window 800. The application window 800 represents a screen displayed immediately after the input of defect data has been completed for the previous inspection target image, and the next image has been displayed. Therefore, no defect data input by the user is superimposed on the inspection target image 801. In the progress display area 802, the progress bar has reached the upper limit, and the progress rate 803 is 100%, which indicates that a sufficient amount of data required for training has been stored. At this time, in the present exemplary embodiment, an "Execute Training" button 804 is displayed near the progress rate 803 to notify the user that the training has become executable and prompt the user to execute the training. In the present exemplary embodiment, the "Execute Training" button 804 is displayed after the determination process in S506. However, different display methods are also applicable. For example, the "Execute Training" button 804 can be displayed in a disabled state all the time, and can be enabled immediately after the determination process in S506. The disabled state and the enabled state can be indicated by, for example, changing the color of the button.

[0059] In order to prompt the user to issue an instruction for performing training, display methods other than the above display methods are also applicable. For example, displaying a pop-up window to prompt the user to perform training is one possible method. An example of a pop-up window is shown in FIG. Figure 8B . At this time, only the pop-up window is activated, while the other windows are disabled. By changing the active window in this way, the user can easily notice that the training has become executable even when the user is performing the defect data input operation. Basically, the timing of displaying the pop-up window can be immediately after the determination process in S506. At this timing, it is easier for the user to perform the necessary training most quickly. However, another timing is also applicable. For example, a pop-up window can be displayed at the timing of suspending or ending the application to prompt the user to perform the training.

[0060] The first exemplary embodiment has been described above, in which the storage status of the training data is displayed, and the user is notified that the training is executable based on the storage status of the training data, and the user is prompted to perform the training. For example, in particular in a certain application, the form of entrusting the user with the determination of performing the training in this way can be effectively utilized. This application is combined with a service that provides the training of the classifier as a charging option and the processing of detecting the detection target by using the trained classifier. In this case, the user can determine whether to rely on the automatic detection of the classifier after performing the minimum necessary operation while feeling the burden associated with the user's defect data input operation. However, the information processing device 300 can automatically perform the training without the user's instruction. More specifically, if the determination unit 316 determines that the storage amount has reached the reference amount ("Yes" in S506), the processing proceeds to S509. In S509, the training unit 317 performs the training. By automatically performing the training, it is possible to avoid the training data from being redundantly stored and prevent the user from forgetting to perform the training.

[0061] In the above exemplary embodiments, the case where pairs of user input defect data and corresponding images are stored as training data has been described. However, in some embodiments, other data is used as training data. For example, a training data database (DB) may be prepared in advance, and when the storage amount of training data is calculated in the process in S504, data is taken out from the training data DB to be used as training data.

[0062] In the above-described exemplary embodiment, in the inspection of infrastructure, defects such as cracks are inspected in the operation of inspecting a captured image of an inspection target. However, the application of the exemplary embodiment is not limited to a specific field. For example, the exemplary embodiment is applicable to an appearance inspection operation for inspecting faults such as scars (detection targets) from an image of a product (inspection target) in a factory, and a medical diagnosis operation for inspecting lesions (detection targets) from a captured image of a human body (inspection target) in a hospital. In addition, the image is not limited to an optically captured image and may be, for example, vector data.

[0063] In the first exemplary embodiment described above, the user can check the storage status of the training data on the screen while performing an input operation of the training data to detect and identify defects such as cracks or scars. When the training data reaches a predetermined amount, a screen output indicating that the training has become executable is performed, so that the user can appropriately determine the timing of executing the training.

[0064] The first exemplary embodiment has been described above using an example, in which, in the middle of the defect data input operation, a pair of defect data input to a portion of the inspection target image and a corresponding image is stored as training data, and the storage status is displayed. When the amount of training data is less than the reference amount (necessary amount), the user continues the defect data input operation to compensate for the insufficient amount of training data. In order to efficiently perform the input operation, it is necessary to input defect data so that training data of insufficient types (data classifications) are preferentially stored. The second exemplary embodiment will be described below using an example, in which an image corresponding to a type of training data insufficient for the reference amount is selected from the inspection target image group and displayed to support preferential execution of the defect data input operation for this type of training data. More specifically, based on the storage status of the training data, a data classification in which the amount of training data is insufficient for the reference amount is identified, and an image having a background texture consistent with (or similar to) the background texture of the data classification is selected from the inspection target image group. The selected image is displayed in the defect data input screen, and if the defect data input by the user contains a defect falling into the insufficient data classification, the defect data is stored as training data belonging to the data classification. In this exemplary embodiment, for the defect data input operation, the inspection target image that may be the training data for the data classification with respect to the reference amount is preferentially displayed to prompt the user to input the defect data. Below, the second exemplary embodiment will be described with the difference from the first exemplary embodiment as the center.

[0065] The hardware configuration of the information processing apparatus 300 according to the second exemplary embodiment is Figure 3A The hardware configuration of the information processing apparatus 300 according to the first exemplary embodiment shown in is similar. Therefore, a redundant description thereof will be omitted. Fig. 9A is a diagram illustrating an example of a software configuration of the information processing apparatus 300 according to the second exemplary embodiment. Figure 3B The software configuration according to the first exemplary embodiment shown in is different in that an analyzing unit 901 and a selecting unit 902 are additionally provided. The analyzing unit 901 is a functional unit of the CPU 301, and analyzes the inspection target images on which the defect data input operation is to be performed, stored in the storage unit 312, to acquire the characteristics of each inspection target image. The selecting unit 902 is a functional unit of the CPU 301, and selects an image that is identified to prompt the user to input defect data from the inspection target image group stored in the storage unit 312.

[0066] Fig.10 2 is a flowchart illustrating an example of main processing performed by the information processing apparatus 300 according to the second exemplary embodiment. Fig.10In the flowchart shown in FIG. 1 , the Figure 5 In the operation indicated by the same operation number as the operation number in the flowchart shown in , the processing equivalent to the processing according to the first exemplary embodiment is performed. In the second exemplary embodiment, after the reference amount of training data is set in S501, the processing proceeds to S1001. In S1001, the analysis unit 901 reads the inspection target image stored in the storage unit 312, analyzes the inspection target image, and acquires information about the characteristics of each image. Similar to the first exemplary embodiment, the acquired characteristics of each image according to the present exemplary embodiment include the color and texture of the concrete wall surface. The method for acquiring the characteristics is similar to the method (S504) described in the first exemplary embodiment, and its detailed description will be omitted. In the second exemplary embodiment, if the determination unit 316 determines that no instruction for executing training is issued ("No" in S508) and determines that no instruction for ending the application is issued ("No" in S510), then the processing proceeds to S1002. In S1002, the selection unit 902 selects an image for which input of defect data is to be prompted from the remaining images in the inspection target image group based on the characteristics of the inspection target image acquired in S1001 and the storage status of the training data acquired in S504. Then, the process returns to S502. In S502, the display control unit 315 controls the display unit 305 to display the selected image.

[0067] The following will refer to Fig.11A , Fig. 11B , Fig. 11C and Fig.11D The processing in S1002 for selecting an image for which input of defect data is to be prompted is described in detail. Fig.11A An example of a verification target image group 1101 and corresponding defect data 1102 according to the present exemplary embodiment is schematically illustrated. The defect data 1102 corresponds to an image group 1103 which is a part of the verification target image group 1101. Fig. 11B An example of a storage condition of training data is illustrated, and is similar to the storage condition of training data according to the first exemplary embodiment. Figure 7B The example shown in corresponds to Fig. 11B In the data classification 1111, the training data is insufficient. In order to make up for the deficiency, Fig.11A An inspection target image corresponding to the data classification 1111 is selected from the image group 1104 shown in FIG. 1 to support a user's defect data input operation to be preferentially performed on the selected image.

[0068] In the present exemplary embodiment, in order to select an image for which input of defect data is to be prompted from the image group 1104 for which defect data has not yet been input, the image is used. Fig.10The information on the characteristics of the inspection target image acquired in S1001 shown in . More specifically, an image having the same characteristics as the data classification 1111 (gray color and smooth texture of the concrete wall surface) is selected from the image group 1104 . Fig. 11C The images 1105, 1106, and 1107 shown in 1104 represent images having the same image characteristics as the data classification 1111 and selected from the image group 1104. In S1002, the CPU 301 further selects one of the images 1105, 1106, and 1107 and displays the image on the display unit 305, thereby preferentially performing input of defect data. Fig.11D Illustrated Fig. 11C 1105 and a progress display area 1122 as a display item indicating the storage status of training data are displayed together in a window 1121. The user performs a defect data input operation while checking the image 1105. Therefore, if the defect pattern of the obtained defect data is a crack that is the same as the defect pattern of the data classification 1111, the pair of the defect data and the image 1105 is stored as training data belonging to the data classification 1111. In the second exemplary embodiment, the information obtained in S1001 is only background information for checking the target image. Therefore, if the result of inputting the type of defect data is not a crack, the training data belonging to the data classification 1111 is not increased.

[0069] After the defect data input operation is performed on the image 1105, the display changes to other images 1106 and 1107, and the input of the defect data is preferentially received until the storage amount of the training data of the data classification 1111 reaches the reference amount. In order to change the image on the window, any method is applicable. For example, click Fig.11D The "next image" button 1123 shown in is a possible method. In the second exemplary embodiment, the "execute training" button is also displayed when the storage amount of the training data of the data classification 1111 reaches the reference amount. In the second exemplary embodiment, as described above, the CPU 301 selects an image that is recommended to be preferentially stored as training data, and presents the image to the user. Therefore, the user does not have to search for an image for which input of defect data is to be performed from among many images, and training data can be efficiently generated.

[0070] In the second exemplary embodiment, even if some modifications are added as described below, a similar effect can be obtained. The method for displaying an image for which the user is prompted to input defect data for a specific data classification to preferentially generate training data for data classification for insufficient storage is not limited to the above-mentioned method in which images are displayed one by one. For example, in a displayed image, a local area for which a priority prompt for input of defect data can be prominently displayed. In the inspection of infrastructure, the size of an inspection target image is very large (e.g., 10000*10000 pixels or more). Therefore, it takes time and effort to input defect data while checking the image corner by corner. In this case, CPU 301 displays a local area in an image that is likely to be used as training data for data classification for insufficient reference amount for the amount of training data, as a part for which a priority prompt for input of defect data is to be made. This enables users to store training data efficiently.

[0071] The following will refer to Fig. 12A , Fig. 12B and Fig. 12C A process for prominently displaying a local area in an image for which input of defect data is to be prompted first is described. Fig.10 After S1002 in the flowchart shown in , the display process described below is inserted. The display control unit 315 acquires the inspection target image from the storage unit 312 and divides the image into predetermined division sizes. The division size can be any size that is easy to view, such as 800x600 pixels. Fig. 12A An inspection target image 1200 and a grid 1201 dividing the image are illustrated. Fig. 12A The local area 1202 shown in is an example of a divided area. Then, CPU 301 obtains information about the image characteristics of each divided area (i.e., local area). The image characteristics to be obtained include the color and texture of the concrete wall surface. The method for obtaining these image characteristics is similar to the method according to the first exemplary embodiment, and its redundant description will be omitted. Based on the image characteristics obtained for each local area and the storage status of the training data, CPU 301 displays the local area for which the input of defect data is to be prompted in the application window.

[0072] Fig. 12BThe storage status 1204 for each data classification and the area 1205 as a part of the inspection target image displayed in the application window 1203 are illustrated. The frame 1206 indicated by the dotted line represents the local area corresponding to the data classification 1207 for which the training data is insufficient. Prominently displaying the local area in the image in this way makes it easier for the user to identify the area to be checked to store insufficient training data. A user interface (e.g., a scroll bar display item) for facilitating searching in the surrounding area can also be displayed. Fig. 12B In the Frame, the area corresponding to a single data classification is displayed. However, the image display range on the window can be expanded to display areas corresponding to multiple data classifications in the frame at the same time. For each data classification, the color, decoration, and thickness of the frame can be changed.

[0073] The prominent display may be automatically performed to suit the storage conditions of the data, or may be performed in response to a user instruction. If the prominent display is to be performed in response to a user instruction, then the prominent display may be performed in response to the user selecting any item of the data classification. More specifically, the user clicks Fig. 12B 1205 to turn on or off the prominent display in the area 1205. As another method for performing the prominent display, the target partial area or the target image may be popped up. Fig. 12C 1200 is a diagram illustrating a state in which an area corresponding to a data classification for which training data is insufficient is extracted from an inspection target image and popped up. When the defect data input operation is completed, the user clicks a "display area change" button 1209 to change the display. The user does not have to move the display position of the inspection target image in the window, and can perform the defect data input operation with little trouble.

[0074] The method for selecting an image for which an input of defect data is to be preferentially prompted from the inspection target image group is not limited to narrowing the range by background information (background color and texture) as described above. For example, another possible method may be to select images or regions in descending order of the possibility of the presence of defects by using a simple defect detector prepared in advance. The defect detector in this method is a trained detector, and its output indicates a score of the possibility of the presence of defects in the image with a real number from 0 to 1. The CPU 301 performs recognition processing by using the detector to obtain a score indicating the possibility of the presence of defects for each image or each region. Based on the acquired score, the CPU 301 displays the image or region on the display unit 305 in descending order of the score to prompt the input of defect data. Selecting an image or region in this way makes it easier to efficiently store training data containing many defects. By adopting any of the above modifications, in the second exemplary embodiment, similar to the first exemplary embodiment, when the storage amount has reached the reference amount, a screen for prompting the user to issue an instruction for performing training is displayed. As described above, in the second exemplary embodiment, necessary training data can be stored more efficiently.

[0075] The first exemplary embodiment and the second exemplary embodiment have been described above using examples in which, in the middle of a defect data input operation, a pair of defect data input to a portion of an inspection target image and a corresponding image is stored as training data, and a storage status is displayed. On the other hand, the third exemplary embodiment will be described below using an example in which a classifier is retrained by using the result of a defect data correction operation performed by a user as training data. Defect data is data acquired as a result of defect detection processing using a trained classifier (trained model).

[0076] For example, if there is a trained classifier that has been trained based on the results of the inspection of the infrastructure performed in the past, then a defect detection process based on the classifier can be performed once to improve the efficiency of the defect data input operation performed on the newly acquired inspection target image group. Therefore, the third exemplary embodiment has the advantage that the operation efficiency can be greatly improved by the user performing the operation for correcting the acquired detection results, compared with the case where the defect data is input from the beginning through the tracking operation. However, in the case where the defect detection performance is reduced due to various factors (such as a large difference in characteristics between the previously inspected structure and the newly inspected structure), the amount of operation for correcting the detection results increases. In this case, the user operation amount is not reduced. Therefore, the classifier is retrained using the detection results that have been corrected so far as retraining data. Thus, the defect detection performance is improved. Then, the defect detection process is performed on the remaining images that have not yet undergone the correction operation. Therefore, the defect detection results of the remaining images become data that are improved relative to the data before retraining, so that the burden of the user's correction operation can be reduced.

[0077] However, in the case where the defect detection performance is sufficiently high, it is unlikely that an effect in reducing the correction operation can be obtained by performing retraining. In other words, retraining a classifier with low performance provides a higher retraining effect than retraining a classifier with sufficiently high performance. For this reason, the third exemplary embodiment will be described below using the following example in which the detection performance is calculated while the user is performing a correction operation on the defect detection result, and the calculated detection performance is used as a factor for determining the timing of prompting the user to perform retraining. More specifically, in the case where the detection performance is lower than a predetermined standard, that is, in the case where retraining is likely to reduce the burden of the user's correction operation, display control is performed to prompt the user to perform retraining.

[0078] The present exemplary embodiment will also be described using the following example, in which it is assumed that a wall surface image of a concrete structure is inspected, the inspection target is a concrete wall surface, and the detection target is a crack. In the present exemplary embodiment, an F-measure value (F-measure) as a performance index that takes into account both an erroneous detection result and a non-detection result is used as an example of an index for comparing detection performance. The F-measure value takes a value from 0 to 1, and the higher the performance, the closer the F-measure value is to 1. However, the index to be used in the present exemplary embodiment is not limited to this.

[0079] The hardware configuration of the information processing apparatus 300 according to the third exemplary embodiment is Figure 3A The hardware configuration of the information processing apparatus 300 according to the first exemplary embodiment shown in FIG. 1 is similar, and a redundant description thereof will be omitted. Fig. 9Bis a diagram illustrating an example of a software configuration of the information processing apparatus 300 according to the third exemplary embodiment. Figure 3B The software configuration according to the first exemplary embodiment shown in is different in that an evaluation unit 911 is additionally provided. In the third exemplary embodiment, the receiving unit 311 receives an operation for correcting defect data information (position and type). The evaluation unit 911 is a functional unit of the CPU 301, and evaluates the performance of the detection processing performed by the detection unit 318 based on the result of the correction operation. In the present exemplary embodiment, the evaluation unit 911 evaluates the performance by calculating the F-measurement value. In the present exemplary embodiment, the determination unit 316 performs determination processing based on the storage amount of training data and the calculated F-measurement value. Based on the determination result, the display control unit 315 controls the display to prompt the user to issue an instruction for performing retraining.

[0080] FIG. 13A to FIG. 13C 1301 is a diagram illustrating an example of defect data indicating a defect detection result according to the third exemplary embodiment (i.e., defect data before and after the user performs a correction operation). Image 1301 is a captured inspection target image of a concrete wall surface including a plurality of cracks. However, elements other than the cracks are omitted to schematically illustrate the state of the cracks. In an actual inspection target image, elements such as a line of a concrete joint, stain, and unevenness appear together with the cracks. Detection result 1302 represents defect data as a result of applying a defect detection method to image 1301. Detection result 1303 represents defect data as a result of a correction operation performed by a user based on detection result 1302. For defect data 1311 in detection result 1302, there is no crack in original image 1301. This means that defect data 1311 is erroneously detected. For example, if a classifier incorrectly detects an element other than a crack as a crack, an erroneous detection occurs. The user deletes defect data 1311 through a correction operation. Therefore, the detection result 1303 does not include the defect data 1311. The defect data 1312 in the detection result 1303 is new data that is input by the user and is not detected in the detection result 1302. In this way, the detection result 1302 includes an erroneous crack detection result and a non-detection result as a result of incorrect crack detection. The amount of correct answers and incorrect answers in the detection result can be calculated based on the detection results 1302 and 1303 before and after the correction operation. The F-measure used in the present exemplary embodiment can be obtained from the calculated amount.

[0081] Fig.14 is a diagram illustrating an example of display data according to the present exemplary embodiment. More specifically, Fig.14 is an example of display data indicating the storage status of retraining data when the user is performing a correction operation on a defect detection result. Fig.14 Compared with the first exemplary embodiment Figure 7B The application window 711 shown corresponds. In the third exemplary embodiment, the display data includes a display item of an F-measurement value 1401 indicating the detection performance as a display factor. In this exemplary embodiment, in order to determine whether to prompt retraining, it is determined whether the F-measurement value is less than a predetermined value. For example, the determination method can be represented by the following inequality (2).

[0082] γ p >f p (p=1, 2, ..., P) (2)

[0083] The parameter fp represents an F-measure indicating the detection performance for the p-th data classification. The parameter γ represents a performance standard for each data classification and is set to an arbitrary constant from 0 to 1. As a possible method for determining the parameter γ, for example, an experimentally based determination method can be used.

[0084] Fig.15 is a flowchart illustrating an example of main processing performed by the information processing apparatus 300 according to the present exemplary embodiment. Figure 5 The operations of the same processing in the flowchart shown are indicated by the same operation numbers, and redundant description thereof will be omitted. In the third exemplary embodiment, when the application for performing defect detection processing from the inspection target image displays the result of defect detection performed by the trained classifier, the work area of ​​the CPU 301 is initialized, and the CPU 301 starts to execute the defect detection processing. Fig.15 The processing in S1501 shown in . In S1501, the setting unit 314 sets the reference amount of new training data required for retraining. In the present exemplary embodiment, the setting unit 314 reads a predetermined value. In S1502, the display control unit 315 controls the display unit 305 to display the inspection target image and the defect data detected from the image in an overlapping manner. In S1503, the receiving unit 311 receives the correction operation and stores the new training data (a pair of the inspection target image and the defect data) generated in the correction operation in the storage unit 312. In S1504, the acquisition unit 313 displays the storage status of the new training data stored in the storage unit 312. In S1505, the display control unit 315 controls the display unit 305 to display the storage status of the training data and the reference amount set in S1501 in a comparable manner. Depending on the definition of the set reference amount, the training data processed in S1505 and S1506 can be combined with the training data that has been stored before the correction operation.

[0085] In S1506, the evaluation unit 911 calculates the F-measure value for each data classification. As a method of calculating the F-measure value, a known method can be used. However, since an accurate F-measure value is not calculated until the correction data is stored to a certain extent, when the amount of data is insufficient, the process may proceed to S1507 without calculating the F-measure value. In the present exemplary embodiment, for example, the display control unit 315 generates the following Fig.14 The display control unit 315 displays the calculated F-measure value on the display unit 305. In S1507, the determination unit 316 determines whether the storage amount of the acquired training data has reached the set reference amount. In the present exemplary embodiment, unless otherwise specified, the determination unit 316 comprehensively determines whether the storage amount of the training data has reached the predetermined amount for all data classifications. If the determination unit 316 determines that the storage amount has reached the reference amount ("Yes" in S1507), the process proceeds to S1508. If the determination unit 316 determines that the storage amount has not yet reached the reference amount ("No" in S1507), the process proceeds to S510. In S1508, the determination unit 316 determines whether the F-measure value for any data classification has reached the predetermined value based on inequality (2). If the F-measure value for at least one data classification has not yet reached the predetermined value ("Yes" in S1508), the process proceeds to S507. If the F-measure values ​​for all data categories have reached the predetermined value (No in S1508), the process proceeds to S510. However, if the F-measure value has not been calculated in S1506, the determination unit 316 may always determine that the F-measure value has reached the predetermined value (No in S1508). The processes in S507 and subsequent operations are similar to those according to the first exemplary embodiment, and a repeated description thereof will be omitted.

[0086] As described above, in the present exemplary embodiment, if the detection performance for any data classification satisfies inequality (2) ("Yes" in S1508), then in S507, a screen for prompting retraining is displayed. As the amount of erroneous detection data and undetected data corrected by the user increases, the F-measurement value decreases. Therefore, in the present exemplary embodiment, when the F-measurement value becomes lower than a predetermined value as the user operation proceeds, the user is prompted to perform retraining. Therefore, the user is prompted to issue an instruction for performing training at an appropriate timing, thereby achieving improved efficiency of user operation. However, the method of performing determination based on inequality (2) for each classification is only an example, and another determination standard or determination method can also be used. For example, a possible determination method may be to determine whether the detection performance of the entire training data is less than a reference value, rather than whether the performance index for each data category is less than a reference value. In addition, as another determination method, if the F-measurement value of any one of the data categories having the top five storage amounts or storage rates of the training data is less than a fixed value, then a screen for prompting retraining can be displayed. Inequality (2) is used to determine whether the F-measurement value is less than a predetermined value. However, when selecting the determination method and parameters, whether the case where the comparison targets are equal is regarded as failing to achieve the predetermined performance or satisfying the predetermined performance can be arbitrarily determined.

[0087] According to the first exemplary embodiment Fig. 8A and Figure 8B The display method shown in can be used as a method for displaying a screen for prompting retraining according to the third exemplary embodiment. The above method makes it possible to determine whether to prompt retraining based on defect detection performance while the user is correcting defect detection results. This enables the user to determine the timing for performing retraining. If the detection performance is high enough from the beginning and the F-measurement value does not become less than a predetermined value during the correction operation, the user can perform the correction operation until completion without being interrupted by retraining. Although the present exemplary embodiment has been described above using a case in which the detection results of a trained classifier are corrected, the defect data subjected to correction may include not only the detection results of the classifier, but also defect data input through the user's tracking operation.

[0088] The third exemplary embodiment allows a modification to omit the determination based on the storage status (S1507). More specifically, in a case where the user's correction operation is performed to the extent that the F-measure value can be calculated and a further correction operation has been performed, if the F-measure value becomes less than a predetermined value for any data classification ("Yes" in S1508), then in S507, an "Execute Training" button is displayed. In this case, after checking the storage status of the training data displayed in S1504 and S1505, the user can decide to execute retraining when a sufficient storage amount is acquired or at an arbitrary timing.

[0089] Modifications applicable to the first exemplary embodiment to the third exemplary embodiment will be described below. Fig.16A , 16B 16C is a diagram illustrating an example of display data displayed according to the modification. In the first exemplary embodiment to the third exemplary embodiment, the reference amount of the training data is set as, for example, an absolute amount (such as the number of images). The display data includes a progress bar indicating the progress of the storage status of the training data relative to the reference amount, wherein the reference amount is represented by 100%. However, the reference amount can be set as a relative amount. For example, 40% of the inspection target images input in large quantities are set as the reference amount. Fig.16A The application window 1600 shown in FIG. 1 shows a progress display area 1601, similar to Fig. 8A and Figure 8B Window 800 shown in . Progress bar 1602 and progress rate 1604 indicate that the progress of the storage status of the training data is 20%. In this modification, the reference amount (40% in this example) is indicated by indicator 1603. Fig. 16B This example illustrates the display state when the training data has reached the reference amount due to the user's continuous defect data input operation. Fig. 16B , the progress bar 1602 has reached the indicator 1603, and the progress rate 1604 indicates 40%. In addition, a training button 1605 is displayed to enable the user to confirm that the training data has been stored in the reference amount. The reference amount of 40% is only an example. Even when the reference amount is defined as a relative amount, the reference amount can be set to be suitable for the size of the inspection target, the total amount of the inspection target image, and the content of the classifier.

[0090] like Fig.16A and Fig. 16BAs shown in , even in the case where the reference amount is set as an absolute amount, a method of displaying the reference amount as an indicator in the middle of the progress bar instead of at the end can be used. This display method makes it easier for the user to recognize that not only the training data can be used for training, but also the manual defect data input operation can be continued. This method is advantageous in the case where generating training data exceeding the reference amount enables further improvement of the performance of the classifier and in the case where additional training is a paid option. In this case, the user can easily determine whether to continue the operation depending on the burden of the user operation.

[0091] like Fig. 16C As shown in , a “detection” button 1606 for issuing an instruction for performing a detection process using a trained classifier after training may be displayed together with a “training” button 1605. In this case, in response to the user's operation of the “detection” button 1606, the detection unit 318 performs the detection process, and the display unit 305 changes the display to a screen for displaying the detection result. In each of the above-described exemplary embodiments, the purpose of training a classifier on training data is to detect a detection target. Therefore, by providing a GUI for issuing an instruction for performing a detection process when a sufficient amount of training data has been acquired, the overall operational efficiency can be improved. In order to enable instructions for performing training and detection to be issued immediately, a display item such as a button icon integrating the functions of the “training” button 1605 and the “detection” button 1606 may be displayed. Needless to say, if the “detection” button 1606 is displayed together with a progress display area, and the reference amount set in the progress display area corresponds to 100% (such as Fig. 8A and Figure 8B 800 shown in FIG), a similar effect can be obtained.

[0092] Other embodiments

[0093] Some embodiments may also be implemented by reading out and executing computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more completely referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more of the above-mentioned (one or more) embodiments and / or a system or device including one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing the functions of one or more of the above-mentioned (one or more) embodiments, and by a method performed by a computer of the system or device, for example, by reading out and executing computer executable instructions from a storage medium to perform the functions of one or more of the above-mentioned (one or more) embodiments and / or controlling one or more circuits to perform the functions of one or more of the above-mentioned (one or more) embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read out and execute computer executable instructions. Computer executable instructions may be provided to the computer, for example, from a network or a storage medium. The storage medium may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a storage device of a distributed computing system, an optical disk (such as a compact disk (CD), a digital versatile disk (DVD), or a Blu-ray disk (BD)) TM ), one or more of flash memory devices, memory cards, etc.

[0094] The embodiments of the present disclosure may also be implemented by the following method, that is, software (program) that performs the functions of the above embodiments is provided to a system or device through a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0095] Although the present disclosure has described exemplary embodiments, it is to be understood that some embodiments are not limited to the disclosed exemplary embodiments.The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An information processing device, comprising: a receiving unit configured to receive an input specifying a position of a detection target included in the image; an acquisition unit configured to acquire a storage amount of training data, the training data including pairs of information indicating an image and information indicating a position specified by an input; A training unit is configured to train a training model for detecting a detection target from an image based on the stored training data; as well as a display control unit configured to control the display unit to display the storage amount and a reference amount of the training data in a comparable manner, the reference amount being set to an amount of the training data necessary for the training unit to train the training model, The training data is classified based on the characteristics of the image, and each characteristic of the image is used to obtain a different local area by dividing the image into predetermined sizes. The acquisition unit acquires the storage amount of training data for each classification. wherein the display control unit performs control so as to display the reference amount and the storage amount of the training data for each classification in a comparable manner, and Wherein, when the storage amount of training data of a classification does not reach a reference amount, the display control unit displays a local area corresponding to the classification whose training data does not reach the reference amount among the multiple local areas. 2 . The information processing apparatus according to claim 1 , wherein the display control unit performs control to display, on the display unit, a state in which the storage amount changes in accordance with reception of the input by the reception unit. 3 . The information processing apparatus according to claim 1 , wherein the image refers to a captured image of a structure to be inspected as a subject, and the detection target refers to a crack occurring on a surface of the structure.

4. The information processing device according to claim 1, wherein: When the storage amount reaches a reference amount, the training unit performs training.

5. The information processing device according to claim 1, in, In a case where the storage amount reaches a reference amount, the display control unit controls the display unit to display an item for receiving an instruction for training the training model by the training unit, and The training unit performs training in response to user operations on the displayed items.

6. The information processing device according to claim 1, further comprising a detection unit configured to detect defects from the image based on the training model, Here, the image refers to a captured image of a structure to be inspected as a subject, and the inspection target refers to a defect occurring on the surface of the structure. 7 . The information processing apparatus according to claim 6 , wherein the defect includes a crack occurring on a surface of the structure. 8 . The information processing device according to claim 1 , further comprising a setting unit configured to set the reference amount based on a user input or a past training result.

9. The information processing device according to claim 1, wherein: While the receiving unit is receiving input, the display control unit displays an image and data indicating the position of the detection target to be input on the display unit, and controls the display unit to display a state in which the storage amount changes as the receiving unit receives the input. 10 . The information processing apparatus according to claim 1 , wherein the receiving unit receives an operation for inputting the position information by tracking the detection target on the image displayed on the display unit.

11. The information processing device according to claim 1, further comprising a detection unit configured to detect a detection target from an image based on a trained model that has been trained by the training unit, The receiving unit receives an operation for correcting position information, the position information indicating a detection target detected from the image by the detection unit. 12 . The information processing device according to claim 11 , wherein the training unit performs retraining on the trained model based on a result of the operation for correcting the position information received by the receiving unit.

13. The information processing device according to claim 12, further comprising an evaluation unit configured to evaluate a detection result performed by the detection unit; in, The training unit performs retraining in a case where the evaluation unit evaluates that the performance of the detection process performed by the detection unit is lower than a predetermined reference value.

14. The information processing device according to claim 13, in, in a case where the evaluation unit evaluates that the performance of the detection process performed by the detection unit is lower than a predetermined reference value, the display control unit controls the display unit to further display an item for receiving an instruction for performing retraining performed by the training unit, and The training unit performs retraining in response to a user operation on the displayed item.

15. The information processing device according to claim 6, in, In a case where the storage amount reaches a reference amount, the display control unit controls the display unit to further display an item for receiving an instruction for executing a process of detecting a defect from an image based on a training model that has been trained with the stored training data, The detection unit performs detection processing in response to a user operation on the displayed item.

16. A method for controlling an information processing device, the method comprising: receiving, by a receiving unit, an input specifying a location of a detection target included in an image; acquiring, by an acquisition unit, a storage amount of training data, the training data including pairs of information indicating an image and information indicating a position specified by an input; The training unit trains a training model for detecting a detection target from an image based on the stored training data; as well as The display control unit controls a predetermined display unit to display the storage amount and a reference amount of the training data in a comparable manner, the reference amount being set as the amount of the training data necessary for the training unit to train the training model, The training data is classified based on the characteristics of the image, and each characteristic of the image is used to obtain a different local area by dividing the image into predetermined sizes. The acquisition unit acquires the storage amount of training data for each classification. wherein the display control unit performs control so as to display the reference amount and the storage amount of the training data for each classification in a comparable manner, and Wherein, when the storage amount of training data of a classification does not reach a reference amount, the display control unit displays a local area corresponding to the classification whose training data does not reach the reference amount among the multiple local areas.

17. A non-transitory computer-readable recording medium storing a program for causing a computer to: a receiving unit configured to receive an input specifying a position of a detection target included in the image; an acquisition unit configured to acquire a storage amount of training data including pairs of information indicating an image and information indicating a position specified by an input; A training unit is configured to train a training model for detecting a detection target from an image based on the stored training data; as well as a display control unit configured to control the display unit to display the storage amount and a reference amount of the training data in a comparable manner, the reference amount being set to an amount of the training data necessary for the training unit to train the training model, The training data is classified based on the characteristics of the image, and each characteristic of the image is used to obtain a different local area by dividing the image into predetermined sizes. The acquisition unit acquires the storage amount of training data for each classification. wherein the display control unit performs control so as to display the reference amount and the storage amount of the training data for each classification in a comparable manner, and Wherein, when the storage amount of training data of a classification does not reach a reference amount, the display control unit displays a local area corresponding to the classification whose training data does not reach the reference amount among the multiple local areas.

18. An information processing system comprising: a receiving unit configured to receive an input specifying a position of a detection target included in the image; an acquisition unit configured to acquire a storage amount of training data, the training data comprising pairs of information about the image and information about the input position; A training unit is configured to train a training model for detecting a detection target from an image based on the stored training data; a display control unit configured to control the display unit to display the storage amount and a reference amount of the training data in a comparable manner, the reference amount being set to an amount of the training data necessary for the training unit to train the training model; as well as a detection unit configured to detect a detection target from an image based on a trained model that has been trained by a training unit after the storage amount reaches a reference amount, The training data is classified based on the characteristics of the image, and each characteristic of the image is used to obtain a different local area by dividing the image into predetermined sizes. The acquisition unit acquires the storage amount of training data for each classification. wherein the display control unit performs control so as to display the reference amount and the storage amount of the training data for each classification in a comparable manner, and Wherein, when the storage amount of training data of a classification does not reach a reference amount, the display control unit displays a local area corresponding to the classification whose training data does not reach the reference amount among the multiple local areas.

Images