Information processing apparatus, information processing method, and program

Abstract

To provide an information processing apparatus, an information processing method, and a program that can create training data for predicting an appearance of a deformation even with an unknown captured image.SOLUTION: An information processing apparatus comprises: acquisition means (processes of S501, S502) which acquires first deformation data indicating a deformation included in a first inspection image obtained by imaging a first region, and second deformation data indicating a deformation included in a second inspection image obtained by imaging a second region including at least a part of the first region at a timing newer than a timing at which the first inspection image is captured; calculation means (process of S503) that calculates difference deformation data indicating a difference between the first deformation data and the second deformation data; and creation means (process of S504) that creates training data for training a model that predicts an appearance of the deformation based on the difference deformation data.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In recent years, from an image of a structure made of concrete or the like, deformations such as cracks are extracted by manual tracing or an image recognition technique such as Non-Patent Document 1, and the soundness is determined based on their positions, scales, and quantities. Visual inspections using images are widely carried out.

[0003] Furthermore, according to Patent Document 1, a method of creating a strain distribution from an image of a known initial test piece and an image of a test piece in which strain has occurred and predicting the occurrence of strain is disclosed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, the method of Patent Document 1 makes a determination based on the strain distribution obtained for a known test body, and does not take an image as input. It is difficult to generate learning data for training a learning model that predicts the occurrence of damage based on the captured image.

[0007] The present invention has been made in view of the above problems, and an object thereof is to provide an information processing apparatus, an information processing method, and a program capable of creating learning data for training a learning model for predicting the occurrence of deformation even for a captured image that is not known.

Means for Solving the Problems

[0008] To solve this problem, for example, the information processing apparatus of the present invention has the following configuration. That is, Acquisition means for acquiring first deformation data indicating a deformation included in a first inspection image obtained by photographing a first region, and second deformation data indicating a deformation included in a second inspection image obtained by photographing at least a part of the first region at a time later than the time when the first inspection image was photographed; Calculation means for calculating difference deformation data indicating the difference between the first deformation data and the second deformation data; Creation means for creating learning data for training a model for predicting the occurrence of the deformation based on the difference deformation data; and comprising.

Effects of the Invention

[0009] According to the present invention, it is possible to create learning data capable of predicting the occurrence of deformation even for a captured image that is not known.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0012] <First Embodiment> In this embodiment, a computer operates as an information processing apparatus. When a captured image is input, the information processing apparatus of this embodiment creates learning data for predicting whether a change will occur in the future. The information processing apparatus of this embodiment creates learning data for predicting the occurrence of a change using the difference between change data at two different times and an image with an older capture time. In this embodiment, an example of creating information on whether a change will occur in an image as learning data will be described.

[0013] Note that a change refers to cracks or the like that occur on the concrete surface due to damage, deterioration, or other factors of concrete structures such as motorways, bridges, tunnels, dams, etc. A crack is a linear damage having a starting point, an ending point, a length, and a width that occurs on the wall surface or the like of a structure due to aging deterioration, the impact of an earthquake, or the like.

[0014] <Hardware Configuration> First, with reference to FIG. 1, the hardware configuration of the information processing apparatus of this embodiment will be described. FIG. 1 is a block diagram showing the hardware configuration of the information processing apparatus 100 of this embodiment.

[0015] In this embodiment, a computer operates as the information processing apparatus 100. Note that the processing of the information processing apparatus of this embodiment may be realized by a single computer, or each function may be distributed and realized by a plurality of computers as necessary. The plurality of computers are connected to be communicable with each other.

[0016] The information processing apparatus 100 includes a control unit 101, a non-volatile memory 102, a work memory 103, a storage device 104, an input device 105, an output device 106, a network interface 107, and a system bus 108.

[0017] The control unit 101 comprehensively controls the entire information processing apparatus 100. The control unit 101 includes at least one of arithmetic processing processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), and a QPU (Quantum Processing Unit).

[0018] The non-volatile memory 102 is, for example, a ROM (Read Only Memory). The non-volatile memory 102 stores data such as programs and parameters executed by the processor of the control unit 101. Here, the program includes a program for executing the learning data creation process described later.

[0019] The work memory 103 is, for example, a RAM (Random Access Memory). The work memory 103 temporarily stores data such as programs and parameters supplied from an external device or the like by the control unit 101 or the like.

[0020] The storage device 104 is a non-volatile storage device built in the information processing apparatus 100 or a non-volatile storage device detachably connected to the information processing apparatus 100. The storage device 104 is, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), a memory card, etc. composed of a semiconductor memory or a magnetic disk, etc. Further, the storage device 104 includes a storage medium composed of a disk drive that reads / writes data to an optical disk such as a DVD or a Blu-ray Disc (registered trademark).

[0021] The input device 105 is an operation member such as a mouse, a keyboard, or a touch panel that receives a user operation. The input device 105 outputs an operation instruction received from the user to the control unit 101.

[0022] The output device 106 is a display device such as a liquid crystal display and a display and monitor including an organic EL (Electro Luminescence). The output device 106 displays images such as data held by the information processing device 100 and data supplied from external devices.

[0023] The network interface 107 is communicably connected to any network such as the Internet and a LAN (Local Area Network).

[0024] The system bus 108 enables data transfer between the control unit 101, non-volatile memory 102, work memory 103, storage device 104, input device 105, output device 106, and network interface 107 that constitute the information processing device 100. The system bus 108 includes an address bus, a data bus, and a control bus.

[0025] The non-volatile memory 102 or the storage device 104 stores the OS (Operating System), which is basic software executed by the control unit 101, and applications that realize application functions in cooperation with this OS. Also, in the present embodiment, the non-volatile memory 102 or the storage device 104 stores an application for the information processing device 100 to realize the learning data creation process described later.

[0026] The processing of the information processing device 100 of the present embodiment is realized by loading software provided by an application. The application has software for utilizing the basic functions of the OS installed in the information processing device 100. The OS of the information processing device 100 may have software for realizing the processing in the present embodiment.

[0027] Hereinafter, in the first embodiment, an embodiment of creating two-class information on whether or not a change appears in an image as learning data will be described.

[0028] <Change data> In this embodiment, deformation is represented by vector data, and an example in the case where the deformation is a crack will be described. The deformation data representing the deformation by vector data consists of information obtained from an inspection image obtained by photographing an inspection target, and is represented in a format as shown in FIG. 2 described later. Each deformation data is input by a user tracing on an image with a tablet or the like, automatically generated by image analysis processing or the like, or input by a combination thereof. Further, the image analysis processing may be executed using a learning model created by machine learning / deep learning of AI (artificial intelligence) as shown in Non-Patent Document 1.

[0029] FIG. 2 is a diagram showing an example of a deformation data table 201 in which deformation data obtained from a certain inspection image is described.

[0030] The deformation data table 201 is a table of a plurality of deformation data representing the shape of each deformation such as a crack by a continuous polyline. Each deformation data includes a deformation ID 202 for identifying each deformation data, the number of vertices 203 of the deformation such as a crack, and a vertex coordinate list 204 of the crack.

[0031] FIG. 3 is a diagram showing an example of a superimposed image 300 in which the deformation data registered in the deformation data table 201 shown in FIG. 2 is superimposed on the inspection image.

[0032] In the superimposed image 300, C001 to C012 are cracks in which the deformation ID 202 of the deformation data table 201 is C001 to C012, respectively.

[0033] <Learning data creation application> FIG. 4 is a diagram showing an example of a user screen 400 of an application for creating learning data executed in the information processing apparatus 100 of this embodiment. The user screen 400 is a so-called GUI (Graphical User Interface). The user screen 400 includes a first inspection data input button 401, a second inspection data input button 402, a prediction learning data creation button 403, a differential deformation selection switch 404, a differential minimum number input area 405, and a differential minimum length input area 406.

[0034] On the user screen 400, the first inspection data input button 401 is a button for selecting an inspection image taken at the first time. When the user operates the first inspection data input button 401 by clicking or the like, a file selection dialog (not shown) is displayed. The file selection dialog displays, for example, a list of either the file names or thumbnails of the images taken at the first time and stored in the storage device. When the user selects any of the images displayed in the file selection dialog, the control unit 101 acquires the selected image as the first inspection image. The control unit 101 stores the first inspection image or the identification information identifying the first inspection image in the work memory 103 or the like.

[0035] The second inspection data input button 402 is a button for selecting a second inspection image taken at a second time newer than the first time. The second inspection image is an image taken of an inspection target area including at least a part of the inspection target area of the first inspection image. In the present embodiment, the second inspection image is an image taken of the same inspection target area as the first inspection image. When the user operates the second inspection data input button 402, the control unit 101 causes the output device 106 to display a file selection dialog (not shown). The file selection dialog is, for example, a screen including a list of at least either the file names or thumbnails of the images taken at a plurality of second times newer than the first time and of the same inspection target area as the first inspection image and stored in the storage device 104. When the user selects an image from the file selection dialog, the control unit 101 acquires the image as the second inspection image from the storage device 104. Note that, when the imaging areas of the first inspection image and the second inspection image are misaligned, the control unit 101 may correct the misalignment. The control unit 101 stores the second inspection image or the identification information identifying the second inspection image in the work memory 103 or the like.

[0036] The prediction learning data creation button 403 is a button for inputting an instruction to create learning data. When the user operates the prediction learning data creation button 403, the control unit 101 creates musician data.

[0037] The differential deformation selection switch 404 is a button for selecting the type of differential deformation data to be used for the learning data described later. Although details will be described later, the differential deformation data is the difference in the shape of the same deformation data at two different times. For example, the user operates the differential deformation selection switch 404 to select whether to use only the differential deformation data of the extension or to use the newly appeared differential deformation data in addition to the extension among the differential deformation data.

[0038] The differential minimum number input area 405 is a numerical input area for setting the minimum number of differential deformation data included in the learning data as a threshold. When the differential deformation data of the learning data is larger than the minimum number, the control unit 101 uses the learning data.

[0039] The differential minimum length input area 406 is a numerical input area for setting the minimum pixel length of all the differential deformation data included in the learning data as a threshold. When the differential deformation data is longer than the minimum pixel length, the control unit 101 uses the learning data.

[0040] <Learning data creation process> FIG. 5 is a diagram showing an example of a flowchart of the process of an application for creating learning data from differential deformation data in the first embodiment. In this embodiment, inspection images at two different times taken at the same location are input, differential deformation data is calculated from the deformation data detected from each image, and learning data for prediction is created from the differential deformation data and the images.

[0041] The process of FIG. 5 is realized by the control unit 101 of the information processing apparatus 100 shown in FIG. 1 reading out the program stored in the non-volatile memory 102 or the storage device 104, expanding it in the work memory 103, and executing it to control each component. By executing the program, the control unit 101 functions as, for example, an acquisition means for acquiring inspection images and deformation data, a calculation means for calculating differential deformation data, and a creation means for creating learning data.

[0042] In S501, the control unit 101 reads out the first inspection image selected by the user with the first inspection data input button 401, and acquires the first deformation data corresponding to the read first inspection image. The control unit 101 may detect and acquire the first deformation data at the timing when the first inspection image is read out, or may acquire the detected first deformation data associated with the first inspection image.

[0043] In S502, the control unit 101 reads out the second inspection image selected by the user with the second inspection data input button 402, and acquires the second deformation data corresponding to the read second inspection image. The control unit 101 may detect and acquire the second deformation data at the timing when the second inspection image is read out in the same manner as in S501, or may acquire the detected second deformation data associated with the second inspection image.

[0044] In S503, the control unit 101 uses the first deformation data acquired in S501 and the second deformation data acquired in S502 to calculate the deformation data that appears only in the second deformation data as differential deformation data. The details of the method for calculating the differential deformation data will be described later with reference to FIG. 6.

[0045] In S504, the control unit 101 sets conditions for selecting the differential change data to be used for creating the learning data, which will be described later, from the differential change data calculated in S503. The control unit 101 sets the conditions based on the information input by the user at the differential change selection switch 404, the differential minimum number input area 405, and the differential minimum length 406 of the user screen 400. The control unit 101 sets the learning label of the learning data, which will be described later, based on the said conditions.

[0046] In S505, when the control unit 101 detects that the user has pressed the prediction learning data creation button 403, it uses the differential change data that satisfies the conditions set in S504 to create the learning data.

[0047] In S506, the control unit 101 allows the user to confirm the learning data created in S505 and accepts the editing of the learning data from the user. For example, the control unit 101 allows the user to confirm and edit the learning data by displaying the learning data on the output device 106. The control unit 101 outputs the confirmed and edited learning data as a file and stores it in the work memory 103 or the storage device 104.

[0048] Note that in this embodiment, an example is given in which the user is allowed to select the first inspection image and the second inspection image respectively, and the inspection images at two times are acquired separately. However, the method for selecting the inspection images is not limited to this. For example, the information processing apparatus 100 may generate a list in which the file of the first inspection image and the file of the second inspection image are linked as a pair and present it to the user. In this case, the information processing apparatus 100 may acquire two inspection images by allowing the user to select the file names of a plurality of inspection images at two times from the list at once.

[0049] <Calculation of differential change data> The calculation process of the differential change data in S503 will be described. FIG. 6 is a diagram showing an example of the differential calculation process between the change data.

[0050] FIG. 6(a) represents the first deformed data 601. FIG. 6(b) shows the inflated deformed data 611 which is the inflated range of the first deformed data 601 by the inflation width D. FIG. 6(c) shows a state where the inflated deformed data 611 is superimposed on the second deformed data 650 composed of the line segment 651 to the line segment 655. FIG. 6(d) is a diagram showing that the second deformed data 650 within the range of the inflated deformed data 611 is regarded as deformed data that coincides at two different times, and the deformed data outside the inflated range is treated as differential deformed data.

[0051] As a method for determining duplication inside and outside the range, techniques such as collision determination techniques widely known in fields such as computer games may be applied. In this case, the control unit 101 generates an intermediate point 681 and an intermediate point 682 which are switching points between the coinciding deformed data and the differential deformed data. For example, the control unit 101 creates inflated deformed data by inflating the deformed data in the width direction. Next, the control unit 101 may generate the intersection points of the inflated deformed data 611 and the second deformed data 650 as the intermediate points 681 and 682. The control unit 101 divides the line segment 652 into a line segment 661 and a line segment 662 at the intermediate point 681, and divides the line segment 654 into a line segment 663 and a line segment 664 at the intermediate point 682. The control unit 101 divides the deformed data 671 where the first deformed data and the second deformed data coincide into the line segment 662, the line segment 653, and the line segment 663. The control unit 101 generates two pieces of deformed data as differential deformed data. Specifically, the control unit 101 generates the deformed data 672 composed of the line segment 651 and the line segment 661, and the deformed data 673 composed of the line segment 664 and the line segment 655.

[0052] When the control unit 101 divides the second deformation data into deformation data that matches the first deformation data and differential deformation data that does not match the first deformation data, the control unit 101 sets the type of the differential deformation data as "extension". In other words, when a part of the second deformation data overlaps with the first deformation data and the second deformation data extends from the first deformation data, the control unit 101 sets the type of the differential deformation data as "extension". On the other hand, when the control unit 101 determines that the second deformation data that does not overlap with the first deformation data is differential deformation data (not shown), the control unit 101 sets the type of the differential deformation data as "new". In other words, when the control unit 101 determines that the second deformation data appears at a position where the first deformation data does not exist, the control unit 101 sets the type of the differential deformation data as "new". Note that the overlap here excludes the case where the second deformation data intersects with the first deformation data at substantially one point.

[0053] FIG. 7 illustrates a deformation data table in which the deformation data and the differential deformation data of the differential calculation result are represented as a table.

[0054] The deformation data table 701 in FIG. 7(a) illustrates the deformation data 601 in FIG. 6(a) with the deformation ID Cb901. The deformation data table 701 has the same configuration as the deformation data table 201 in FIG. 2, and includes a deformation ID 702, the number of vertices 703, and a vertex coordinate list 704.

[0055] The deformation data table 711 of the matching part in FIG. 7(b) illustrates the case where the deformation ID of the deformation data 671 of the matching part in FIG. 6(d) is Cbm901. The deformation data table 711 of the matching part has an item of a reference deformation ID 715 in addition to the configuration of the deformation data table 701. In the example shown in FIG. 7(b), Cb901, which is the deformation ID in FIG. 7(a), is described as the ID of the deformation data that is the source of the matching deformation data.

[0056] The differential deformation data table 721 in FIG. 7(c) exemplifies the case where the deformation IDs of the deformation data 672 and the deformation data 673 in FIG. 6(d) are Cbx901_1 and Cbx901_2, respectively. In addition to the items in the deformation data table 711, the differential deformation data table 721 has an item of type 726 added. The type 726 indicates whether the type of the differential deformation data is either "extension" or "new".

[0057] <Learning data> Regarding the learning data in the first embodiment calculated by the control unit 101 in S505, it will be described with reference to FIG. 8. FIG. 8 is a diagram showing an example of a learning data table 801.

[0058] The learning data table 801 includes a data ID 802, an image file 803, the number of differences 804, the difference length 805, the number of extensions 806, the extension length 807, and a learning label 808.

[0059] The data ID 802 indicates an ID for identifying the learning data. The image file 803 indicates the image file name of the first inspection image. The number of differences 804 indicates the number of differential deformation IDs calculated in S503 per one image file. The difference length 805 indicates the pixel length of the differential deformation data per one image. The number of extensions 806 and the extension length 807 respectively indicate the number and length of the deformations indicating extensions among the number of differences 804 and the difference length 805.

[0060] The learning label 808 indicates a label for estimating and classifying the prediction of deformation. For example, the learning label 808 is set to either "appearance" which is classified when differential deformation data appears or "non - appearance" which is classified when differential deformation data does not appear. The control unit 101 may use the data with the learning label set to "appearance" as learning data. In other words, the control unit 101 assigns and classifies the learning label in units of images (here, units of the first inspection image) and sets whether to use it as learning data. The control unit 101 sets the learning label 808 based on the conditions of the differential deformation data used in S504. For example, in the differential deformation selection switch 404, when only elongation is involved, since the number of elongation and the elongation length of the image file with data ID "119" are 0, the control unit 101 changes the learning label from "appearance" to "non - appearance". Also, when it is determined that the number of differences in the differential deformation data calculated for each differential deformation data corresponding to the first inspection image does not satisfy the minimum number input to the differential minimum number input area 405, the control unit 101 sets the learning label 808 to "non - appearance". Similarly, when the differential length of the differential deformation data is less than the length input to the differential minimum length input area 406, the control unit 101 sets the learning label 808 to "non - appearance".

[0061] <Confirmation of learning data> Figure 9 represents the learning data confirmation screen 900. The confirmation screen 900 is a screen for confirming and editing the learning data created in S505. The confirmation screen 900 includes a learning data display area 901, a learning label area 902, a target file display area 903, an apply button 904, a display change button 905, and a display change button 906.

[0062] The learning data display area 901 displays an image with the differential deformation data, which is the differential calculation result, superimposed on the first inspection image. The learning data display area 901 draws the deformation that has appeared due to elongation or new appearance, that is, the difference in deformation, with a double line.

[0063] The learning label area 902 displays the learning labels corresponding to the images displayed in the learning data display area 901. The user can change the learning labels by checking the images or the calculation results of the differential change data and selecting the buttons in the learning label area 902.

[0064] The target file display area 903 displays the file names of the images used to create the displayed learning data among the plurality of learning data.

[0065] The apply button 904 is a button for reflecting the items of the learning labels changed by the user in the learning label area 902 on the images in the learning data display area 901. After the user changes the learning labels in the learning label area 902 and operates the apply button 904, the control unit 101 changes the items of the learning labels 808 of the corresponding learning data in the learning data table 801.

[0066] The display change button 905 and the display change button 906 change the display contents of the learning data, the learning label area 902, and the target file display area 903 displayed in the learning data display area 901. For example, when the user operates the display change button 905 and the display change button 906, the control unit 101 changes the learning data displayed in the learning data display area 901 in the order of the image files in the learning data table 801. In addition, the control unit 101 changes the display contents of the learning label area 902 and the target file display area 903 to the contents associated with the changed learning data along with the change.

[0067] According to the present embodiment, learning data for training a learning model capable of predicting the occurrence of changes can be created based on the differential change data, which is a state change from the change data of a first inspection image and a second inspection image that are not known and are taken at two different times with the same area as the inspection target.

[0068] In this embodiment, learning data with learning labels created from differential deformation data is used to perform two-class classification learning and create a learning model. An existing method may be used for the two-class classification learning method. Thereby, in this embodiment, it becomes possible to predict the appearance of deformation by using the captured image as input and the created learning model.

[0069] In this embodiment, in the differential deformation selection switch 404, it is possible to select whether to use only the extended deformation as the type of differential deformation data for the learning target or to use both the extended and new deformations. This is because in the case of new differential deformation data that appears in a location where there is no deformation data in the existing, i.e., the first inspection image, there may be less texture information in the first inspection image compared to the extended differential deformation data. Therefore, this embodiment has installed a differential deformation selection switch 404 to enable selection of whether to use only the extension or also the new one, thus realizing the creation of learning data that enables highly accurate learning.

[0070] In this embodiment, in the differential minimum number input area 405 and the differential minimum length input area 406, the user can specify at least either the number or the length of the differential deformation data. Thereby, in this embodiment, when the number of differential deformation data is small or the length of the differential deformation data is short, the learning data can be excluded from learning as noise.

[0071] In this embodiment, by displaying a confirmation screen 900 that can also be edited, not only can the conditions be set collectively for the differential deformation selection switch 404, the differential minimum number input area 405, and the differential minimum length input area 406, but the learning label of the learning data ID can also be changed individually. Thereby, in this embodiment, learning data that meets the user's requirements can be created.

[0072] In addition, in this embodiment, when a displacement occurs between the imaging region of the first inspection image and the imaging region of the second inspection image, by correcting the displacement, it is possible to calculate more accurate differential deformation data. As the processing method for displacement correction, image matching between images, similarity between crack deformation lines, etc. may be applied, and displacement detection and correction of existing matching methods may be applied.

[0073] In addition, in this embodiment, differential calculation is performed on the deformation data of linear cracks to create learning data, but the method for creating learning data is not limited to this. For example, in this embodiment, differential calculation may be performed on deformation data of area-shaped regions such as water leakage and rust juice to create learning data. The method for calculating the difference of area-type deformation data compares the area of the second deformation data with the area of the first deformation data, and sets a learning label based on whether there is deformation data having an area difference. Also, in the differential deformation selection switch 404, when the linear crack-type deformation data is extended, this embodiment may be treated as if the area-type deformation data is expanded (enlarged).

[0074] In addition, the learning data may add either time information such as the number of years or days between the first inspection image and the second inspection image, or information on the type of structure such as a tunnel or a bridge to the learning. For example, when learning by adding time information to the learning data, the learning model can also estimate the expected appearance time. Also, by using the type of structure and learning for each type, the learning model can improve the prediction accuracy of appearance.

[0075] <Second Embodiment> In the first embodiment, an example of inputting the first inspection image and creating learning data for predicting the occurrence of deformation over the entire image was described. In the second embodiment, an example will be described in which the control unit 101 divides the inspection image into a plurality of regions, assigns learning labels in units of the divided regions (patches), and classifies and creates learning data.

[0076] FIG. 10 shows an example of dividing an image in which differential deformation data is superimposed on a first inspection image in the second embodiment into patches. As shown in FIG. 10, the control unit 101 divides, for example, an image (learning data) in which differential deformation data is superimposed on the first inspection image into predetermined pixels. In the present embodiment, the control unit 101 divides the image into 48 patches by dividing the image into 6 parts in the horizontal direction and 8 parts in the vertical direction every 500 pixels. The image includes patches 1001 to 1048.

[0077] FIG. 11 is a learning data table 1101 in the second embodiment. The learning data table 1101 includes an ID 1102 for identifying an image, an image file 1103, a patch ID 1104, a patch address 1105, a learning label 1106, and a type 1107.

[0078] The image file 1103, the learning label 1106, and the type 1107 indicating the first inspection image are the same as those in the learning data table 801 of the first embodiment. However, the learning label 1106 and the type 1107 are associated with each patch instead of the learning data. The control unit 101 sets "appearance" in the learning label 1106 of the patch including the double line in the differential deformation data of FIG. 10.

[0079] The patch ID 1104 indicates an ID for identifying the divided patch. For example, the patch ID of patch 1001 in FIG. 10 is "001". The patch ID of patch 1048 in FIG. 10 is "048". The learning data may be identified, for example, by a combination of the ID 1102 and the patch ID 1104. In other words, the identification information of the learning data may be "ID1102" + "patch ID1104".

[0080] The patch address 1105 indicates the values of the horizontal and vertical addresses from the upper left of the original divided image. For example, the patch address 1105 of patch 1001 in FIG. 10 is (1, 1). The patch address 1105 of patch 1048 in FIG. 10 is (6, 8).

[0081] Figure 12 shows the confirmation screen 1200 in the second embodiment. The control unit 101 displays, in the learning data display area 1201, hatching indicating the learning label associated with the learning data in the learning data table 1101 on the patch corresponding to the patch address. The learning label 1202 for each patch includes "out of scope" in addition to "appearance" and "non-appearance". "Out of scope" refers to a patch that is set as out of the scope of the learning data even if there is differential change data.

[0082] As described above, in the second embodiment, the control unit 101 divides the target image of the learning data into a plurality of patches, assigns a learning label to each patch, and creates learning data. As a result, in this embodiment, the learning data required for learning can be increased, and the estimation accuracy using the generated learning model for predicted appearance can be improved. As the learning method, patches with the learning label "out of scope" can be excluded from the target of the learning data as "non-appearance", and two-class classification learning may be performed in the same manner as in the first embodiment.

[0083] Also, in this embodiment, when dividing an image into patches with a predetermined number of pixels, there may be a case where there is an image portion that does not reach the predetermined number of pixels. For example, when the control unit 101 divides an image into patches with 500 pixels, the number of pixels in the width and height of the inspection image to be divided is often not a multiple of 500. In this case, the control unit 101 may set the target for creating patches to the central portion of the inspection image and set the peripheral portion of the inspection image as a non-target area for patches. Further, when there is a bias in the appearance position of the differential change data in the inspection image, the control unit 101 may create patches so as to include the differential change data that appears in the peripheral portion in order to increase the number of learning target data.

[0084] <Third Embodiment> In the above embodiments, an example in which learning data is created based on information on whether differential change appears in the first inspection image has been described. In the third embodiment, an example in which the first inspection image and the differential change data are used as learning data will be described. For example, in the third embodiment, learning data is created using the differential change data as a unit.

[0085] Figure 13 shows the learning data table 1301 in the third embodiment. The learning data table 1301 includes an ID 1302 for identifying an image, a first inspection image file 1303, a differential deformation ID 1304, a type 1305, the number of vertices 1306, a vertex coordinate list 1307, and a learning label 1308.

[0086] Figure 14 shows the confirmation screen 1400 in the third embodiment. The control unit 101 displays, in the learning data display area 1401, an image in which the differential deformation data indicated by a double line is superimposed on the first inspection image. The pointer 1402 is a UI for the user to set whether to target the differential deformation data displayed by the double line for the learning label. The user operates the pointer 1402 to specify any of the differential deformation data and specifies whether to use or not use the specified differential deformation data as learning data. The control unit 101 sets, in the learning label 1308, whether to use the differential deformation data as learning data based on the specification by the user.

[0087] As described above, in the third embodiment, when an image is input and predicted using the learning model generated by learning to predict deformation by using the differential deformation data itself as learning data for the first inspection image, it is possible to estimate the deformation data that will appear in the future. As for the learning method, an existing machine learning method may be used, and learning for detecting deformation in pixel units may be performed using the method described in Non-Patent Document 1.

[0088] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiment to a system or device via a network or a storage medium and causing one or more processors in a computer of the system or device to read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0089] The above embodiments may be combined. For example, the first to third embodiments may be combined so that a user can make a selection.

[0090] The disclosure of this specification includes the following information processing apparatus, information processing method, and program. (Item 1) An acquisition means for acquiring first change data indicating a change included in a first inspection image obtained by photographing a first region, and second change data indicating a change included in a second inspection image obtained by photographing a second region including at least a part of the first region at a time newer than the time when the first inspection image was photographed; A calculation means for calculating difference change data indicating a difference between the first change data and the second change data; A creation means for creating learning data for learning a model for predicting the occurrence of the change based on the difference change data; An information processing apparatus comprising the above. (Item 2) The creation means creates the learning data in units of inspection images based on the difference change data. The information processing apparatus according to Item 1, characterized by the above. (Item 3) The creation means creates the learning data in units of regions obtained by dividing the inspection image based on the difference change data. The information processing apparatus according to Item 1 or Item 2, characterized by the above. (Item 4) The creation means creates the learning data in units of the difference change data based on the difference change data. The information processing apparatus according to any one of Items 1 to 3, characterized by the above. (Item 5) The creation means classifies and creates the learning data based on the difference change data. The information processing apparatus according to any one of Items 1 to 4, characterized by the above. (Item 6) The creating means creates the learning data based on the type of the second deformation data with respect to the first deformation data associated with the differential deformation data. The information processing apparatus according to any one of Items 1 to 5, characterized in that. (Item 7) The creating means classifies and creates the learning data based on the number of the differential deformation data. The information processing apparatus according to any one of Items 1 to 6, characterized in that. (Item 8) The creating means classifies and creates the learning data based on the length of the differential deformation data. The information processing apparatus according to any one of Items 1 to 7, characterized in that. (Item 9) The creating means displays the learning data. The information processing apparatus according to any one of Items 1 to 8, characterized in that. (Item 10) The creating means edits the learning data based on an instruction of the learning data received from a user. The information processing apparatus according to any one of Items 1 to 9, characterized in that. (Item 11) The calculating means corrects a misalignment between the first inspection image and the second inspection image, and calculates the differential deformation data. The information processing apparatus according to any one of Items 1 to 10, characterized in that. (Item 12) The calculating means calculates, as the differential deformation data, the second deformation data existing outside an expansion range obtained by expanding the first deformation data in a width direction. The information processing apparatus according to Item 1, characterized in that. (Item 13) The creating means sets the type of the second deformation data based on an overlap between the first deformation data and the second deformation data. The information processing apparatus according to Item 6, characterized in that. (Item 14) An acquisition step of acquiring first deformation data indicating a deformation included in a first inspection image obtained by photographing a first region, and second deformation data indicating a deformation included in a second inspection image obtained by photographing a second region including at least a part of the first region at a time later than the time when the first inspection image was photographed; A calculation step of calculating difference deformation data indicating a difference between the first deformation data and the second deformation data; A creation step of creating learning data for training a model for predicting the occurrence of the deformation based on the difference deformation data; An information processing method, characterized by comprising the above. (Item 15) A program for causing a computer to function as each means of the information processing apparatus according to any one of Items 1 to 13.

[0091] The invention is not limited to the above embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, claims are attached to disclose the scope of the invention.

Explanation of Signs

[0092] 100... Information processing apparatus, 101... Control unit, 601, 650, 671, 672, 673... Deformation data, 611... Expansion deformation data, 726, 1107, 1305... Types, 804... Number of differences, 805... Difference length, 808, 1106, 1202, 1308... Learning labels, 1001 to 1048... Patches.

Claims

1. An acquisition means for acquiring first deformation data indicating a deformation included in a first inspection image obtained by photographing a first region, and second deformation data indicating a deformation included in a second inspection image obtained by photographing a second region including at least a part of the first region at a time later than the time when the first inspection image was photographed; A calculation means for calculating difference deformation data indicating a difference between the first deformation data and the second deformation data; A creation means for creating learning data for learning a model for predicting the occurrence of the deformation based on the difference deformation data; An information processing apparatus comprising the above.

2. The creation means creates the learning data in units of inspection images based on the difference deformation data. The information processing apparatus according to claim 1, characterized in that.

3. The creation means creates the learning data in units of regions obtained by dividing the inspection image based on the difference deformation data. The information processing apparatus according to claim 1, characterized in that.

4. The creation means creates the learning data in units of the difference deformation data based on the difference deformation data. The information processing apparatus according to claim 1, characterized in that.

5. The creation means classifies and creates the learning data based on the difference deformation data. The information processing apparatus according to claim 1, characterized in that.

6. The creation means creates the learning data based on the type of the second deformation data with respect to the first deformation data associated with the difference deformation data. The information processing apparatus according to claim 1, characterized in that.

7. The creation means classifies and creates the learning data based on the number of the difference deformation data. The information processing apparatus according to claim 1, characterized in that.

8. The creation means classifies and creates the learning data based on the length of the difference deformation data. The information processing apparatus according to claim 1, characterized in that.

9. The creation means displays the learning data. The information processing apparatus according to claim 1, characterized in that.

10. The creation means edits the learning data based on an instruction of the learning data received from a user. The information processing apparatus according to claim 1, characterized in that.

11. The calculation means corrects a positional shift between the first inspection image and the second inspection image, and calculates the difference deformation data. The information processing apparatus according to claim 1, characterized in that...

12. The calculating means calculates the second deformation data existing outside the expansion range obtained by expanding the first deformation data in the width direction as the differential deformation data. The information processing apparatus according to claim 1, characterized in that...

13. The creating means sets the type of the second deformation data based on the overlap between the first deformation data and the second deformation data. The information processing apparatus according to claim 6, characterized in that...

14. An acquisition step of acquiring first deformation data indicating a deformation included in a first inspection image obtained by photographing a first region, and second deformation data indicating a deformation included in a second inspection image obtained by photographing a second region including at least a part of the first region at a time later than the time when the first inspection image was photographed; A calculation step of calculating differential deformation data indicating the difference between the first deformation data and the second deformation data; A creation step of creating learning data for training a model for predicting the occurrence of the deformation based on the differential deformation data; An information processing method, characterized by comprising the above.

15. A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 13.

Images