Information processing device, information processing method, and program

Abstract

To reduce time and labor spent prior to inspection work.SOLUTION: An information processing apparatus generates reference information including information on a first target image, and a position and size of a defect identified from the first target image, selects any of one or more pieces of generated reference information, and displays a reference image, which is at least a part of the first target image, on the basis of the selected reference information. The information processing apparatus also acquires a first display parameter concerning display of the reference image, and sets a second display parameter concerning display of the target image, in which the defect is inspected, on the basis of the acquired first display parameter. Then, the information processing apparatus displays the target image obtained by adjusting the display parameter on the basis of the set second display parameter on a display device so as to be compared with the reference image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a technique used for inspecting objects using images. [Background technology]

[0002] Inspection work for inspecting structures and other objects involves going to the site and taking photographs of the structure. After returning to the office, the photographed images are carefully observed to determine any abnormalities such as cracks, lifting, or peeling on the concrete surface, and an inspection report is then created. Determining abnormalities here refers to the act of identifying the type and extent of the abnormality shown in the image. For example, determining that a certain abnormality is a type of "crack" and the extent of the abnormality is "the crack width is 0.2 (mm)".

[0003] When determining abnormalities, even if the abnormalities are of the same type and degree, the appearance changes significantly depending on the material of the subject and the shooting environment, making it difficult for inspectors to have a consistent standard for their judgment. Also, when the structure to be inspected is large, the number of images that the inspector must check tends to be enormous. For these reasons, it is difficult to continue consistent abnormality judgments in the inspection work of structures, and the judgment results are likely to vary.

[0004] In response to the issue of maintaining consistency of judgment criteria in judging and diagnosing abnormalities by observing such images, Patent Document 1 discloses a method of comparing an inspection image with a reference image that is used as a comparison target in the inspection of electronic components. In addition, Patent Document 1 also discloses that a parameter for adjusting the appearance is obtained from the reference image, and the inspection image is photographed so as to match the appearance with the reference image, making it easier to compare the two. Patent Document 2 discloses a technology that holds parameters for adjusting the appearance of lesions from a CT (computed tomography) image as presets, and adjusts the appearance of a specified region of interest by applying the presets, making it easier to make a consistent judgment. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2016-65875 A [Patent Document 2] JP 2002-165786 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the technique disclosed in Patent Document 1 requires time and effort to prepare reference images for each type and level of abnormality.The technique disclosed in Patent Document 2 also requires time and effort to prepare parameter presets in advance.

[0007] Therefore, an object of the present invention is to make it possible to reduce the effort required prior to inspection work. [Means for solving the problem]

[0008] The present invention has an acquisition means for acquiring a first image of a structure, a display control means for classifying a plurality of second images of the structure according to the width of the deformation of the structure and displaying them on a display unit, a selection means for selecting one of the second images based on a user operation, and a determination means for determining the width of the deformation of the structure included in the first image based on the second image selected by the selection means, and the display control means displays the first image and the second image selected by the selection means on the same screen. When the zoom magnification is changed based on a user operation, the ratio of pixel units to distance units of the first captured image and the second captured image selected by the selection means is adjusted, and the first captured image is displayed with a position adjusted so that the partial image displayed immediately before the zoom magnification was changed is at the center, and the second captured image is displayed with a position adjusted so that the abnormality is included in the displayed partial area. It is characterized by: Effect of the Invention

[0009] According to the present invention, it is possible to reduce the effort required prior to inspection work. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an overall configuration of an information processing apparatus according to an embodiment. [Diagram 2] FIG. 2 is a functional block diagram of a structure inspection process according to the first embodiment. [Diagram 3] FIG. 4 illustrates a table for storing reference information according to the first embodiment; [Figure 4] FIG. 2 is a diagram illustrating a screen configuration according to the first embodiment. [Diagram 5] 11A to 11C are diagrams illustrating alternative examples of the screen configuration of the first embodiment. [Figure 6] 13 is a flowchart of an image adjustment process according to reference information selection. [Figure 7] 13 is a flowchart illustrating a reference information generating process. [Figure 8] FIG. 11 is a functional block diagram of a structure inspection process according to a second embodiment. [Figure 9] 13 is a flowchart of an image adjustment process in response to a user operation on a target image. [Figure 10] FIG. 13 is a functional block diagram of a structure inspection process according to a third embodiment. [Figure 11] FIG. 13 illustrates a table for storing reference information according to the third embodiment. [Figure 12] FIG. 13 is a diagram illustrating a screen configuration according to a third embodiment. [Figure 13] 13 is a flowchart of an image adjustment process according to a third embodiment. [Figure 14] 11A to 11C are diagrams illustrating an example of a transition of a display state. [Figure 15] 11A to 11C are diagrams illustrating an example of a transition of a display state. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to those configurations. In this embodiment, an example of a structure inspection task is described, in which an image of a structure such as a building as an inspection target is used to inspect the structure for abnormalities. <First embodiment> FIG. 1 is a schematic hardware configuration diagram of an information processing device according to the first embodiment. The CPU 101 is a central processing unit that controls the computer system. The CPU 101 performs calculations and processing of information, controls each hardware, and the like based on a control program and a processing program according to this embodiment, thereby realizing various processes including a structure inspection process described later and the configuration of each functional block. The RAM 102 is a random access memory, and functions as a main memory of the CPU 101, as a work memory required for loading an execution program and executing the program. The ROM 103 is a read-only memory that records a control program that defines the operation processing procedure of the CPU 101 and a processing program according to this embodiment. The ROM 103 includes a program ROM that records an operating system (OS), which is a system program that controls the equipment of the computer system, and a data ROM that records information required to operate the system. The HDD 107 described later may be used instead of the ROM 103. The NETIF 104 is a network interface that controls input and output of data such as images transmitted and received via a network. The program according to this embodiment may be downloaded via the NETIF 104 and recorded in the HDD 107 or the like. The display device 105 is, for example, a CRT display or a liquid crystal display. The input device 106 is an operation input unit for receiving operation instructions from a user, and is, for example, a touch panel, a keyboard, a mouse, etc. The HDD 107 is a hard disk drive and a storage device. The HDD 107 is used for storing application programs and data such as images. The bus 108 is an input / output bus (address bus, data bus, and control bus) for connecting the above-mentioned units.

[0012] FIG. 2 is a diagram showing functional blocks for implementing a structure inspection process when inspecting a structure, which is an object of inspection, using a photographed image of the structure in the information processing device of this embodiment. In this embodiment, by executing the structure inspection process using the functional blocks of FIG. 2, it is possible to reduce the amount of work required prior to the inspection work while suppressing the variation in the criteria for determining anomalies in the structure. The structure inspection process using each functional block shown in FIG. 2 is implemented, for example, by the CPU 101 in FIG. 1 executing the program according to this embodiment and controlling each unit. Note that the functional blocks shown in FIG. 2 may be implemented by a hardware configuration, a software configuration, or a combination of a hardware configuration and a software configuration. This also applies to the functional blocks of each embodiment described later.

[0013] The target image input unit 201 acquires an image of a structure to be inspected (hereinafter, referred to as a target image) from the NETIF 104 or the HDD 107. The target image display unit 202 displays a partial area of ​​the target image input to the target image input unit 201 in a predetermined display area on the screen of the display device 105. The partial area is determined by the user specifying an area in the target image by scrolling, zooming in and out, etc. via the input device 106.

[0014] The deformation designation unit 203 designates deformations in a partial region of the target image displayed on the target image display unit 202 based on input from the user via the input device 106. In this embodiment, the deformations are, for example, cracks, lifting, peeling, efflorescence, cold joints, junk (peanut brittle), surface bubbles, sand streaks, rust, and the like that occur on the surface of a structure. In this embodiment, the designation of the deformation is performed by the user encircling an area in the target image that is thought to be deformed with a rectangle or ellipse via the input device 106, tracing the outline, and the like. When the deformation is linear, such as a crack, the deformation may be designated by tracing over the linear deformation. The deformation designation unit 203 sends information representing the deformation designated in this way as a rectangle surrounding the deformation, the outline of the deformation, and a polyline tracing the deformation itself to the target image display unit 202. As a result, the target image display unit 202 displays the rectangle surrounding the deformation, the outline of the deformation, the polyline, and the like on the target image. Since polylines tend to have a large amount of data, the deformation designation unit 203 may calculate a circumscribing rectangle of the polyline and send information expressing the deformation area as a rectangle to the target image display unit 202. The deformation designated by a rectangle in this way is represented, for example, by the coordinates of the upper left corner and the size (width and height) of the rectangle, which makes it possible to uniquely identify the position and range within the target image. In this embodiment, the deformation is designated by surrounding it with a rectangle as described above. Specific display examples of partial areas of the target image and rectangles designating deformations within the target image will be described later.

[0015] The deformation determination unit 204 acquires the result of the deformation determination input by the user via the input device 106 for the partial area of ​​the target image for which a deformation has been specified by the deformation specification unit 203. In this embodiment, the result of the deformation determination input by the user includes, for example, the type of deformation and the degree of the deformation. Deformation determination by the user and specific examples of the determination results will be described later.

[0016] The reference information display unit 207 displays the reference information managed by the reference information storage unit 206, which will be described later, on the display device 105. The reference information is information in which, for a reference image that serves as a reference when performing a deformation determination of a target image, information indicating at least the type of deformation of the reference image and the degree of the deformation is associated with the reference image. In this embodiment, the reference image includes a partial image of the target image that includes a deformation specified by the deformation designation unit 203. The reference information includes information indicating the result of the deformation determination determined by the deformation determination unit 204 in the partial region. The information associated with the reference image may include display parameters acquired from the reference information storage unit 206 by the display parameter acquisition unit 205, which will be described later, or display parameters acquired from the partial region specified by the deformation designation unit 203. In this embodiment, multiple pieces of reference information are stored, for example, in an external recording device via the HDD 107 or the NETIF 104, and the reference information storage unit 206 manages the multiple pieces of reference information stored therein.

[0017] The reference information selection unit 209 displays a list of multiple pieces of reference information managed by the reference information storage unit 206 in a predetermined display area of ​​the display device 105, and allows the user to select one of them. In the list display, a reference image is acquired based on the reference information, and thumbnail images created by applying appropriate trimming or reduction processing to the reference image may be displayed side by side, or character strings that can uniquely identify the reference image may be displayed side by side. A specific display example of the list of reference information will be described later.

[0018] In addition, when displaying a list of reference information, the reference information may be grouped, sorted, or filtered according to the type and degree of deformation contained in each piece of reference information. In this case, the reference information selection unit 209 has a grouping function for grouping multiple pieces of reference information based on the type and degree of each piece of reference information, a sorting function for sorting based on the type and degree of reference information, and a filtering function for filtering based on the type and degree of reference information.

[0019] Then, when a piece of reference information is selected by the user from the list display via the input device 106, the reference information display unit 207 acquires a reference image based on the selected reference information, and displays it in a predetermined display area of ​​the display device 105. A specific example of displaying the reference image will be described later.

[0020] Here, when abnormality designation and abnormality determination are performed for a partial region of a target image, the information processing device of this embodiment displays the partial region of the target image and the reference image on the display device 105 so that the user can visually compare them. When displaying the partial region of the target image and the reference image so that they can be compared, the information processing device of this embodiment performs an adjustment process to match the appearance (display manner) of the target image and the reference image on display. In the first embodiment, the display manner of the target image is matched to the display manner of the reference image, thereby realizing an adjustment process to match the appearance of the target image and the reference image on display.

[0021] For this reason, the display parameter acquisition unit 205 acquires display parameters from a reference image based on the reference information selected by the reference information selection unit 209. Furthermore, if the selected reference information already contains display parameters, the display parameter acquisition unit 205 acquires the display parameters contained in the selected reference information from the reference information storage unit 206. In this embodiment, the display parameters are parameters used to align the display manner of images that appear differently and to align the appearance on the display.

[0022] In this embodiment, by displaying the target image and the reference image in a comparable manner, it becomes easy to compare the abnormality already identified in the reference image with the candidate abnormality before the determination contained in the target image, and the work of determining the abnormality in the target image is supported. Therefore, in this embodiment, "matching the appearance on the display" means making the display environment of each of the target image and the reference image equivalent so that it is easier to compare them. For example, the zoom magnification of both is matched. In this embodiment, since the reference image includes a partial image of the target image in which an abnormality has already been determined, by matching the zoom magnification of the target image and the zoom magnification of the reference image, if an abnormality of the same width is shown in each of the target image and the reference image, it appears to be the same width on the display. Details of the display parameters will be described later using a specific example in FIG. 6. Then, the display parameter acquisition unit 205 sends the acquired display parameters to the display parameter application unit 208.

[0023] The display parameter application unit 208 applies the display parameters to a partial region of the target image, thereby adjusting the image so that the appearance of the image (reference image) from which the display parameters are obtained matches the appearance of the image (target image) to which the display parameters are applied. Specific examples of application of the display parameters will be described later.

[0024] Furthermore, when an abnormality is specified and an abnormality determination is performed for a partial region of the target image, the reference information storage unit 206 of this embodiment generates and manages information that associates the abnormality determination result (type and degree of abnormality, etc.) with the partial region as new reference information. Then, the reference information storage unit 206 stores the newly generated reference information in, for example, the HDD 107 or an external recording device via the NETIF 104. That is, in this embodiment, the reference information related to the partial region in the target image where an abnormality has been determined by the user can be used as reference information when an abnormality determination is subsequently performed for this target image.

[0025] 3 is a diagram illustrating a reference information table used for managing the reference information held by the reference information holding unit 206. In the following, an example will be given of anomalies handled in a structure inspection, particularly a crack. In FIG. 3, a reference ID for uniquely identifying reference information stored in the reference information table is described in a reference ID item 301 of the reference information table. The relative coordinates of a rectangle that represents the region of the abnormality in the target image are described in the coordinate item 302. In this embodiment, it is assumed that the relative coordinates of the upper left corner of the rectangle in the target image are used as the coordinates described in the coordinate item 302, but the coordinates are not limited to this and may be the center coordinates or the lower right coordinates of the rectangle.

[0026] The size of the rectangle representing the area of ​​the abnormality is described in the size item 303. Here, the size of the rectangle is made up of the width and height of the rectangle. A character string representing the type of deformation is written in the type item 304. A value representing the degree of deformation is written in the degree item 305. In the example of this embodiment, the width of the crack is written as the value representing the degree of deformation. Display parameters are described in the display parameter item 306. As described above, the display parameters are used to match the appearance of another image (a partial region in this embodiment) with respect to the reference image specified by the coordinates in the coordinate item 302 and the size in the size item 303. In this embodiment, a case where a zoom magnification is adopted as an example of the display parameters will be described.

[0027] 4(a) and 4(b) are diagrams for explaining an example of a display on the display device 105 when a structure inspection process is being executed in the information processing device of this embodiment. The difference between Fig. 4(a) and Fig. 4(b) is that the method of expressing the reference information list is different. The target image read button 401 is a button that is operated by the user when the target image is read by the target image input unit 201 .

[0028] The target image display area 402 is a display area in which the target image is displayed by the target image display unit 202. After acquiring one target image, the user can scroll or zoom in / out to move the partial area displayed in the display area 402 while determining the abnormality and inspecting the required range of the target image. For example, scroll bars 409 and 410 can be used to move the partial area. In addition, in response to a predetermined operation on the target image display area 402 (for example, right-clicking, double-tapping, etc.), An interface for changing display parameters such as zoom magnification, brightness adjustment value, hue adjustment value, gamma value, etc. may be displayed.

[0029] The reference image display area 403 is a display area in which the reference image is displayed by the reference information display unit 207 . The deformation area indication rectangle 404 is a rectangle that represents the deformation area specified by the deformation specification unit 203 . The deformation determination button 405 is a button that is operated when the user inputs the type and degree of deformation by the deformation determination unit 204 for the deformation indicated by the deformation area indication rectangle 404. Although not shown in the figure, when the deformation determination button 405 is pressed, the information processing device displays a dialog window or the like on the screen for inputting the type and degree of deformation, and prompts the user to input them.

[0030] 4(a) is an area that displays a list of reference information managed by the reference information storage unit 206. This allows the user to select reference information from the list. The reference thumbnail image 407 is a thumbnail image that represents the reference image, and is created by appropriately trimming, reducing, or enlarging the reference image acquired based on the reference information. In the example of Fig. 4(a), when a deformation is determined, a partial image corresponding to the deformation area indication rectangle 404 designated by the deformation designation unit 203 is trimmed, and the reduced thumbnail image is displayed in the reference information list area 406. Figure 4(a) shows an example in which reference thumbnail images 407 are grouped based on the type and degree of deformation contained in the reference information (in this example, the crack width of the crack) and listed in the reference information list area 406.

[0031] The reference information list area 408 in Fig. 4(b) is an area where a list of reference information managed by the reference information storage unit 206 is displayed. The reference information list area 406 illustrated in Fig. 4(a) displays a list of thumbnail images representing reference images, while the reference information list area 408 in Fig. 4(b) displays a list of character strings that can uniquely identify the reference images. An example of a character string that can uniquely identify a reference image is a character string such as "deformation_(reference ID)". FIG. 4(b) shows an example in which character strings that can uniquely identify reference images are grouped based on the type and degree (crack width) contained in the reference information and listed in the reference information list area 408.

[0032] 4(a) and 4(b) show an example in which the target image display area 402 and the reference image display area 403 are displayed side-by-side to enable comparison between the two, but a display such as that shown in Fig. 5 may also be used. Fig. 5 shows a display example in which the target image display area 402 and the reference image display area 403 are arranged so as to be superimposed on each other in the same area on the screen (the same display area), and the visibility of the two is switched to enable comparison.

[0033] In the example of Fig. 5, the inspection / reference image display area 501 is a display area for switching between the target image display area 402 and the reference image display area 403. The switching between the target image display area 402 and the reference image display area 403 in the inspection / reference image display area 501 is performed, for example, by operating a toggle button such as an inspection / reference image switching button 502. Note that in Fig. 5, the target image load button 401, the abnormality determination button 405, the reference information list area 406, and the reference thumbnail image 407 are the same as those in Fig. 4(a).

[0034] Hereinafter, a process for aligning the display manner of the target image and the reference image (a process for aligning the appearance) will be described with reference to Fig. 6. In addition, a process for generating new reference information based on the result of the abnormality determination by the user will be described with reference to Fig. 7. Fig. 6 is a flowchart for explaining the process executed when reference information is selected from the reference information list in the information processing device of this embodiment. In this embodiment, the reference information is generated based on an abnormality in the target image displayed on the target image display unit 202, as described later in Fig. 7. The process shown in the flowcharts of Figs. 6 and 7 below is realized by the CPU 101 of Fig. 1 executing the program according to this embodiment. In the following description, steps S601 to S604, step S701, and step S702 are abbreviated as S601 to S604, S701, and S702, respectively. The same applies to other flowcharts described later.

[0035] In S601, when the user selects one of the above-mentioned reference information list areas (406 or 408), the reference information selection unit 209 acquires, from the reference information storage unit 206, reference information corresponding to the selection. Next, in S602, the display parameter acquisition unit 205 acquires display parameters from the reference information acquired in S601. Here, the above-mentioned zoom magnification is acquired as an example of the display parameters.

[0036] Next, in S603, the reference information display unit 207 displays a reference image in the reference image display area 403 based on the reference information acquired in S601. At this time, the reference information display unit 207 first acquires the upper left coordinate and size of a rectangle surrounding the deformation from the selected reference information. Then, the reference information display unit 207 cuts out a rectangular image based on the upper left coordinate and size of the rectangle from the image from which the reference information is generated (i.e., the target image displayed by the target image display unit 202). Furthermore, the reference information display unit 207 applies the zoom magnification acquired by the display parameter acquisition unit 205 in S602 to the cut-out rectangular image to reproduce the display manner at the time of the deformation determination when the reference information was generated. The reproduction of the display manner can be realized by applying a scale transformation matrix created based on the zoom magnification to the cut-out rectangular image. Then, the reference information display unit 207 displays the rectangular image that reproduces the display manner as a reference image in the reference image display area 403.

[0037] In this embodiment, if the rectangular image that reproduces the display manner of the rectangle (inside the deformation area indication rectangle 404) surrounding the deformation when the reference information is generated is smaller than the reference image display area 403, the periphery of the rectangle surrounding the deformation is also displayed. As a result, when the reference thumbnail image 407 is selected, the reference image displayed in the reference image display area 403 is the part of the target image that was displayed in the entire target image display area 402 when the deformation was determined. However, this is not limited to this. For example, the rectangle surrounding the deformation when the reference information is generated may be displayed so as to be located at the center of the reference image display area 403. Also, if the rectangle surrounding the deformation is smaller than the reference image display area 403, the outside of the rectangle may be left blank.

[0038] Next, in S604, the display parameter application unit 208 applies the zoom magnification acquired by the display parameter acquisition unit 205 in S602 to the target image displayed on the target image display unit 202, thereby reproducing the same display manner as the reference image. This realizes image adjustment to match the appearance (display parameters) of the target image and the reference image. In this case, as described above, a scale transformation matrix created based on the zoom magnification may be applied to the target image. If the selected reference image is a partial image of the target image, the zoom magnification of the target image and the zoom magnification of the reference image are matched, so that the width of the deformation of the same width appearing in each of the target image and the reference image also matches on the display device. However, in this embodiment, even after the reference image is selected and the display parameters of the target image are changed by the information processing device, the user can move the displayed partial area by scrolling or zooming in / out and change the zoom magnification. For example, after confirming that the target image contains a deformation having a width similar to the referenced deformation by displaying it at the same zoom magnification as the reference image, the length of the deformation can be confirmed by zooming out arbitrarily. When an abnormality is judged in a state where the zoom-out is performed arbitrarily, the zoom magnification finely adjusted by the zoom-out is held as reference information of a reference image newly generated in accordance with the judgment. In the first embodiment, when the display parameters of the target image are changed by the user, the change does not affect the reference image.

[0039] In this embodiment, an example has been given in which the zoom magnifications of the target image and the reference image are made uniform to make them look the same, but a display parameter other than the zoom magnification may be used. In this case, the display parameter may be, for example, a brightness adjustment value, a hue adjustment value, a gamma value, etc. Furthermore, the display parameter used may be not only one of the zoom magnification, the brightness adjustment value, the hue adjustment value, and the gamma value, but also a combination of these.

[0040] For example, when a brightness adjustment value is used as a display parameter, in S602, the display parameter acquisition unit 205 acquires brightness from the reference image to generate a brightness adjustment value. Then, in S604, the display parameter application unit 208 uses the brightness adjustment value to adjust the brightness of the target image. Also, for example, when the HSV color space is used, the maximum of RGB becomes the brightness adjustment value V, and in this case, the display parameter application unit 208 adjusts the brightness adjustment value Vi of the target image to the brightness adjustment value Vr of the reference image. Note that at this time, it is necessary to change values ​​other than Vi (for example, if R is Vi, G and B) according to the change in the brightness adjustment value Vi of the target image so as not to destroy the ratio of RGB before the change. Regardless of which display parameter is adopted, the environment in which the target image and the display image are displayed can be aligned, making it easier to compare the abnormality already identified in the reference image with the candidate for change before determination included in the target image. In this embodiment, the above-described process is performed every time reference information is selected from the reference information list.

[0041] FIG. 7 is a flowchart illustrating a process for generating new reference information, which is executed when the abnormality determination button 405 is pressed in the information processing device of this embodiment. In S701, when the deformation determination button 405 is pressed, the deformation determination unit 204 generates new reference information including the partial area of ​​the target image determined to be deformed at that time and information indicating the type and degree of deformation, and stores it in the reference information storage unit 206. In this case, the reference information storage unit 206 describes the reference information in each item of the reference information table of Fig. 3 described above. The reference information at this time also includes the current zoom magnification of the target image acquired by the display parameter acquisition unit 205. Depending on the type of display parameter, it is not always necessary to acquire and store the display parameter at this timing. For example, in the case of the brightness adjustment value described above as an example of a display parameter, the brightness adjustment value may be calculated when the reference information is generated and stored as part of the reference information, but the brightness adjustment value may also be calculated when the reference image is selected.

[0042] Next, in S702, the reference information display unit 207 additionally displays the display element generated based on the reference information generated in S701, for example, in the reference information list area 406 of Fig. 4(a). Specifically, the reference information display unit 207 resizes the deformed portion of the reference image generated based on the new reference information, for example, to the size of a thumbnail image, and additionally displays it in the reference information list area 406. Alternatively, the reference information display unit 207 may generate an appropriate character string based on a reference ID indicating the reference information, and additionally display it, for example, in the reference information list area 408 of Fig. 4(b). The flowchart in FIG. 7 may be started not only when the abnormality determination button 405 is pressed, but also at any other timing, such as when the user specifies an abnormality.

[0043] Here, with reference to FIG. 14, an example of transition of the display state on the display device 105 when a structure inspection process is executed in the information processing device of this embodiment will be described. 14(a) to (d) show, in chronological order, the changes in the display state that occur when a user performs a structure inspection operation using the screen shown in FIG. 4(a). In FIG. 14(a), the user sets a defect designation region 404 in the target image display region 402 as shown in FIG. 4(a).

[0044] Here, thumbnail image 1400 is selected as the reference image, and as a result, a reference image based on thumbnail image 1400 is displayed in reference image display area 403. In FIG. 14(a), the selected thumbnail image 1400 is outlined in a thick frame, so that other thumbnail images can be clearly identified. Furthermore, in FIG. 14(a), it is assumed that the user operates determination button 405, and the inside of deformation area indication rectangle 404 is registered as a crack with a width of 5.0 mm. As a result, thumbnail image 1401 is added to reference information list area 406.

[0045] 14(b) shows the state in which, as the user proceeds with the work, he or she selects thumbnail image 1402, which is grouped as a 0.2 mm-wide defect, from reference information list area 406. The selected thumbnail image 1402 is outlined in a thick frame, making it clearly identifiable from other thumbnail images. In the reference image display area 403, a reference image based on the selected thumbnail image 1402 is displayed.

[0046] Here, the area cut out as thumbnail image 1402 is the lower right portion of the entire reference image displayed in reference image display area 403. In response to the change in the reference image, the zoom magnification of the target image is changed in target image display area 402, and the target image is enlarged more than in the state of FIG.

[0047] The target image displayed in Fig. 14(b) is the part of the target image displayed in Fig. 14(a) that is below the part selected in the abnormality designation area 404. For example, when the user wants to perform work focusing on an abnormality with a narrower width than the state in Fig. 14(a) (where the abnormality with a width of 0.5 mm was being referenced), the user selects thumbnail image 1402 as shown in Fig. 14(b) in order to compare it with the abnormality with a width of 0.2 mm. In response to the selection of thumbnail image 1402, the information processing device zooms in on the target image. As a result, the user can easily compare the newly focused fine abnormality in the target image with the abnormality with a width of 0.2 mm at the same zoom magnification.

[0048] 14(c) shows the state in which the user has set a new abnormality designation area 1403 in the target image display area 402, pressed the determination button 405, and registered an abnormality with a width of 0.2 mm. As a result, a thumbnail image 1403 is added to the 0.2 mm wide grouping in the reference information list area 406. The thumbnail image 1404 is a reduced image in which the portion of the target image surrounded by the abnormality designation area 1403 has been cut out.

[0049] In this manner, in this embodiment, each time a thumbnail image is selected from the reference information list area 406, the display parameters (e.g., zoom magnification) in the target image display area 402 are changed. Also, each time a new abnormality is determined in the target image, reference information is generated. At that time, in the display state, a character string capable of uniquely identifying the reference information is added to the reference information list area 406.

[0050] In addition, in Figures 14(b) and 14(c), an example is shown of determining a 0.2 mm wide anomaly in a target image while referring to a reference image of a 0.2 mm wide anomaly, but it is not necessary to refer to an anomaly of the same width. For example, when comparing with a reference image of a 0.5 mm wide anomaly, it may be confirmed that the anomaly in the target image is narrower than that in the reference image, and as a result, a 0.2 mm wide anomaly may be determined. In other words, when a new thumbnail image is added to the reference information list, it is not necessarily added to the same group as the thumbnail image that is currently selected.

[0051] In this embodiment, the newly generated reference information can be selected for reference in a subsequent abnormality determination operation. For example, FIG. 14(d) shows a state in which the newly added thumbnail image 1401 in FIG. 14(a) has been selected. In the reference image display area 403, a partial image of the target image that was displayed when the abnormality determination was performed on the abnormality designated area 404 in Fig. 14(a) is displayed as a reference image. Also, at this time, in the target image display area 402, the target image is displayed at the zoom magnification of the target image that was displayed when the abnormality determination was performed on the abnormality designated area 404 in Fig. 14(a). As a result, in Fig. 14(a) and Fig. 14(d), the images displayed in the target image display area 402 and the reference image display area 403 look exactly the same.

[0052] However, this is not limited to the present embodiment. For example, in the reference image display area 403, a reference image may be displayed such that the area cut out from the selected thumbnail image 1401 is at the center. Also, in the target image display area 402, the part to be displayed may be aligned so that the partial image that was displayed immediately before the zoom magnification was changed is at the center.

[0053] In this way, in the first embodiment, a new reference image can be generated from the target image based on the user's anomaly determination result. As a result, according to the first embodiment, it is possible to save the trouble of preparing a reference image in advance for each type and degree of anomaly. In addition, in the first embodiment, the conditions (environments) of how the target image and the previously determined anomaly image (reference image) are viewed can be aligned. Specifically, in order to display the target image and the reference image, which is a part of the target image, in a state that makes it easier to compare them, the display parameters of each are matched. As a result, according to the first embodiment, it becomes possible to determine a new anomaly by referring to an anomaly in a previously determined anomaly image (reference image), and it becomes possible to provide consistency in the anomaly determination.

[0054] Second Embodiment In the first embodiment described above, the conditions for how the target image appears are aligned based on the reference information selected by the user. Also, when the user performs an operation to change the appearance of the target image, the effect is not extended to the reference image. In contrast, in the second embodiment described below, an operation is described in which, when the way the target image is displayed is changed by a user operation such as a change in the zoom magnification for the target image display area 402, the change is reflected in the display mode of the reference image. That is, in the second embodiment, a process of adjusting display parameters is performed in order to equalize the conditions (environment) for how the reference image appears based on the target image.

[0055] Fig. 8 is a diagram showing functional blocks for realizing a structure inspection process in an information processing device of the second embodiment. The target image input unit 201 to the deformation determination unit 204 and the reference information storage unit 206 to the reference information selection unit 209 are similar to the corresponding units in Fig. 2, and therefore their description will be omitted. The display parameter acquisition unit 801 in the second embodiment acquires display parameters from the target image or the target image display unit 202.

[0056] 9(a) and 9(b) are flowcharts showing two types of processing executed when the display method of the target image is changed by a user operation in the structure inspection processing in the information processing device of the second embodiment. The processing of the flowcharts in Fig. 9(a) and 9(b) is started when the parameters of the target image display area 402 are changed by the user.

[0057] The flowchart in FIG. 9(a) shows a process for matching the appearance of the two images by applying display parameters that have been changed in the target image as display parameters of one selected reference image. In S901, the display parameter acquisition unit 801 acquires the current display parameters set in the target image display unit 202. In the second embodiment, the display parameters are, for example, a zoom magnification, a brightness adjustment value, a hue adjustment value, a gamma value, and the like. Next, in S902, the display parameter application unit 208 changes the zoom magnification of the reference image displayed in the reference information display unit 207 to the zoom magnification acquired in S901.

[0058] Here, an example of transition of the display state on the display device 105 when the process of the flowchart in Fig. 9(a) is executed will be described with reference to Fig. 15. Figs. 15(a)-(b) show in chronological order the change of the display state that occurs when the process of the flowchart in Fig. 9(a) is executed when the user performs the work of inspecting a structure in the second embodiment.

[0059] FIG. 15(a) shows the same state as FIG. 14(b). In the reference image display area 403, a reference image based on the selected thumbnail image 1402 is displayed. 15(a), a zoom magnification change button 1501 is displayed in the target image display area 402. The zoom magnification change button 1501 is an example of an interface for operating display parameters that can be called up in response to a predetermined operation, such as a click or a double tap, in the target image display area 402. By operating the zoom magnification change button 1501, the user can arbitrarily change the zoom magnification of the target image.

[0060] 15(b) shows a state in which the inside of the target image display area 402 is further zoomed in in response to a user operation. In this embodiment, in response to a user operation on the target image, the same change in display parameters as the target image is reflected in the reference image display area 403. In the case of Fig. 15(b), the reference image is also zoomed in. In this embodiment, particularly when the reference image is zoomed in (enlarged), the image is aligned so that the displayed partial area always includes the already determined anomaly, so that the anomaly in the reference image is not lost. As a result, the anomaly cut out in the thumbnail image 1402 is enlarged and displayed in the reference image display area 403 in Fig. 15(b).

[0061] The flowchart in Fig. 9(b) shows the process of applying the display parameters changed in the target image as display parameters reflected not only in the selected reference image but also in all reference information included in the display information list. However, in the explanation of Fig. 9(b), the reference information list is assumed to be composed of thumbnail images of the reference images obtained based on each reference information, which are reduced or enlarged without being cropped. The process of S901 in FIG. 9B is similar to the process of S901 in FIG. 9A, and therefore description thereof will be omitted.

[0062] Next, in S903, the reference information selection unit 209 determines whether the subsequent processes of S904 and S905 have been completed for all reference information stored in the reference information storage unit 206. If the reference information selection unit 209 determines in S903 that the processes have been completed, it ends the process of the flowchart in Fig. 9(b). On the other hand, if the reference information selection unit 209 determines in S903 that the processes have not been completed, it advances the process to S904. In S904, the reference information selection unit 209 acquires, from the reference information storage unit 206, one piece of reference information for which the subsequent processes in S904 and S905 have not been completed.

[0063] Next, in S905, the reference information selection unit 209 obtains a reference image based on the reference information obtained in S904 using the method described in S603. The reference information selection unit 209 then applies a scale transformation matrix created based on the zoom factor obtained in S901 to the reference image, and then resizes it to the size of a thumbnail image. However, in this method, resizing processing must be performed twice, which increases the amount of calculation. For this reason, it is also possible to combine the zoom factor obtained in S901 with a scale coefficient for thumbnail generation calculated from the size ratio between the reference image and the thumbnail image, so that resizing processing can be performed only once. Note that, after the process of the flowchart in FIG. 9B is completed, when a thumbnail image is selected from the reference information list, the process of the flowchart in FIG. 9A may be executed.

[0064] Here, with reference to FIGS. 15(c) and 15(d), an example of transition of the display state on the display device 105 when the process of the flowchart in FIG. 9(b) is executed will be described. 15(c) to (d) show in chronological order the changes in the display state that occur as a result of the processing of the flowchart in FIG. 9(a) being executed when the user performs the task of inspecting a structure in the second embodiment.

[0065] In Fig. 15(c), the state of the target image display area 402 is the same as in Fig. 15(a). In the reference image display area 403, a reference image based on the selected thumbnail image 1503 is displayed. Here, the reference image corresponding to the thumbnail image 1503 is the same as the thumbnail image 1402 in Fig. 15(a). However, in Fig. 15(c), the thumbnail image displayed in the reference information list area 406 is a reduced version of the reference image itself. Even in this case, the user can arbitrarily change the zoom magnification of the target image by operating the zoom magnification change button 1501.

[0066] Fig. 15(d) shows a state in which the inside of the target image display area 402 has been further zoomed in in response to a user operation. In this embodiment, in response to a user operation on the target image, each of the thumbnail images displayed in the reference information list area 406 is zoomed in. In the case of Fig. 15(d), the reference image is also zoomed in by further performing the process of Fig. 9(a).

[0067] In this way, according to the second embodiment of the information processing device, the appearance of the reference image can always be aligned to match the target image based on user operations on the target image, thereby enabling the user to quickly determine any abnormalities.

[0068] <Third embodiment> In the first embodiment described above, a reference image was generated from the target image being displayed. In contrast, in the third embodiment, previously generated reference information is read (including reading from outside). In this case, it is recommended to shoot the deformed part at a predetermined resolution, but this is not always met depending on the situation at the shooting site. When previously generated reference information is read as in the third embodiment, the resolution of the reference image based on the read reference information may differ from the resolution of the captured target image. If the resolutions of the target image and the reference image are different in this way, it is difficult to align the display methods of the two even if the zoom magnification is simply adjusted as described above.

[0069] 10 is a functional block diagram for implementing the structure inspection process in the information processing device of the third embodiment. The target image display unit 202 to the reference information selection unit 209 are similar to the corresponding units in FIG. 2, and therefore the description thereof will be omitted. 10, a target image input unit 1001 acquires a target image from HDD 107 or NETIF 104. At that time, the target image input unit 1001 requests the resolution (unit: pixels / mm) of the target image to be input from the user. The resolution of the target image input by the user is held as temporary data (for example, held in a predetermined area of ​​RAM 102) while the target image is displayed on the target image display unit 202.

[0070] The reference information input unit 1002 acquires reference information from the HDD 107 or the NETIF 104. In this case, the reference information may be saved as a file or may be stored in a database. In order to input appropriate reference information for a target image to be inspected, the multiple pieces of reference information may be structured so as to be searchable. In this case, the reference information input unit 1002 has a search function for searching by the type or level of reference information, for example. In this case, the type and material of the structure where the deformation occurred, the position where the deformation occurred, the person who made the judgment, etc. can be used as search keys. The reference information acquired by the reference information input unit 1002 is temporarily stored, for example, by the reference information storage unit 206.

[0071] The reference information output unit 1003 outputs the reference information newly generated in the information processing apparatus of the third embodiment to the HDD 107 or the NETIF 104. The output reference information may be saved as a file or stored in a database.

[0072] 11 is a diagram illustrating a reference information table managed by the reference information storage unit 206 in the third embodiment. The reference ID item 301 to the display parameter item 306 are the same as the corresponding items in FIG. 3, and therefore the description thereof will be omitted. The reference information table in FIG. 11 has an image path item 1101 and a resolution item 1102 prepared. The image path item 1101 describes a path indicating an image in which the referenced abnormality appears. The resolution item 1102 describes the resolution of the image indicated by the path in the image path item 1101 (the image in which the referenced defect is reflected).

[0073] Fig. 12 is a diagram for explaining an example of a display screen in the information processing device of the third embodiment. The target image load button 401 to the reference thumbnail image 407 are similar to the corresponding display elements in Fig. 4, and therefore their explanation will be omitted. The reference information input button 1201 is a button that is operated by the user when inputting reference information through the reference information input unit 1002 . The reference information output button 1202 is a button that is operated by the user when the reference information output unit 1003 outputs the reference information.

[0074] Fig. 13 is a flowchart for explaining the process executed when reference information is selected from a list of reference information generated in the past in the structure inspection process realized by the information processing device of this embodiment. S601 is the same as the corresponding step in Fig. 6, so the description thereof will be omitted. In the case of the flowchart of Fig. 13, after S601, the process proceeds to S1301.

[0075] In S1301, the reference information selection unit 209 acquires the path and resolution of the image in which the abnormality appears, and the zoom magnification at the time of abnormality determination, from the reference information selected by the user. Next, in S1302, the target image display unit 202 acquires the resolution of the target image being displayed. The resolution of the target image is assumed to be stored as temporary data associated with the target image being displayed on the target image display unit 202.

[0076] Next, in S1303, the target image display unit 202 matches the appearance (apparent resolution) on the display of the target image being displayed with the image (reference image identified from the images showing the deformation) whose path, resolution, and zoom magnification were acquired in S1301. Specifically, the target image display unit 202 applies a scale coefficient calculated by equation (1) described later to the one of the two images with the higher resolution.

[0077] Here, the scale factor S1 of the image size is the resolution obtained in S1301. a Let the resolution obtained in S1302 be res b Then, equation (1) can be obtained.

number

[0078] Note that, here, the image with the higher resolution is appropriately scaled to match the appearance (apparent resolution) of the other image on the display, but the image with the lower resolution may also be appropriately scaled to match the apparent resolution of the other image on the display. In that case, the scale coefficient S2 calculated by the following formula (2) should be applied to the image with the lower resolution.

number

[0079] Next, in S1304, the display parameter application unit 208 applies the zoom magnification obtained in S1301 to the target image displayed on the target image display unit 202 after matching the appearance (apparent resolution) of both images on the display in S1303.

[0080] In this way, in the third embodiment, it is possible to reuse the reference information created when another user performs an inspection work using the information processing device of this embodiment. As a result, according to the third embodiment, the reference information created by a user (expert) who is skilled in inspection work can be used by an inexperienced user (novice), and even the beginner can make a judgment according to the judgment criteria of the expert.

[0081] In the third embodiment, the display method of the target image is aligned with the reference image, but it is of course possible to align the display method of the reference image with the target image. In this case, the processing is basically as described in the flowchart of FIG. 9(a) or FIG. 9(b), but it is necessary to align the apparent resolution of both images in advance as in S1303. In addition, in the third embodiment, an example was given in which previously generated reference information was displayed in a list for selection, but the reference information displayed in a list may include both previously generated reference information and reference information newly generated from the target image as in the above-mentioned embodiment.

[0082] As described above, according to each of the above-described embodiments of the present invention, it is possible to automatically generate a reference image to be referred to during an abnormality determination based on the abnormality determination result of the user. Also, according to each of the embodiments, it is possible to easily compare the image of the inspection target and the reference image for the abnormality determination by aligning the appearance of the two.

[0083] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiment is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0084] The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main characteristics. [Explanation of symbols]

[0085] 101: CPU, 102: RAM, 103: ROM, 104: NETIF, 105: display device, 106: input device, 402: target image display area, 403: reference image display area, 406, 408: reference information list area

Claims

1. An acquisition means for acquiring a first captured image of a structure; a display control means for classifying the second captured images of the structure according to the width of the deformation of the structure and displaying the classified images on the display unit; a selection means for selecting one of the second captured images based on a user operation; A determination means for determining a width of the deformation of the structure included in the first captured image based on the second captured image selected by the selection means, The display control means displays the first captured image and the second captured image selected by the selection means on the same screen, and when a zoom magnification is changed based on a user operation, adjusts the ratio of pixel units to distance units between the first captured image and the second captured image selected by the selection means, and displays the first captured image by aligning its position so that a partial image that was displayed immediately before the zoom magnification was changed is at the center, and displays the second captured image by aligning its position so that the abnormality is included in the displayed partial area.

23. An information processing apparatus comprising:

2. The information processing device according to claim 1, characterized in that the display control means classifies a specific type of deformation according to the width of the deformation of a structure and displays the classified deformation on the display unit.

3. The information processing apparatus according to claim 1 , further comprising a presentation unit that groups the second captured images for one of the types of the deformation based on a width of the deformation and presents the grouped images.

4. The information processing apparatus according to claim 1 , wherein the second captured image is generated by trimming an area including a deformation of the structure from a captured image of the structure.

5. 2 . The information processing apparatus according to claim 1 , wherein the display control means switches between displaying the first captured image and the second captured image selected by the selection means on the same screen.

6. 2 . The information processing apparatus according to claim 1 , wherein the display control means displays the first captured image and the second captured image selected by the selection means side by side on the same screen.

7. An acquisition step of acquiring a first captured image of the structure; a display control step of classifying the second captured images of the structure according to the width of the deformation of the structure and displaying the second captured images on a display unit; a selection step of selecting one of the second captured images based on a user operation; A determination step of determining a width of the deformation of the structure captured in the first captured image based on the second captured image selected in the selection step, In the display control step, the first captured image and the second captured image selected in the selection step are displayed on the same screen, and when a zoom magnification is changed based on a user operation, the ratio of pixel units to distance units of the first captured image and the second captured image selected in the selection step are adjusted, and the first captured image is displayed with a position adjusted so that a partial image displayed immediately before the zoom magnification was changed is at the center, and the second captured image is displayed with a position adjusted so that the abnormality is included in the displayed partial area.

23. An information processing method comprising:

8. An acquisition step of acquiring a first captured image of the structure; a display control step of classifying the second captured images of the structure according to the width of the deformation of the structure and displaying the second captured images on a display unit; a selection step of selecting one of the second captured images based on a user operation; A determination step of determining a width of the deformation of the structure captured in the first captured image based on the second captured image selected in the selection step, In the display control step, the first captured image and the second captured image selected in the selection step are displayed on the same screen, and when a zoom magnification is changed based on a user operation, the ratio of pixel units to distance units of the first captured image and the second captured image selected in the selection step are adjusted, and the first captured image is displayed with a position adjusted so that a partial image displayed immediately before the zoom magnification was changed is at the center, and the second captured image is displayed with a position adjusted so that the abnormality is included in the displayed partial area. A program for causing a computer to execute an information processing method comprising the steps of:

Images