System for inspecting object

Abstract

To provide a computer utilization system for supporting work for inspecting an object by using many images acquired from the object or work for compiling the inspection results in a report.SOLUTION: A data processing server 3 connects many images obtained by photographing an object, creates many images representing the whole of an inspection object and different parts, and displays an inspection work screen displaying an image requested from a user terminal 7 and a comment input tool in the user terminal 7. A comment inputted onto the work screen by the user is stored in association with the displayed image. The data processing server 3 automatically creates a report according to a report form by using the stored comment and an image associated with the comment.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a computer - based system for assisting in the work of inspecting, checking, or evaluating a target using images and creating a report describing the results.

Background Art

[0002] For the purpose of investigating and maintaining artificial facilities and natural objects, the target land or facility is photographed with a camera mounted on a moving object in the air or on the ground or carried by a person, and a large number of obtained images are inspected or evaluated by a person while looking at them, and a report is created in which the results are expressed using a combination of the photographed images and attached marks and text comments, etc. This kind of work is being carried out in various industrial fields.

[0003] Systems for creating reports are disclosed in, for example, Patent Document 1 and Patent Document 2.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] For example, when inspecting the state of a vast target such as a solar power generation facility based on images obtained by photographing it with a camera mounted on an unmanned aerial vehicle, the number of photographed images is very large for the entire target. It is not an easy task for a person to grasp the overall picture of the target by looking at those numerous images or to carefully inspect individual parts.

[0006] Furthermore, in order to create a report that can be easily understood by others based on the inspection results, it is necessary to skillfully arrange many images on the report and attach marks, comments, etc. to those images. Creating the report takes a lot of time. For example, it is said that it usually takes about a week to create a report by inspecting based on images taken of a solar power generation facility from an unmanned aircraft.

[0007] However, the prior art disclosed in Patent Documents 1 and 2 is not directed to solving the above problems.

[0008] One object is to provide a computer utilization system that supports the work of inspecting an object using a large number of images obtained from the object or the work of summarizing the inspection results in a report.

Means for Solving the Problems

[0009] A system for inspecting an object and creating a report according to one embodiment includes means for communicating with a user terminal, means for preparing a plurality of images representing the whole of the object and different parts of the object respectively, means for displaying a work screen on the user terminal that displays the image requested by the user and a comment input tool, means for receiving the comment input by the user on the work screen and associating and storing the comment with the image displayed on the work screen, means for storing a report form, and means for automatically creating a report according to the report form using the stored comment and the image associated therewith.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Mode for Carrying Out the Invention

[0011] FIG. 1 shows the overall structure of an inspection report support system according to an embodiment.

[0012] As shown in FIG. 1, the inspection report support system 1 has a data processing server 3, and the data processing server 3 can be configured by one or a plurality of server computer machines. The data processing server 3 provides a service to assist one or more users in inspecting one or more objects and summarizing the inspection results in a report. The data processing server 3 can perform data communication with a web browser of an information processing terminal (hereinafter referred to as a user terminal) 7 such as a personal computer or a smartphone used by each user via a web server 5. In FIG. 1, for the sake of convenience, only one user terminal 7 is shown. Hereinafter, the case where one user uses the data processing server 3 will be taken as an example to explain the details of this system.

[0013] The data processing server 3 has processing components such as a processing request receiving unit 11, a data storage / search unit 13, an image registration unit 15, an image combining unit 17, a map / image display unit 19, a comment registration unit 21, a report creation unit 23, a report display unit 25, a report editing unit 27, and a report output unit 29.

[0014] The processing request receiving unit 11 receives various processing requests from the user (for example, various processing requests for performing inspections or various processing requests for creating reports, etc.) and passes the requests to the processing components that perform the requested processing.

[0015] The data storage / search unit 13 stores data such as maps, images, inspection results, and reports of various inspection targets, and reads out the requested map, image, or inspection result data in response to data requests from other processing components and passes it to those processing components.

[0016] The image registration unit 15 receives images of inspection objects from the user terminal 7 (which may include not only general visible light photographic images but also photographic images using non-visible light such as infrared rays and other types of images) and associates those images with the inspection objects and the users and registers them in the data storage / search unit 13. Usually, a large number of images are registered for one inspection target. For example, when a camera mounted on an unmanned aerial vehicle takes pictures of a vast object such as a solar power facility, a large number of high-quality photographic images of various parts of the facility are registered in association with the solar power facility.

[0017] The image combining unit 17 generates various images (hereinafter referred to as inspection images) that the user will observe to inspect the object by combining the multiple images of each registered inspection target. The inspection images are stored in the data storage / search unit 13.

[0018] As inspection images of the object, images with different resolutions at multiple levels from the lowest resolution to the highest resolution may be created. For example, relatively low-resolution images for overlooking the whole of the object, relatively high-resolution images for inspecting individual parts of the object in detail, and images with intermediate resolutions can be created as inspection images. The format of the inspection images and the provision to the user can be performed, for example, in a manner conforming to WMTS (Web Map Tile Service) or WMS (Web Map Service). The inspection images may be ortho-images or mosaic images. The creation of ortho-image inspection images may be made using SfM (Structure from motion).

[0019] In response to a processing request from the user, the map / image display unit 19 sends a map or inspection image of the inspection object to be displayed on the screen of the user terminal 7 to the web server 5. The web server 5 sends a work screen incorporating the map or inspection image to the user terminal 7 and displays it on the screen of its web browser.

[0020] The comment registration unit 21 receives comments etc. (text, options, marks or figures, etc.) input to the user terminal 7 when the user inspects the object, and saves them as inspection result data associated with the object in the data storage / search unit 13.

[0021] In response to a request from the user, the report creation unit 23 automatically creates an inspection report for each inspection object using data such as the map, inspection image, and inspection result of each inspection object saved in the data storage / search unit 13. The automatically created report is associated with the object and saved in the data storage / search unit 13.

[0022] In response to a request from the user, the report display unit 25 sends the automatically created inspection report to the web server 5. The web server 5 sends the report to the user terminal 7 and displays it on the screen of its web browser.

[0023] The report editing department 27 enables the user to freely edit the report displayed on the screen of the user terminal 7. The report edited by the user is associated with the target and stored in the data storage and search department 13.

[0024] The report output department 29 sends the edited report to the Web server 5 in response to a request from the user. The Web server 5 enables the user to download the report to the user terminal 7.

[0025] Figure 2 shows the basic communication flow between the data processing server 3 and the user 9 in the inspection report support system 1.

[0026] In Figure 2, at step 31, the user 9 registers an image of a certain inspection target (usually there are multiple images) with the data processing server 3. At step 33, the data processing server 3 saves the registered image, and at step 35, combines the saved images to create inspection images with different resolutions for the inspection target.

[0027] At step 37, the user requests a desired inspection image (for example, an image for overlooking the whole of the target or an image for observing a specific part of the target in detail) to inspect the inspection target. In response to the request, at step 39, the data processing server 3 selects and reads out the requested inspection image from the saved inspection images, and at step 41, displays the inspection image on the user terminal 7.

[0028] At step 43, the user views and examines the displayed inspection image, makes a necessary judgment about the inspection target, and at step 45, inputs a comment or the like (text describing the judgment result or opinion, options, graphic marks, etc.) according to the judgment result. Then, at step 47, the data processing server 3 stores the examined image and the input comment or the like in association with each other.

[0029] In the inspection process of one inspection target, steps 37 to 47 can be repeated multiple times. For example, the user will first overview the entire target and input comments, etc., and then repeat the operation of viewing the detailed images of each part of the target, inspecting that part, and inputting comments, etc. until the inspection of all parts is completed. Each time this repetitive operation is performed, steps 37 to 47 are carried out.

[0030] After that, in step 49, the user requests a report summarizing the inspection results of the inspection target. In response to this request, in step 51, the data processing server 3 automatically creates a report using the saved inspection results, and in step 53, the report is displayed on the user terminal 7.

[0031] In step 55, the user checks the automatically created report, and in step 57, makes desired edits to the automatically created report and sends the edited report to the data processing server 3. In step 59, the data processing server 3 saves the edited report, and in step 61, outputs the report. In step 63, the user downloads the output report.

[0032] Figure 3 shows the flow of the inspection process (the processes corresponding to steps 39, 41, and 47 in Figure 2) performed by the data processing server 3.

[0033] As shown in Figure 3, when starting the inspection process, the data processing server 3 performs the processes of steps S1, S2, and S3 in parallel. In step S1, it is determined whether an image display request has come from the user and what area of the image, such as the entire inspection target or a specific part, the user has requested to be displayed. In step S2, it is determined whether the user has requested zooming or scrolling of the displayed image. In step S3, it is determined whether the image is of the area selected by the user within the displayed image.

[0034] In step S1, when it is determined that the display of an image of a certain area of a certain inspection target is requested, the data processing server 3, in step S4, prepares an image of the area of the inspection target and causes the work screen incorporating the image to be displayed on the screen of the user terminal 7. For example, when the user requests the display of an image for overlooking the entire inspection target, a work screen as illustrated in FIG. 4 is displayed, and an image 71 showing a wide area including the entire inspection target is incorporated therein. Also, for example, when the user requests an image for examining a specific part of the inspection target in detail, a work screen as illustrated in FIG. 6 is displayed, and an image 91 showing a narrow area of the specific part is incorporated therein.

[0035] Referring again to FIG. 3, when it is determined in step S2 that zooming (enlarging or reducing) or scrolling of the display image is requested, the data processing server 3 performs the requested zooming or scrolling of the display image in step S5. For example, on the work screen illustrated in FIG. 4, the user can zoom in (enlarge) the image 71 showing the entire target to view a part of this target in more detail at a higher resolution. Or, for example, on the work screen illustrated in FIG. 6, the user can scroll the image 91 showing a specific part of the target to view another part of this target.

[0036] Referring again to FIG. 3, when it is determined in step S3 that a certain area within the image displayed on the work screen is selected by the user, the data processing server 3, in step S6, displays the contour frame of the selected area on the image. Further, the data processing server 3, in step S7, displays one or more tools (hereinafter referred to as comment input tools) for the user to input comments such as the area name and inspection results of the selected area on the work screen via the Web server 5. For example, on the work screen shown in FIG. 4, the contour frame 75 of the area selected by the user (in this example, the selected area is the entire site of the solar power generation facility which is the inspection target) is displayed on the image, and comment input tools 76, 77 for inputting the area name and arbitrary comments regarding the selected area are displayed.

[0037] Referring again to FIG. 3, if the user inputs the area name of that area or a comment regarding that area using the displayed comment input tool or the like, the data processing server 3 displays the input area name or comment on the work screen in step S8. For example, on the work screen illustrated in FIG. 4, the input area name is displayed in the comment input tool 76, and the input comment is displayed in the comment display field 79 together with the time stamp at the time of input.

[0038] Referring again to FIG. 3, the data processing server, in step S9, associates and saves the image displayed on the work screen, the position coordinates of the area (or area frame) selected by the user, the area name input by the user, and the comment as data elements constituting the inspection result of the selected area in association with the selected area.

[0039] FIGS. 4 to 7 show some examples of work screens for inspecting a target, which are displayed on the screen of the user terminal 7 by the inspection process shown in FIG. 3. In the examples shown in FIGS. 4 to 7, a certain solar power generation facility is the inspection target.

[0040] FIG. 4 shows an example of a work screen for allowing the user to overlook the entire solar power generation facility that is the inspection target and select the entire area of the site.

[0041] As shown in FIG. 4, the image 71 displayed on the work screen is a relatively low-resolution image showing the entire solar power generation facility. The user can zoom in on the displayed image to display a higher-resolution image and view the details of the solar power generation facility.

[0042] The user can select the entire site of the inspection target, that is, the entire solar power generation facility, on the displayed image 71 using the area selection tool 73 (typically, draw the outline of the entire site). Then, the outline frame 75 of the selected area is displayed on the image 71. Along with that, tools (comment input tools) 76 and 77 for inputting the area name and comment regarding the selected area are displayed on the work screen.

[0043] The user can use the input tool 76 to input the area name of the selected area (for example, "Facility E, C Town, B City, A County"), and can also use the input tool 77 to input free comments such as inspection results and explanations of the selected area. When the user clicks the send buttons of the input tools 76 and 77, the input area name and comments are sent to the data processing server 3 and saved in association with the display image 71, the position coordinates of the selected area (or area frame), etc. In the example of FIG. 4, in addition to the comment input tools 76 and 77, the comments previously input and saved for the selected area are also displayed on the comment display tool 79.

[0044] When the user double-clicks on the contour frame 75 of the inspection target or designates the inspection target by other predetermined methods on the work screen as exemplified in FIG. 4, a work screen for inspecting while examining the state of each part of the inspection target in detail will be displayed on the screen.

[0045] FIG. 5 shows an example display of a work screen that enables setting one or more sub-areas within an arbitrarily selected area that has already been selected.

[0046] The work screen illustrated in FIG. 5 exemplifies a case where a plurality of sub-regions are set within the area of the entire site of the solar power generation facility selected in the screen example of FIG. 4. That is, when the user uses the region selection tool 73 to draw the outline of each sub-region, the drawn outline frames 81A and 81B are displayed on the display image 75. At the same time, comment input tools 83A, 85A, 83B, and 85B for inputting the region names and comments of each sub-region are also displayed on the work screen. When the user uses the input tools 83A and 83B to input the names of the respective regions (for example, "Area A" and "Area B"), the respective region names are displayed on the display image 75 in association with the outline frames 81A and 81B of the respective sub-regions. Also, when the user uses the input tools 85A and 85B to input comments regarding the respective regions, those comments are displayed on the input tools 85A and 85B respectively. In the example of FIG. 5, further, regarding the sub-region named "Area A", comments input in the past are also displayed in the comment display field 87A.

[0047] The user can further add and set new sub-regions. The comment input tools for the added sub-regions are added below the comment input tool 85B at the bottom position shown in FIG. 5. The user can access any comment input tool by scrolling the comment input tool group on the work screen 75.

[0048] When the user double-clicks on the outline frame 81A, 81B of any sub-region or its region name on the work screen as illustrated in FIG. 5, or designates any sub-region by other predetermined methods, a work screen for performing an inspection while examining each part of the designated sub-region in detail will be displayed on the screen.

[0049] FIG. 6 shows an example of a work screen for performing an inspection while examining each part of the inspection target in detail.

[0050] The example shown in FIG. 6 is a work screen for inspecting each part of the sub-region "Area A" shown in FIG. 5. In this work screen, a relatively high-resolution image 91 that details the state of a part of the area within Area A is displayed. Additionally, an auxiliary image 95 indicating with a frame 94 the position within the inspection target corresponding to the area represented by the displayed image 91 is also shown on the work screen.

[0051] When the user selects a point of interest within the display screen 91 using the region selection tool 73 (for example, by drawing a frame around that point), a frame 91 of that point of interest is displayed on the display screen 91. At the same time, a mark 96 indicating the position of that point of interest within the inspection target is shown on the auxiliary image 95. Additionally, tools (comment input tools) 97, 99, 101, 103 for inputting the location number, position information, maintenance urgency, and comments on the state evaluation regarding that point of interest are also presented on the work screen.

[0052] Here, the location number may be automatically set or the user may be able to input any number. The urgency level may, for example, allow selection of standard levels such as "high", "medium", or "low" from a menu. Also, comments on the state evaluation may be such that any of several standard comments can be selected from the pull-down menu 105 as illustrated in FIG. 5, and in addition, free text may be inputtable.

[0053] When the location number (for example, "(1)") of that point of interest is set or input, on the display screen 91, that location number is displayed in association with the frame 93 of that point of interest

[0054] FIG. 7 shows an example display of a work screen for presenting a list of already inspected points of interest within the inspection target and enabling confirmation of the inspection results for each point of interest.

[0055] As shown in FIG. 7, location marks 113 indicating respective points of interest are displayed on an image 71 that allows an overall view of the entire inspection target. When the user designates an arbitrary location mark 113 with the cursor 111, an image 115 showing the detailed state of the corresponding point of interest and the inspection result 117 of that point of interest are displayed on the work screen.

[0056] FIG. 8 shows the flow of a comparison inspection process that can be performed by the data processing server in the inspection process described above.

[0057] The comparison inspection process is a process that enables the user to perform an inspection by arranging and displaying a plurality of different images of the inspection target on the work screen and comparing those images. The plurality of images to be displayed can be, for example, images of the same part of the inspection target taken at different times, images of the same part taken in different ways (e.g., visible light photographic images and infrared photographic images, etc.).

[0058] As shown in FIG. 8, in step S21, the data processing server 3 displays a plurality of images requested by the user in parallel on the work screen.

[0059] Thereafter, in step S22, when a zoom or scroll of any of the displayed images is requested by the user, in step S23, the plurality of images displayed in parallel on the work screen are zoomed or scrolled synchronously by the same zoom ratio or in the same scroll direction and distance.

[0060] Also, in step S24, when the user draws a frame indicating an arbitrary area on any of the displayed images, in step S25, the area frame is displayed on the plurality of images, and in step S26, a tool (such as a comment input tool) for inputting inspection results, etc. regarding that area is displayed on the work screen.

[0061] Also, in step S27, when the user inputs a comment or the like using an input tool for comments or the like, the input comment or the like is displayed on the work screen. Further, in step S28, the input comment or the like, the coordinates of the area (or area frame), and a plurality of images on the work screen are associated with each other and stored as inspection result data.

[0062] In step S27, when the user requests the end of the inspection, the comparison inspection process ends.

[0063] FIG. 9 shows an example of a display of a work screen for the comparison inspection process.

[0064] On the work screen illustrated in FIG. 9, two images 121 and 123 representing the same part of the inspection target are arranged and displayed. For example, the arranged and displayed images may be images taken at different times with the same or different sensors, such as one image 121 being an infrared photographic image and the other image 123 being a visible light photographic image.

[0065] When the user zooms or scrolls one image (for example, image 121), both images 121 and 123 are simultaneously zoomed or scrolled in the same manner.

[0066] Also, when the user uses the area selection tool 73 to draw an area frame on one image (for example, image 121), the area frames 125 and 127 are displayed on both images 121 and 123. In addition, an input tool 129 for comments or the like regarding the area is also displayed.

[0067] FIG. 10 shows the flow of the report creation process performed by the data processing server 3.

[0068] As shown in FIG. 10, in step S31, a previously saved report form (data defining the composition of the report (for example, what title items are arranged in what order, and for each item, what images and text comments are arranged in what layout, etc.)) is read. Here, the report form can be prepared in several different ways. One method is for the user to define the composition of the report by programming. Another method is for the user to send an image or document file of an existing printed report (for example, an image file obtained by scanning the report with an image printer, or a document file of the report created with a word processor, etc.) from the user terminal 7 to the data processing server 3, and the data processing server 3 analyzes the file to automatically create the report form.

[0069] In step 32, among the various items to be reported defined in the report form, uncreated items are extracted. Then, in step S32, for each uncreated item, an image, region coordinates, region name, comment, etc. corresponding to the component are read from the saved inspection result data. And in step S34, based on the read data, an image, region name, region frame, comment, etc. are arranged and described in the corresponding item part of the report according to the layout defined in the report form. Steps S33 and S34 are repeated until there are no uncreated items, and thus the report is created.

[0070] Thereafter, in step S35, the report is saved as a file in a format that the user can edit later.

[0071] FIG. 11 shows a part of an example of a report automatically created by the process shown in FIG. 10.

[0072] As shown in FIG. 11, in the automatically created report, display images used on the work screens as exemplified in FIGS. 4 to 7, area frames on the images, area names, input comments, etc. are described according to the order and layout defined in the report form. The user can complete the report by downloading the file of such an automatically created report to the user terminal 7 and editing it as necessary.

[0073] As described above, one embodiment has been explained. However, the scope of the present invention is not limited only to this embodiment. For example, the area frames, area names, input comments, etc. on the images included in the automatically generated report may be automatically created by artificial intelligence or the like, and can be implemented in various other modes without departing from the gist thereof.

Explanation of Signs

[0074] 1 Inspection Report Support System 3 Data Processing Server 7 User Terminal 11 Processing Request Reception Unit 13 Data Storage and Search Unit 15 Image Registration Unit 17 Image Combining Unit 19 Map / Image Display Unit 21 Comment Registration Unit 23 Report Creation Unit 25 Report Display Unit 27 Report Editing Unit 29 Report Output Unit

Claims

1. In a system for testing objects and generating reports, A means for communicating with a user terminal used by a user; means for preparing a plurality of images each representing the entire object and a different portion of the object; a means for displaying on the user terminal a work screen displaying an image and a comment input tool requested by the user; a means for receiving a comment input by the user on the work screen and storing the comment in association with the image displayed on the work screen; means for automatically generating a report using the stored comments and the associated images; Equipped with when the user selects an area in the image displayed on the work screen, the display means displays the area on the image and also displays the comment input tool for inputting a comment regarding the area; the saving means saves the image displayed on the work screen, the position coordinates of the area, and the comment input using the comment input tool in association with one another; the means for automatically creating the report creates the report in which the image, the area, and the comment are arranged based on the image, the position coordinates, and the comment that are stored in association with each other; A system configured as follows.

2. When the user selects the area in the image displayed on the work screen, the display means further displays, on the work screen, an auxiliary image indicating which part of the object the area corresponds to. The system according to claim 1 , configured as follows.

3. means for storing a report form; the means for automatically creating the report creates the report in which the image, the area, and the comment are arranged according to a layout defined in the report form, based on the stored image, the position coordinates of the area, and the comment.

3. The system according to claim 1 or 2, configured so as to

Images