System for inspecting objects and generating reports

Abstract

To provide a computer-using system that supports the work of inspecting an object using a large number of images obtained from the object, or the work of compiling the test results into a report.SOLUTION: A data processing server 3 combines a large number of images obtained by photographing an object to create a plurality of images representing the entire inspection object and different parts, and displays on a user terminal 7 an inspection work screen that displays an image requested from the user terminal 7 and a comment input tool. A comment entered by a user into the work screen is saved in association with the displayed image. The data processing server 3 automatically creates a report according to a report form using the saved comment and the image associated with it.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a computer-based system for assisting in the examination, inspection or evaluation of an object using images and generating a report describing the results. [Background technology]

[0002] For purposes such as investigating or inspecting man-made facilities or natural objects, the land or facility in question is photographed with a camera mounted on a mobile vehicle in the air or on the ground, or carried by a person, and the resulting numerous images are then viewed by a person who then inspects or evaluates the object, and a report is prepared that presents the results using a combination of the photographed images and accompanying marks and text comments. This work is carried out in a variety of industrial fields.

[0003] Systems for creating reports are disclosed in, for example, Patent Documents 1 and 2. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2005-352848 A [Patent Document 2] JP 2007-34453 A Summary of the Invention [Problem to be solved by the invention]

[0005] For example, when inspecting a vast object such as a solar power plant based on images taken by a camera mounted on an unmanned aerial vehicle, the number of images taken for the entire object is very large. It is not an easy task for a human to look at these many images to get a grasp of the whole object or to inspect each part in detail.

[0006] Furthermore, in order to create a report that allows others to easily understand the inspection results, it is necessary to skillfully arrange many images on the report and add marks and comments to the images. It takes a lot of time to create a report. For example, it is said that it usually takes about one week to inspect a solar power generation facility based on images taken by an unmanned aerial vehicle and create a report.

[0007] However, the conventional techniques disclosed in Patent Documents 1 and 2 are not directed to solving the above problems.

[0008] One object is to provide a computer-based system that assists in the examination of an object using a large number of images obtained from the object, and in the compilation of the results of that examination into a report. [Means for solving the problem]

[0009] According to one embodiment, a system for inspecting an object and generating a report comprises means for communicating with a user terminal, means for preparing a plurality of images each representing the entire object and different parts of the object, means for displaying a work screen on the user terminal displaying an image and a comment input tool requested by a user, means for receiving a comment entered by the user on the work screen and storing the comment in association with the image displayed on the work screen, means for storing a report form, and means for automatically generating a report in accordance with the report form using the stored comment and the image associated therewith. [Brief description of the drawings]

[0010] [Figure 1] 1 shows an overall structure of an examination report support system according to an embodiment. [Diagram 2] 2 shows a basic flow of communication between a data processing server and a user in the system. [Diagram 3] 4 shows the flow of inspection processing performed by the data processing server. [Figure 4]13 shows an example of a screen display that allows a user to get an overview of the entire image of an object to be inspected during inspection processing. [Diagram 5] 1 shows an example screen display for allowing a user to segment an object to be inspected into sub-regions during the inspection process. [Figure 6] 13 shows an example of a screen display for allowing a user to closely inspect a portion of an inspection target during inspection processing. [Figure 7] 13 shows an example of a screen display that allows a user to view points of interest in an inspection process and check the inspection results of each point. [Figure 8] 13 shows a flow of a comparison inspection process performed by the data processing server. [Figure 9] 1 shows an example of a screen display for allowing a user to perform an inspection by comparatively observing two images of an object during inspection processing. [Figure 10] 4 shows the flow of a report creation process performed by a data processing server. [Figure 11] 1 shows a portion of an example report automatically generated by the data processing server. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] FIG. 1 shows the overall structure of an examination report support system according to one embodiment.

[0012] As shown in FIG. 1, the inspection report support system 1 has a data processing server 3, which may be composed of one or more server computer machines. The data processing server 3 provides a service to one or more users to assist in the work of inspecting one or more objects and compiling the inspection results into a report. The data processing server 3 can perform data communication with a web browser of an information processing terminal (hereinafter, a user terminal) 7 such as a personal computer or a smartphone used by each user via a WEB server 5. For convenience, only one user terminal 7 is illustrated in FIG. 1, and the following describes the details of this system by taking the case where one user uses the data processing server 3 as an example.

[0013] The data processing server 3 has processing components such as a processing request receiving unit 11, a data storage and search unit 13, an image registration unit 15, an image combination unit 17, a map and image display unit 19, a comment registration unit 21, a report creation unit 23, a report display unit 25, a report editing unit 27, and a report output unit 29.

[0014] The processing request receiving unit 11 receives various processing requests from users (for example, various processing requests for performing inspections, or various processing requests for creating reports, etc.) and passes the requests to a processing component that performs the requested processing.

[0015] The data storage and retrieval unit 13 stores data such as maps, images, inspection results, and reports of various inspection subjects, and in response to a data request from another processing component, reads the requested map, image, or inspection result data and passes it to that processing component.

[0016] The image registration unit 15 receives images of the object of inspection from the user terminal 7 (which may include not only general photo images in visible light, but also photo images using invisible light such as infrared light, and other types of images), and registers these images in the data storage and retrieval unit 13 in association with the object of inspection and its user. Typically, a large number of images are registered for one object of inspection. For example, when a photo of a vast object such as a solar power generation facility is taken with a camera mounted on an unmanned aerial vehicle, a large number of high-quality photo images each capturing various parts of the facility are registered in association with the solar power generation facility.

[0017] The image combining unit 17 combines a number of images of each registered inspection object to generate various images (hereinafter referred to as inspection images) that the user will observe to inspect the object. The inspection images are stored in the data storage and retrieval unit 13.

[0018] As the inspection image of the object, images with different resolutions from the lowest resolution to the highest resolution may be created. For example, a relatively low-resolution image for overlooking the entire object, a relatively high-resolution image for inspecting each part of the object in detail, an image with intermediate resolution, etc. may be created as the inspection image. The format of the inspection image and its provision to the user may be performed, for example, according to a method conforming to WMTS (Web Map Tile Service) or WMS (Web Map Service). The inspection image may be an orthoimage or a mosaic image. The inspection image of the orthoimage may be created using SfM (Structure from motion).

[0019] In response to a processing request from a user, the map / image display unit 19 sends a map or an inspection image of the object to be inspected to be displayed on the screen of the user terminal 7 to the Web server 5. The Web server 5 sends a work screen incorporating the map or inspection image to the user terminal 7 and displays it on the screen of the web browser.

[0020] The comment registration unit 21 receives comments (such as text, options, marks or figures) inputted into the user terminal 7 by the user when inspecting an object, and stores them in the data storage and retrieval unit 13 as inspection result data associated with the object.

[0021] In response to a request from a user, the report creation unit 23 automatically creates an inspection report for each inspection object using data such as a map, inspection image, and inspection results of each inspection object stored in the data storage and retrieval unit 13. The automatically created report is associated with the object and stored in the data storage and retrieval unit 13.

[0022] In response to a request from a user, the report display unit 25 sends the automatically created inspection report to the Web server 5. The Web server 5 sends the report to the user terminal 7, where it is displayed on the screen of the Web browser.

[0023] The report editing unit 27 enables the user to freely edit the report displayed on the screen of the user terminal 7. The report edited by the user is associated with its subject and stored in the data storage and retrieval unit 13.

[0024] In response to a request from the user, the report output unit 29 sends the edited report to the Web server 5. The Web server 5 enables the user to download the report to the user terminal 7.

[0025] FIG. 2 shows a basic flow of communication between the data processing server 3 and the user 9 in the examination report support system 1.

[0026] 2, in step 31, a user 9 registers an image (usually many images) of an inspection object in the data processing server 3. In step 33, the data processing server 3 stores the registered images, and in step 35 combines the stored images to create inspection images of the inspection object with different resolutions.

[0027] In step 37, the user requests a desired inspection image (e.g., an image for overlooking the entire object, or an image for observing a specific part of the object in detail) to inspect the object. In response to the request, in step 39, the data processing server 3 selects and reads the requested inspection image from among the stored inspection images, and in step 41 causes the user terminal 7 to display the inspection image.

[0028] In step 43, the user views and examines the displayed inspection image, makes a necessary judgment about the inspection object, and then in step 45, inputs comments, etc. according to the judgment result (text, options, graphic marks, etc. that describe the judgment result or opinion). Then, in step 47, the data processing server 3 stores the examined image and the input comments, etc. in association with each other.

[0029] In the inspection process of one inspection object, steps 37 to 47 may be repeated multiple times. For example, a user may first look at the entire object and input comments, etc., and then look at detailed images of each part of the object, inspect the part, and input comments, etc., repeating this process until all parts have been inspected. Steps 37 to 47 are performed each time the process is repeated.

[0030] Thereafter, in step 49, the user requests a report summarizing the test results for that test subject. In response to the request, in step 51, the data processing server 3 automatically creates a report using the stored test results, and in step 53, displays the report on the user terminal 7.

[0031] In step 55, the user reviews the automatically generated report and, in step 57, makes any desired edits to the automatically generated report and sends the edited report to the data processing server 3. In step 59, the data processing server 3 stores the edited report and, in step 61, outputs the report. In step 63, the user downloads the output report.

[0032] FIG. 3 shows the flow of the inspection process (processing corresponding to steps 39, 41 and 47 in FIG. 2) performed by the data processing server 3.

[0033] As shown in Fig. 3, when the data processing server 3 starts the inspection process, it performs the processes of steps S1, S2, and S3 in parallel. In step S1, it is determined whether an image display request has been made by the user, and what area of ​​the image the user has requested to display, such as the entire inspection object or a specific part, is determined. In step S2, it is determined whether the user has requested to zoom or scroll the displayed image. In step S3, it is determined whether the image is of an area selected by the user within the displayed image.

[0034] When it is determined in step S1 that a request has been made to display an image of a certain area of ​​an inspection target, the data processing server 3 in step S4 prepares an image of the area of ​​the inspection target and displays a work screen incorporating the image on the screen of the user terminal 7. For example, if the user requests a display of an image for an overview of the entire inspection target, a work screen such as that shown in Fig. 4 is displayed, incorporating an image 71 showing a wide area including the entire inspection target. Alternatively, if the user requests an image for examining a specific part of the inspection target in detail, a work screen such as that shown in Fig. 6 is displayed, incorporating an image 91 showing a narrow area of ​​the specific part.

[0035] 3 again, if it is determined in step S2 that zooming (enlarging or reducing) or scrolling of the display image is requested, the data processing server 3 performs the requested zooming or scrolling of the display image in step S5. For example, in the work screen illustrated in FIG. 4, the user can zoom in (enlarge) an image 71 showing an entire object to view a portion of the object in greater detail and at a higher resolution. Or, for example, in the work screen illustrated in FIG. 6, the user can scroll an image 91 showing a specific portion of the object to view another portion of the object.

[0036] 3 again, if it is determined in step S3 that a certain area in the image displayed on the work screen has been selected by the user, the data processing server 3 displays a contour frame of the selected area on the image in step S6. Furthermore, in step S7, the data processing server 3 displays one or more tools (hereinafter referred to as comment input tools) on the work screen via the Web server 5 for the user to input comments such as the area name and inspection results of the selected area. For example, on the work screen shown in FIG. 4, a contour frame 75 of the area selected by the user (in this example, the selected area is the entire site of the solar power generation facility to be inspected) is displayed on the image, and comment input tools 76, 77 for inputting the area name and any comments regarding the selected area are also displayed.

[0037] 3 again, when the user inputs the area name of the area or a comment related to the area using the displayed comment etc. input tool, the data processing server 3 displays the input area name or comment on the work screen in step S8. For example, in the work screen illustrated in Fig. 4, the input area name is displayed in the comment etc. input tool 76, and the input comment is displayed in the comment display field 79 together with the timestamp of the input.

[0038] Referring again to FIG. 3, in step S9, the data processing server stores the image displayed on the work screen, the position coordinates of the area (or area frame) selected by the user, and the area name and comments entered by the user, in association with the selected area, as data elements constituting the inspection results for that selected area.

[0039] Figures 4 to 7 show some examples of work screens for inspecting an object, which are displayed on the screen of the user terminal 7 by the inspection process shown in Figure 3. In the examples shown in Figures 4 to 7, a certain photovoltaic power generation facility is the inspection object.

[0040] FIG. 4 shows an example of a work screen that allows the user to obtain an overview of the entire photovoltaic power generation facility to be inspected and select the entire area of ​​the site.

[0041] 4, the image 71 displayed on the work screen is a relatively low-resolution image showing the entire solar power generation facility. The user can zoom in on the displayed image to display a higher resolution image and view the solar power generation facility in detail.

[0042] The user can use an area selection tool 73 on the display image 71 to select the inspection target, that is, the entire site of the photovoltaic power generation facility (typically by drawing the outline of the entire site). Then, an outline frame 75 of the selected area is displayed on the image 71. At the same time, tools 76 and 77 for inputting an area name and a comment about the selected area (comment input tools) are displayed on the work screen.

[0043] The user can use input tool 76 to input the name of the selected area (for example, "A Prefecture, B City, C Town, E Facility"), and can use input tool 77 to input free comments such as the inspection results and explanations of the selected area. When the user clicks the send buttons of the input tools 76 and 77, the input area name and comment are sent to the data processing server 3 and stored in association with the display image 71 and the position coordinates of the selected area (or area frame). In the example of FIG. 4, in addition to the comment input tools 76 and 77, the comments previously input and stored about the selected area are also displayed in the comment display tool 79.

[0044] When the user double-clicks on the outline 75 of the object to be inspected on a work screen such as that shown in FIG. 4, or specifies the object to be inspected by any other predetermined method, a work screen is displayed on the screen for inspecting the object while examining in detail the condition of each part of the object.

[0045] FIG. 5 shows an example of a work screen display that allows one to set one or more sub-regions within any region already selected.

[0046] The work screen illustrated in FIG. 5 illustrates a case where multiple sub-areas are set within the area of ​​the entire site of the photovoltaic power generation facility selected in the screen example of FIG. 4. That is, when the user uses the area selection tool 73 to draw the outline of each sub-area, the drawn outline frames 81A, 81B are displayed on the display image 75. At the same time, comment input tools 83A, 85A, 83B, 85B for inputting the area name and comment of each sub-area are also displayed on the work screen. When the user inputs the name of each area (for example, "Area A" and "Area B") using the input tools 83A, 83B, each area name is displayed in association with the outline frames 81A, 81B of each sub-area on the display image 75. In addition, when the user inputs comments regarding each area using the input tools 85A, 85B, the comments are displayed in the input tools 85A, 85B, respectively. In the example of FIG. 5, furthermore, comments previously input regarding the sub-area named "Area A" are also displayed in the comment display field 87A.

[0047] The user can further set a new sub-area. The comment, etc. input tools of the added sub-area are added below the comment, etc. input tool 85B at the bottom position shown in Fig. 5. The user can access any comment, etc. input tool by scrolling through the comment, etc. input tools on the work screen 75.

[0048] When the user double-clicks on the outline 81A, 81B or name of any sub-area on a work screen such as that shown in FIG. 5, or specifies any sub-area in any other predetermined manner, a work screen is displayed on the screen for inspecting each part of the specified sub-area in detail.

[0049] FIG. 6 shows an example of a work screen for performing an inspection while examining each part of the inspection target in detail.

[0050] The example shown in Fig. 6 is a work screen for inspecting in detail each part of the sub-area "Area A" shown in Fig. 5. This work screen displays a relatively high-resolution image 91 showing in detail the condition of a part of Area A. In addition, an auxiliary image 95 is also displayed on the work screen, showing with a frame 94 which part of the object to be inspected corresponds to the area shown by the display image 91.

[0051] When the user uses the area selection tool 73 to select a location of interest on the display screen 91 (for example, by drawing a frame around the location), a frame 91 of the location of interest is displayed on the display screen 91. At the same time, a mark 96 indicating where the location of the location of interest in the inspection object is located is displayed on the auxiliary image 95. Furthermore, at the same time, tools 97, 99, 101, 103 for inputting the location number, position information, urgency of maintenance, and comments on the state evaluation related to the location of interest (comment input tools) are also displayed on the work screen.

[0052] Here, the location number may be set automatically, or the user may be able to input any number. The urgency level may be selected from a menu of standard levels such as "high", "medium" or "low". The status evaluation comment may be selected from several standard comments from a pull-down menu 105 as shown in Fig. 5, and free text may be entered.

[0053] When the location number of the attention location (for example, "(1)") is set or input, the location number is displayed on the display screen 91 in association with the frame 93 of the attention location.

[0054] FIG. 7 shows an example of a work screen display that displays a list of already inspected locations of interest within an inspection object and enables the inspection results of each location of interest to be confirmed.

[0055] 7, location marks 113 indicating respective locations of interest are displayed on an image 71 which provides an overview of the entire inspection target. When a user designates an arbitrary location mark 113 with a cursor 111, an image 115 showing a detailed state of the corresponding location of interest and an inspection result 117 of the location of interest are displayed on the work screen.

[0056] FIG. 8 shows the flow of a comparison inspection process that can be performed by the data processing server in the above-mentioned inspection process.

[0057] The comparison inspection process is a process that displays multiple images of different inspection objects side by side on the work screen, allowing the user to compare and examine the images to perform an inspection. The multiple images displayed can be, for example, images of the same part of the inspection object taken at different times, or images of the same part taken by different methods (for example, a visible light photographic image and an infrared photographic image, etc.).

[0058] As shown in FIG. 8, in step S21, the data processing server 3 displays a plurality of images requested by the user in parallel on the work screen.

[0059] Thereafter, in step S22, when the user requests zooming or scrolling of any of the displayed images, in step S23, the multiple images displayed in parallel on the work screen are synchronously zoomed or scrolled to the same zoom factor or scroll direction and distance.

[0060] Furthermore, in step S24, when the user draws a frame indicating an arbitrary area on any of the displayed images, in step S25, the frame of that area is displayed on the multiple images, and in step S26, a tool for inputting inspection results, etc. regarding that area (a tool for inputting comments, etc.) is displayed on the work screen.

[0061] Furthermore, in step S27, when the user inputs comments, etc. using the comment input tool, the input comments, etc. are displayed on the work screen, and in step S28, the input comments, etc., the coordinates of the area (or area frame), and multiple images on the work screen are associated with each other and saved as inspection result data.

[0062] In step S27, when the user requests the end of the inspection, the comparison inspection process ends.

[0063] FIG. 9 shows an example of a display of a work screen for a comparison inspection process.

[0064] Two images 121, 123 showing the same part of an inspection target are displayed side by side on the work screen exemplified in Fig. 9. For example, the images displayed side by side may be images taken at different times by the same or different sensors, such as one image 121 being an infrared photographic image and the other image 123 being a visible light photographic image.

[0065] When a user zooms or scrolls one image (eg image 121), both images 121, 123 are simultaneously zoomed or scrolled in the same way.

[0066] Furthermore, when the user uses the area selection tool 73 to draw an area frame on one of the images (for example, image 121), the area frames 125, 127 are displayed on both images 121, 123. In addition, a tool 129 for inputting comments, etc. regarding the area is also displayed.

[0067] FIG. 10 shows the flow of the report creation process performed by the data processing server 3.

[0068] As shown in FIG. 10, in step S31, a previously saved report form (data defining the report structure (e.g., what title items are arranged in what order, and what images and text comments are arranged in each item, and in what layout, etc.)) is read out. Here, the report form can be prepared in several different ways. One way is for the user to define the report structure by programming. Another way is for the user to send an image or document file of an existing printed report (e.g., an image file obtained by scanning the report with an image printer, or a document file of the report created with a word processor, etc.) from the user terminal 7 to the data processing server 3, and the data processing server 3 analyzes the file and automatically creates the report form.

[0069] In step S32, uncreated items are extracted from the items to be reported defined in the report form. Then, in step S32, images, area coordinates, area names, comments, etc. corresponding to the components of each uncreated item are read from the stored inspection result data, and in step S34, based on the read data, the images, area names, area frames, comments, etc. are arranged and written in the corresponding item part of the report according to the layout defined in the report form. Steps S33 and S34 are repeated until there are no more uncreated items, and the report is created.

[0070] Then, in step S35, the report is saved as a file in a format that can be later edited by the user.

[0071] FIG. 11 shows a portion of an example report automatically generated by the process shown in FIG.

[0072] As shown in Fig. 11, the automatically created report describes the display image used on the work screen as exemplified in Fig. 4 to Fig. 7, the area frame on the image, the area name, the input comment, etc., in the order and layout defined in the report form. The user can complete the report by downloading such an automatically created report file to the user terminal 7 and editing it as necessary.

[0073] One embodiment has been described above, however, the scope of the present invention is not limited to this embodiment. For example, the area frames, area names, input comments, etc. on the images included in the automatically generated report may be automatically created by artificial intelligence, etc., and the present invention can be embodied in various other forms without departing from the spirit of the present invention. [Explanation of symbols]

[0074] 1. Inspection report support system 3 Data Processing Server 7 User terminal 11 Processing request reception unit 13 Data storage and search section 15 Image Registration Section 17 Image Combination Section 19 Map / Image Display Section 21 Comment Registration Section 23 Report Writing Department 25 Report display section 27 Report Editorial Department 29 Report Output Section

Claims

1. In a system for testing objects and generating reports, means for communicating with a user terminal used by a user; means for providing a plurality of images representing the object; a means for displaying on the user terminal a work screen displaying an image and a comment input tool requested by the user; a means for receiving a comment input by the user on the work screen and storing the comment in association with the image displayed on the work screen; means for automatically generating a report using the stored comments and the associated images; Equipped with when the user draws an area on any one of the plurality of images displayed on the work screen, the display means displays the area on each of the plurality of images at a position corresponding to the area, and displays the comment input tool for inputting a comment regarding the area; the saving means saves the images displayed on the work screen, the position coordinates of the areas, and the comments inputted by the comment input tool in association with one another; the means for automatically creating the report creates the report in which the images, the areas, and the comments are arranged, based on the plurality of images, the position coordinates, and the comments that are stored in association with each other; A system configured as follows.

2. A system for testing an object and generating a report, comprising: means for communicating with a user terminal used by a user; means for providing a plurality of images representing the object; a means for displaying on the user terminal a work screen displaying an image and a comment input tool requested by the user; a means for receiving a comment input by the user on the work screen and storing the comment in association with the image displayed on the work screen; means for automatically generating a report using the stored comments and the associated images; Equipped with when the user selects an area in the image displayed on the work screen, the display means displays the area on the image, and displays the comment input tool for inputting a comment regarding the area, and further displays an auxiliary image on the work screen indicating which part of the object the area corresponds to; the saving means saves the image displayed on the work screen, the position coordinates of the area, and the comment input using the comment input tool in association with one another; the means for automatically creating the report creates the report in which the image, the area, and the comment are arranged based on the image, the position coordinates, and the comment that are stored in association with each other; A system configured as follows.

3. means for storing a report form; the means for automatically creating the report creates the report in which the image, the area, and the comment are arranged according to a layout defined in the report form, based on the stored image, the position coordinates of the area, and the comment.

3. The system according to claim 1 or 2, configured so as to

Images