Inspection support system, inspection support method, and program

Abstract

To provide a mechanism for assisting in determining whether an inspection target requires repair.SOLUTION: An inspection support system comprises: image information acquisition means for acquiring information related to inspection videos capturing an inspection target; support personnel information provision means for providing information about one or more candidate inspection support personnel, who determine whether the inspection target requires repair, to a first terminal device used by a support requester; selection acceptance means for accepting any of selections from the first terminal device of one or more of the candidate inspection support personnel; image information provision means for providing information related to the inspection video to a second terminal device used by the selected inspection support personnel; and additional information management means for accepting, from the second terminal device, additional information related to repairs added by the selected inspection support personnel regarding the inspection target, and recording the same in association with the inspection video.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to an inspection support system, an inspection support method, and a program. [Background technology]

[0002] Japanese Patent Laid-Open Publication No. 2020-098568 (Patent Document 1) is a document disclosing background technology in this technical field. This document states, "Furthermore, at building and construction sites, inspection and maintenance sites for facility management, it is necessary to record not only information about the target object itself, but also information about the surrounding environment, whether related to the construction or work or not, and there has been a demand for a more flexible storage method using a variety of data formats, such as image data and document data. According to this embodiment, by visually viewing a live view video LV of a space including the target object and performing a touch operation (e.g., tapping) on ​​the position of the target object, a mark image M can be displayed superimposed on the live view video LV in real time. Furthermore, as long as the same information management program 20 is installed on a different device (information management device 10), the mark image M will be reproduced when the live view video LV is displayed in the same space" (see paragraphs

[0314] to

[0315] ). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-098568 Summary of the Invention [Problem to be solved by the invention]

[0004] The above-mentioned Patent Document 1 describes that, in relation to inspection and maintenance at building and construction sites and in facility management, a mark image M is displayed by touching the position of an object OB while a live view video LV is being displayed. However, this document does not describe a mechanism for supporting the determination of whether or not repairs are required for an inspection object. The present invention has been made in view of the above circumstances, and provides a mechanism for supporting a decision on whether or not an inspection target needs repair. [Means for solving the problem]

[0005] In order to solve the above problems, for example, the configurations described in the claims are adopted. The present application includes multiple means for solving the above-mentioned problems, and one example is an inspection support system having an image information acquisition means for acquiring information regarding an inspection video of an inspection object; an assistant information provision means for providing information regarding one or more candidate inspection assistants who will determine whether or not repairs are necessary for the inspection object to a first terminal device used by a person requesting support; a selection acceptance means for accepting a selection from the first terminal device of one or more candidate inspection assistants; an image information provision means for providing information regarding the inspection video to a second terminal device used by the selected inspection assistant; and an additional information management means for accepting additional information regarding repairs added to the inspection object by the selected inspection assistant from the second terminal device, and recording it in association with the inspection video. [Effects of the Invention]

[0006] According to the present invention, it is possible to provide a mechanism for supporting the determination of whether or not an inspection target needs repair. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 shows an example of the configuration of an inspection support system 100. [Figure 2] FIG. 2 shows an example of the configuration of the inspection support server 102. [Figure 3] FIG. 3 shows an example of the configuration of the inspector terminal 103. [Figure 4] FIG. 4 shows an example of the configuration of the supporter terminal 104. [Figure 5]FIG. 5 shows an example of the configuration of the user terminal 105. [Figure 6] FIG. 6 shows an example of a video related information table 600. [Figure 7] FIG. 7 shows an example of the additional information management table 700. As shown in FIG. [Figure 8] FIG. 8 shows an example of the supporter information table 800. [Figure 9] FIG. 9 shows an example of a pin information table 900. [Figure 10] FIG. 10 shows an example of a processing flow during inspection. [Figure 11] FIG. 11 shows an example of an inspection support sequence. [Figure 12] FIG. 12 shows an example of a flow for determining whether or not repair is necessary. [Figure 13] FIG. 13 shows an example of an inspection support sequence. [Figure 14] FIG. 14 shows an example of an inspection support sequence. [Figure 15] FIG. 15 shows an example of a map display flow. [Figure 16] FIG. 16 shows an example of a repair target extraction flow. [Figure 17] FIG. 17 shows an example of the supporter selection screen. [Figure 18] FIG. 18 shows an example of the photographing screen. [Figure 19] FIG. 19 shows an example of a display screen. [Figure 20] FIG. 20 shows an example of a display screen. [Figure 21] FIG. 21 shows an example of a map screen. [Figure 22] FIG. 22 shows an example of a map screen. DETAILED DESCRIPTION OF THE INVENTION

[0008] 1. Example Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Overview First, an outline of this embodiment will be described. When a structure inspection is carried out and an inspection report is prepared, simple two-dimensional drawings and photos of the damage (close-up and distant views) are included in the inspection report.Then, based on these photos of the damage, the soundness of the structure and whether repair is necessary are judged.For example, if it is determined that repair is necessary, a repair design is carried out for the structure based on the inspection report.

[0009] When inspecting and assessing structures, their integrity is usually determined based on close-up photographs included in the inspection report. However, because the inspection covers only the overall structural shape of the structure, repair design may be carried out even in areas that do not require repair. There are also cases where repair design and construction are carried out in areas and areas that do not require repair in the first place. The current situation is that such over-spec repair work is depleting the financial resources of each local government. The causes of these issues include a lack of people within the local government who can properly diagnose damage, and the inexperience of local companies (consultants, repair construction companies) in repair techniques. Furthermore, in emergencies, it takes time for experts or experienced personnel to provide on-site guidance during inspection work, making it difficult to make an immediate judgment.

[0010] Therefore, in this embodiment, in order to confirm the status of damaged areas of a structure, structure data such as image data and video data of the entire structure is stored in the cloud. By attaching photos of the damaged areas (images and videos of the damaged areas) to the structure data, the structural shape of the structure can be grasped even from a location far from the structure, making it possible to determine whether or not the structure needs repair. In addition, by immersing multiple people in the cloud, it becomes possible to revise inspection results, discuss repair design policies (scope of countermeasures, timing of repair work), and points to note during repair work.

[0011] Specifically, in this embodiment, the following can be implemented. (1) Video data and photos of damage with location information are stored on the cloud to grasp the entire structure. (2) During inspections at the inspection site, data (videos, photos, etc.) with location information attached is recorded as inspection information. (3) The accumulated image and video data is reflected in virtual reality (VR), and specialist engineers provide remote support for the image and video data to determine whether repairs are necessary and to inspect the equipment. (4) The remote supporter simultaneously views image data and video data and makes voice calls. (5) The remote support staff will reassess the inspection results, and reach a consensus on repair policies and whether or not the problem is urgently needed. (6) Remote supporters (specialized technicians) are registered on the system, and the person receiving remote support (including the client, photographer, etc.) selects a specialized technician. (7) The system uses an omnidirectional camera to broadcast and share the entire site in real time, allowing specialized technicians to provide immediate remote support. (8) The video data stored in the cloud makes it possible to compare video data and image data from different shooting times on the system, and the progression of damage to structures can be confirmed through image analysis, etc.

[0012] This invention enables prompt response to the inspection and repair of structures, including appropriate diagnosis, repair design, and repair. Furthermore, it ensures the safety of structures, enabling local governments to operate at appropriate maintenance costs. Furthermore, when serious damage is discovered during a disaster, for example, specialized engineers (including experienced engineers) can immerse themselves in the digital space in real time, enabling immediate response. In addition, real-time video streaming data shot on-site or accumulated video data can be shared among various stakeholders without the need to gather on-site, and stakeholders can compare data shot at different times on the system.

[0013] 1-2.Configuration FIG. 1 shows an example of the configuration of an inspection support system 100. The inspection support system 100 shown in the figure includes an inspection support server 102, an inspector terminal 103, a supporter terminal 104, and a user terminal 105, which are interconnected via a network 101 so that they can communicate with each other.

[0014] Of these components, the inspection support server 102 is a server for supporting the inspection of structures. The inspector terminal 103 is a terminal used by an inspector who inspects a structure. The supporter terminal 104 is a terminal used by a supporter who supports the inspection of a structure. The supporter here is an expert in the inspection of the structure. The user terminal 105 is a terminal used by a client who requests an inspection from a supporter.

[0015] Each device constituting the inspection support system 100 described above may be, for example, a portable terminal (mobile terminal) such as a smartphone, tablet, mobile phone, or personal digital assistant (PDA), or may be a wearable terminal such as glasses, a wristwatch, or clothing. Each device may also be a stationary or portable computer, or a server located on the cloud or a network. Each device may also function as a VR (Virtual Reality) terminal, an AR (Augmented Reality) terminal, or an MR (Mixed Reality) terminal. Alternatively, a combination of multiple of these terminals may be used. For example, a combination of one smartphone and one wearable terminal may logically function as a single terminal. Other information processing terminals may also be used.

[0016] Each device constituting the inspection support system 100 includes a processor that executes an operating system, applications, programs, etc., a main storage device such as a RAM (Random Access Memory), an auxiliary storage device such as an IC card, a hard disk drive, an SSD (Solid State Drive), or a flash memory, a communication control unit such as a network card, a wireless communication module, or a mobile communication module, an input device such as a touch panel, a keyboard, a mouse, voice input, or input based on motion detection captured by a camera unit, and an output device such as a monitor or display. The output device may be a device or terminal that transmits information to be output to an external monitor, display, printer, or device.

[0017] The main memory stores various programs, applications, etc. (modules), and the processor executes these programs and applications to realize the various functional elements of the overall system. These modules may be implemented in hardware, such as by integration. Each module may be an independent program or application, or may be implemented as a subprogram or function within a single integrated program or application.

[0018] In this specification, each module is described as a subject that performs processing, but in reality, the processing is performed by a processor that processes various programs, applications, etc. (modules). Various databases (DBs) are stored in the auxiliary storage device. A "database" is a functional element (storage unit) that stores a set of data so that it can accommodate any data manipulation (e.g., extraction, addition, deletion, overwriting, etc.) from a processor or an external computer. There are no limitations on how the database is implemented; for example, it can be a database management system, spreadsheet software, or a text file such as XML or JSON. Each device will be described in detail below.

[0019] 1-2-1. Inspection support server 102 2 shows an example of the configuration of the inspection support server 102. The inspection support server 102 is configured, for example, by a server placed on the cloud.

[0020] The inspection support server 102 includes a main memory device 201 , an auxiliary memory device 202 , a processor 203 , an input device 204 , an output device 205 , and a transmission / reception unit 206 .

[0021] Of these, the main memory device 201 stores programs and applications such as an image information acquisition module 211, a support request module 212, a video distribution module 213, an additional information management module 214, a repair determination module 215, a map display module 216, and a repair target extraction module 217, and these programs and applications are executed by the processor 203 to realize each functional element of the inspection support server 102. Each module will be described below.

[0022] The image information acquisition module 211 acquires information about an inspection video of an inspection target. The acquired information is, for example, inspection video data of the inspection target distributed in real time by an inspector. The inspection target is, for example, a structure such as a bridge, a tunnel, etc. The inspection video may be a video captured by an omnidirectional camera (in other words, a 360-degree camera).

[0023] The support request module 212 requests an inspection supporter to provide inspection support. At that time, the module provides information on one or more candidate inspection supporters who will determine whether or not repairs are required for the inspection target to the user terminal 105 used by the support requester. The module then accepts a selection from one or more candidate inspection supporters from the user terminal 105. The module then provides information on the inspection video to the supporter terminal 104 used by the selected inspection supporter. The provided information is, for example, the URL of the inspection video.

[0024] The video distribution module 213 provides information about the inspection video to the supporter terminal 104 used by the inspection supporter selected by the support requester. The provided information is, for example, inspection video data.

[0025] At that time, the module can provide the supporter terminal 104 with the inspection video in real time. The module can also provide the supporter terminal 104 with information about inspection videos of the inspection target that were taken earlier than the inspection video, in order to display them side by side with the inspection video. Note that the information about the inspection video here is, for example, inspection video data.

[0026] The additional information management module 214 receives additional information about repairs added by the selected inspection assistant for the inspection target from the assistant terminal 104 and records it in association with the inspection video. The recorded additional information is, for example, pin information indicating the degree of damage with color, or text or audio comments. This additional information is recorded in association with the position information of the inspection target (for example, coordinate values ​​in a three-dimensional coordinate system).

[0027] The module can receive and record additional information added to the inspection video by the selected inspection assistant while the inspection video is being played back. The module can also record audio information input by the selected inspection assistant while the inspection video is being streamed in real time as additional information.

[0028] The repair determination module 215 determines the repair priority of the inspection target based on the additional information added to the inspection video. The determined repair priority may be expressed in two stages, "repair required" and "repair not required," or may be expressed in three or more stages. In this embodiment, it is expressed in two stages, "repair required" and "repair not required."

[0029] The map display module 216 provides information about the map on which the additional information is plotted in response to a request from a user of the user terminal 105. The provided information is, for example, data on a map screen on which the additional information is plotted. This module can receive input of the repair budget amount through the map screen. The input repair budget amount is referenced by the repair target extraction module 217. The map on which the additional information is plotted is not limited to a two-dimensional map, but may be a three-dimensional map.

[0030] The repair target extraction module 217 extracts inspection targets that can be repaired within the input repair budget from among the inspection targets to which additional information has been added by the inspection supporter. At this time, this module extracts inspection targets based on the repair costs determined according to the type of additional information. The additional information added to the extracted inspection targets is plotted on a map generated by the map display module 216.

[0031] Next, the auxiliary storage device 202 will be described. The auxiliary storage device 202 stores an inspection data group 221, a video related information table 600, an additional information management table 700, a supporter information table 800, a pin information table 900, an application 222, etc. Each piece of information will be explained below.

[0032] The inspection data group 221 is a collection of inspection data generated at the inspector terminal 103 during inspection. Each piece of inspection data includes an inspection video, position information, and screen capture data.

[0033] The video-related information table 600 is a table that stores information related to inspection videos. Fig. 6 shows an example of this video-related information table 600. The table shown in the figure is made up of records each consisting of a video ID, a video title (in other words, the name of the structure), inspection target information, a repair flag, a shooting date and time, and a shooting user ID.

[0034] Of the information that makes up each record, the video title indicates the name of the inspection target, the inspection target information indicates the attributes of the structure that is the inspection target, and the repair flag indicates whether repair is required.

[0035] The additional information management table 700 is a table for storing additional information added to an inspection target. This additional information management table 700 is created for each inspection video and is associated with the ID of the inspection video.

[0036] 7 shows an example of an additional information management table 700. The table shown in the figure is made up of records each including an additional ID, color, type, additional coordinates, additional time, map coordinates, and comment content.

[0037] Of the information that makes up each record, the additional ID is identification information for the additional information. The color is the color of the pin information (indicating the degree of damage). The type is the type of additional information. The additional coordinates are the coordinates on the image to which the additional information is added. The addition time is the time on the video to which the additional information is added (elapsed playback time). The map coordinates are the geographical location coordinates of the object captured in the image to which the additional information is added. The comment content is the content of the comment added as additional information.

[0038] The supporter information table 800 is a table that stores attribute information of inspection supporter candidates. Fig. 8 shows an example of this supporter information table 800. The table 800 shown in the figure is made up of records each including a supporter ID, a supporter name, attribute information, contact information, and field of expertise.

[0039] The pin information table 900 is a table that stores information related to pin information. Fig. 9 shows an example of this pin information table 900. The table 900 shown in the figure is made up of records each including color information, repair points, and repair costs.

[0040] Of the information that makes up each record, color information indicates the color of the pin information. This color indicates the degree of damage to the inspection object. Repair score is a score that indicates the degree of necessity for repairing the inspection object. Repair cost is the estimated cost required to repair the inspection object.

[0041] The application 222 is a variety of applications executed by the processor 203 .

[0042] 1-2-2. Inspector terminal 103 Next, the configuration of the inspector terminal 103 will be described. Fig. 3 shows an example of the configuration of the inspector terminal 103. The inspector terminal 103 is a terminal used by an inspector at an inspection site. This inspector terminal 103 is, for example, a portable computer with a camera, a tablet terminal, or a head-mounted display equipped with an omnidirectional camera.

[0043] The inspector terminal 103 includes a main memory device 301 , an auxiliary memory device 302 , a processor 303 , an input device 304 , an output device 305 , a camera 306 , and a communication control unit 307 .

[0044] Of these, the main memory device 301 stores programs and applications such as a UI module 311, a recording module 312, a location information recording module 313, and a video distribution module 314, and the processor 303 executes these programs and applications to realize each functional element of the inspector terminal 103.

[0045] The UI module 311 displays various screens on a display and accepts operations from the inspector.

[0046] The recording module 312 performs recording in response to the operation of the inspector.

[0047] The location information recording module 313 records the location information of the inspector terminal 103 in response to the operation of the inspector.

[0048] The video distribution module 314 executes real-time distribution of inspection videos in response to the operation of the inspector.

[0049] Next, the configuration of the auxiliary storage device 302 will be described. The auxiliary storage device 302 stores inspection data 321, applications 322, etc. Among these, the inspection data 321 is data generated by the inspector terminal 103 during inspection. This inspection data includes inspection video, position information, and screen capture data. On the other hand, the applications 322 are various applications executed by the processor 303 .

[0050] 1-2-3. Supporter terminal 104 Next, we will explain the configuration of the supporter terminal 104. Fig. 4 shows an example of the configuration of the supporter terminal 104. The supporter terminal 104 is, for example, a stationary or portable computer device used by an inspection supporter who is a professional engineer.

[0051] The supporter terminal 104 includes a main memory device 401 , an auxiliary memory device 402 , a processor 403 , an input device 404 , an output device 405 , a camera 406 , and a communication control unit 407 .

[0052] Of these, the main memory device 401 stores programs and applications such as an inspection support module 411, and the processor 403 executes these programs and applications to realize the various functional elements of the supporter terminal 104. The inspection support module 411 displays various screens on a display and accepts operations by the inspection supporter.

[0053] The auxiliary storage device 402 stores additional information data 421 and an application 422. Of these, the additional information data 421 is data of additional information generated in the supporter terminal 104. On the other hand, the applications 422 are various applications executed by the processor 403 .

[0054] 1-2-4. User terminal 105 Next, we will explain the configuration of the user terminal 105. Fig. 5 shows an example of the configuration of the user terminal 105. The user terminal 105 is, for example, a stationary or portable computer device used by a support requester such as an administrative official.

[0055] The user terminal 105 includes a main memory device 501 , an auxiliary memory device 502 , a processor 503 , an input device 504 , an output device 505 , a camera 506 , and a communication control unit 507 .

[0056] Of these, the main memory device 501 stores programs and applications such as a UI module 511, and the processor 503 executes these programs and applications to realize the various functional elements of the user terminal 105. The UI module 511 displays various screens on a display and accepts user operations.

[0057] The auxiliary storage device 502 stores applications 521. The applications 521 are various applications executed by the processor 403.

[0058] 1-3.Operation 1-3-1. Inspection process flow A description will be given of the inspection process flow executed by the inspector terminal 103. An example of this inspection process flow is shown in Fig. 10. In flow 1000 shown in the figure, an inspector takes an inspection video of an inspection target.

[0059] First, the inspector performs a predetermined operation on the inspector terminal 103 to start the inspection application (S1010). When the inspection application is started, the UI module 311 displays a shooting screen on the display. Fig. 18 shows an example of this shooting screen.

[0060] A screen 1800 shown in the figure is a screen for taking an image of an inspection target using the camera 306. The screen includes a captured image display area 1810, a recording start / stop button 1860, a zoom-in button 1870, a zoom-out button 1880, and a screen capture button 1890.

[0061] Of these display elements, the captured image display area 1810 is an area for displaying a moving image of the subject being captured. The recording start / stop button 1860 is a button for instructing the start or stop of recording. The enlargement button 1870 and reduction button 1880 are buttons for controlling the enlargement and reduction of the display of the subject to be photographed. The screen capture button 1890 is a button for instructing a screen capture (still image).

[0062] When the inspector selects the recording start / stop button 1860 on this screen, the recording module 312 starts recording (S1020). In addition, the location information recording module 313 starts recording the location information of the inspector terminal 103 (S1030). By recording the location information here, it becomes possible to know the geographical location where each frame of the inspection video was taken.

[0063] During the shooting, the inspector can record a still image of the damaged area by operating the screen capture button 1890. At that time, the generated screen capture data includes location information indicating the shooting location. The inspector may also perform a predetermined operation to add additional information (comments, pin information, etc.) to the captured image. In this case, the generated additional information data includes the additional information ID, coordinate information on the image to which the additional information has been added, addition timing information (in other words, the elapsed time since the start of recording), and location information indicating the geographical location where the addition operation was performed.

[0064] When the inspector has finished the inspection, he or she selects the recording start / stop button 1860. In response to this operation, the recording module 312 stops recording (S1040). Also, the location information recording module 313 stops recording the location information of the inspector terminal 103 (S1050).

[0065] After recording has stopped, the UI module 311 displays an input screen (not shown) for video-related information. The inspector inputs the video title, user ID, and inspection target information on this input screen, and selects a save button (not shown) (S1060). In response to this operation, the UI module 311 transmits a save request for the inspection video to the inspection support server 102 (S1070). The transmitted save request includes the inspection video, location information, screen capture data, additional information data, video title, user ID, and inspection target information.

[0066] When the inspection support server 102 receives the save request, it stores various information in the auxiliary storage device 202 (this processing step is not shown). Specifically, the image information acquisition module 211 stores the inspection video, location information, and screen capture data as inspection data in the auxiliary storage device 202. The module also issues a video ID, associates the video ID, video title, user ID, and inspection target information, and registers them in the video related information table 600.

[0067] Furthermore, the additional information management module 214 registers the additional information data in the additional information management table 700 . The inspection process flow 1000 has been described above.

[0068] According to the flow 1000 described above, inspection videos can be stored in the inspection support server 102. In addition, still images of damaged areas can also be stored in the inspection support server 102. By sharing the image information stored in this way with one or more inspection supporters, it becomes possible to discuss corrections to inspection results, repair design policies (scope of measures, timing of repair work), and points to note during repair work.

[0069] 1-3-2. Inspection support sequence Next, we will explain the inspection support sequence executed by the inspection support system 100. Fig. 11 shows an example of this inspection support sequence. In sequence 1100 shown in the figure, upon receiving a request from a support requester, the inspection supporter determines whether repairs are necessary.

[0070] First, the support requester who operates the user terminal 105 logs in to a web page provided by the inspection support server 102 (S1110). At that time, the UI module 511 of the user terminal 105 transmits the user ID to the inspection support server 102.

[0071] The support request module 212 of the inspection support server 102 receives this login and generates an inspection video list. At that time, the module identifies the inspection video associated with the received user ID in the video related information table 600, and generates a list of the identified inspection videos. Then, the module transmits the generated inspection video list to the user terminal 105 (S1115).

[0072] The UI module 511 of the user terminal 105 displays the received inspection video list (not shown) on the display. The person requesting inspection selects an inspection video for which they are requesting inspection support from the displayed list. At that time, the inspector may select not only the inspection video for which they are requesting inspection support this time, but also a past inspection video to be used for comparison. The past inspection video selected here can be displayed on the supporter terminal 104 alongside the inspection video for which they are requesting inspection support this time. The UI module 511 transmits the ID of the inspection video selected by the support requester to the inspection support server 102 (S1120).

[0073] The support request module 212 of the inspection support server 102 receives the returned inspection video ID and then generates a supporter selection screen. At this time, the module generates the supporter selection screen based on the supporter information table 800. Then, the module transmits the generated supporter selection screen to the user terminal 105 (S1125).

[0074] The UI module 511 of the user terminal 105 displays the received supporter selection screen on the display. Fig. 17 shows an example of this supporter selection screen. The screen 1700 shown in the figure includes a table 1701, a send button 1750, and a cancel button 1760.

[0075] Among these display elements, table 1701 is a table showing candidates for experts to request for inspection support. This table 1701 has a "check box" column 1710, a "supporter (ID) name" column 1720, a "attribute information, contact information" column 1730, and a "specialty" column 1740. In the example shown in FIG. 17, "○△ Construction abc" is selected.

[0076] The send button 1750 is a button for sending the supporter information selected by the support requester to the inspection support server 102 . The cancel button 1760 is a button for canceling the selection of a supporter.

[0077] By referring to this screen, the support requester can select an appropriate expert as an inspection supporter based on the candidate's attribute information and area of ​​expertise.

[0078] When the support requester selects an inspection supporter on this screen and selects the send button 1750, the UI module 511 transmits information indicating the selected inspection supporter to the inspection support server 102 (S1130).

[0079] The support request module 212 of the inspection support server 102 receives this supporter information response and sends a support request to the supporter terminal 104 of the selected inspection supporter (S1135). The support request sent includes the URL of the inspection video selected in S1120 and information related to the inspection video (video title, inspection target information, etc.).

[0080] Upon receiving this support request, the inspection support module 411 of the supporter terminal 104 displays a support request receipt screen (not shown) on the display. This support request receipt screen includes the URL of the inspection video and information related to the inspection video. When the inspection supporter accepts this inspection request, he or she selects the URL of the inspection video and transmits a request to distribute the inspection video to the inspection support server 102 (S1140).

[0081] Upon receiving this distribution request, the video distribution module 213 of the inspection support server 102 starts streaming distribution of the inspection video (S1145). Additional information that has been previously added to the inspection video is superimposed on the distributed inspection video.

[0082] If the inspection assistant selects the URL of a past inspection video to be used for comparison in addition to the URL of the inspection video for which the assistant is currently requesting inspection assistance, the module starts streaming distribution of the two inspection videos. The inspection assistance module 411 of the assistant terminal 104 receives this streaming distribution and displays the two inspection videos side by side on the display.

[0083] FIG. 19 shows an example of the screen displayed at this time. The display screen 1900 shown in the figure includes a left window 1910 and a right window 1920. These windows display inspection videos taken at different times for the same inspection object. The right window 1920 displays the latest inspection video for the inspection object including damaged location a, and the left window 1910 displays an inspection video taken earlier.

[0084] Display screen 1900 also includes a video playback interface displayed below each of left window 1910 and right window 1920. The displayed video playback interface includes playback bars 1970 and 1975, play buttons 1950 and 1955, rewind buttons 1940 and 1945, and fast-forward buttons 1960 and 1965.

[0085] Display screen 1900 also includes UI buttons that are superimposed on left window 1910 and right window 1920. The UI buttons include comment add buttons 1912 and 1914, zoom in buttons 1922 and 1924, zoom out buttons 1932 and 1934, and screen capture buttons 1942 and 1944.

[0086] Of these display elements, comment addition buttons 1912 and 1914 are buttons for adding comments in association with any playback position. The enlargement buttons 1922 and 1924 and the reduction buttons 1932 and 1934 are buttons for controlling the enlargement and reduction of the moving image. Screen capture buttons 1942 and 1944 are buttons for instructing screen capture (still image).

[0087] On this screen, the inspection assistant can select the comment addition button 1912 or 1914 while the inspection video is being played or paused, and enter a comment associated with any playback position (S1150). The additional information data generated at that time includes coordinate information on the image to which the additional information is added, and addition timing information (in other words, the elapsed playback time).

[0088] The inspection assistant may also perform a predetermined operation to add pin information while the inspection video is being played or paused. In this case, the generated additional information data includes coordinate information on the image to which the additional information is added and addition timing information (in other words, the elapsed playback time). The predetermined operation here includes, for example, a mouse click operation. For example, a red pin may be added by right-clicking, a yellow pin by double-clicking the left, and a blue pin by single-clicking the left.

[0089] In the display screen 1900 shown in Figure 19, one purple pin (lightly shaded pin) has been added to the bottom right of damaged location a in the left window 1910. Meanwhile, one red pin (darkly shaded pin) and three yellow pins (unshaded pins) have been newly added around damaged location a in the right window 1920. This shows that damage to damaged location a has progressed between the time of the previous inspection and the time of the current inspection.

[0090] When the inspection assistant finishes checking the inspection video, he or she performs a predetermined operation to transmit the generated additional information data to the inspection assistant server 102 (S1155). The additional information management module 214 of the inspection support server 102 registers the received additional information data in the additional information management table 700 (S1160).

[0091] Next, the repair determination module 215 determines whether or not repair is necessary based on the information registered in the additional information management table 700. Then, the module registers the repair determination result in the video related information table 600 (S1165). The inspection support sequence 1100 has been described above.

[0092] The above-described sequence 1100 enables remote support by an inspection assistant. Furthermore, the inspection assistant can check the progression of damage by comparing inspection videos taken at different times for the same inspection target.

[0093] In the above sequence 1100, the inspection video is streamed to the supporter terminal 104, but the supporter may download the inspection video and view it. The inspection supporter may also download and view a still image corresponding to the currently displayed inspection video from the inspection support server 102. The still image referred to here is a still image of the damaged area that was captured on screen while the currently displayed inspection video was being filmed.

[0094] 1-3-3. Repair necessity determination flow A description will be given of the repair necessity determination flow (S1165) executed by the repair determination module 215. An example of this repair necessity determination flow is shown in Fig. 12. In a flow 1200 shown in the figure, the repair determination module 215 determines whether or not the inspection target needs repair.

[0095] First, the repair determination module 215 acquires additional information associated with the determination target from the additional information management table 700 (S1210). Then, the module calculates the degree of repair necessity based on the color and number of pins associated with the determination target (S1220). At that time, the module refers to the pin information table 900 to identify the number of repair points for each pin, and calculates the degree of repair necessity by summing up the identified repair points.

[0096] Next, the module compares the calculated repair necessity level with a predetermined threshold value (S1230). If the result of this comparison indicates that the repair necessity level exceeds the predetermined threshold value, the module determines that repair is necessary (S1240). On the other hand, if the result of this comparison indicates that the repair necessity level is equal to or less than the predetermined threshold value, the module determines that repair is not necessary (S1240).

[0097] The module registers a determination flag indicating the determination result in the video related information table 600 in association with the determination target (S1250). This concludes the explanation of the repair necessity determination flow 1200.

[0098] According to the flow 1200 described above, it is possible to automatically determine whether or not the inspection target needs repair.

[0099] 1-3-4. Inspection support sequence (real-time distribution) Next, another inspection support sequence executed by the inspection support system 100 will be described. Figures 13 and 14 show an example of this other inspection support sequence. In sequence 1300 shown in these figures, an inspection supporter determines whether repairs are necessary upon receiving a request from an inspector. At this time, an inspection video taken by the inspector is distributed to the inspection supporter in real time.

[0100] First, an inspector carrying the inspector terminal 103 logs in to a web page provided by the inspection support server 102 (S1310). At that time, the UI module 311 of the inspector terminal 103 transmits a user ID to the inspection support server 102.

[0101] The support request module 212 of the inspection support server 102 receives this login and generates a supporter selection screen. At that time, the module generates the supporter selection screen based on the supporter information table 800. Then, the module transmits the generated supporter selection screen to the user terminal 105 (S1315).

[0102] The UI module 311 of the inspector terminal 103 displays the received supporter selection screen on the display. Fig. 17 shows an example of this supporter selection screen. Here, the description of Fig. 17 is omitted.

[0103] When the inspector selects an inspection assistant on this screen and selects the send button 1750, the UI module 311 transmits information indicating the selected inspection assistant to the inspection assistance server 102 (S1320).

[0104] The support request module 212 of the inspection support server 102 receives this supporter information response and sends a support request to the supporter terminal 104 of the selected inspection supporter (S1325). At that time, the module identifies the inspection video associated with the received user ID in the video-related information table 600 and creates a past video list. The sent support request includes the past video list.

[0105] The inspection support module 411 of the supporter terminal 104 receives this support request and displays a support request acceptance screen (not shown) on the display. This support request acceptance screen includes an acceptance button and a list of past videos. If the inspection supporter accepts this inspection request, he or she selects the acceptance button and sends an acceptance notice to the inspection support server 102 (S1330).

[0106] Upon receiving this acceptance notification, the video distribution module 213 of the inspection support server 102 establishes a communication session between the inspector terminal 103 and the supporter terminal 104 (S1335). After this communication session is established, the video distribution module 314 of the inspector terminal 103 starts real-time distribution of the inspection video (S1340). Figure 18 shows an example of the shooting screen displayed at this time. An explanation of Figure 18 will be omitted here.

[0107] 18, the recording module 312 starts recording (S1345). Also, the location information recording module 313 starts recording the location information of the inspector terminal 103 (S1350).

[0108] The inspection supporter can request the distribution of past inspection videos to be used for comparison after or before the start of real-time distribution. At that time, the inspection supporter selects the inspection video they want to view from the received list of past videos. Upon receiving this selection, the inspection support module 411 of the supporter terminal 104 transmits a distribution request for the selected inspection video to the inspection support server 102 (S1355).

[0109] The video distribution module 213 of the inspection support server 102 receives this distribution request and starts streaming distribution of the inspection video (S1360). The inspection support module 411 of the supporter terminal 104 receives this streaming distribution and displays the two inspection videos side by side on the display (S1365).

[0110] FIG. 20 shows an example of the screen displayed at this time. The display screen 2000 shown in the figure includes a left window 2010 and a right window 2020. These windows display inspection videos taken at different times of the same inspection object. The right window 2020 displays real-time streaming video, and the left window 2010 displays past inspection videos.

[0111] Display screen 2000 also includes a video playback interface displayed below left window 2010. The displayed video playback interface includes a play bar 2055, a play button 2035, a rewind button 2025, and a fast forward button 2045.

[0112] The display screen 2000 also includes UI buttons that are superimposed on the left window 2010. The UI buttons include a zoom in button 2036, a zoom out button 2046, a screen capture button 2056, and the like. Of these display elements, the enlargement button 2036 and reduction button 2046 are buttons for controlling the enlargement and reduction of the moving image display. The screen capture button 2056 is a button for instructing a screen capture (still image).

[0113] The display screen 2000 also includes a voice call button 2090 that is superimposed on the right window 2020. This voice call button 2090 is a button that enables a voice call with the inspector. The inspection assistant can convey advice to the inspector during the inspection (for example, instructions on inspection points) through the voice call. The voice data of the voice call may be recorded as additional information data in association with the elapsed time of the real-time distribution.

[0114] When the inspector has finished the inspection, he or she performs a predetermined operation to end the real-time distribution. Also, when the inspector has finished the inspection, he or she selects the recording start / stop button 1860. In response to this operation, the recording module 312 stops recording (S1370). Also, the location information recording module 313 stops recording the location information of the inspector terminal 103 (S1375).

[0115] After recording has stopped, the UI module 311 displays an input screen (not shown) for video-related information. The inspector inputs the video title, user ID, and inspection target information on this input screen (S1380). In response to this operation, the UI module 311 transmits a request to save the inspection video to the inspection support server 102 (S1385). The transmitted save request includes the inspection video, location information, screen capture data, video title, user ID, and inspection target information.

[0116] When the inspection support server 102 receives the save request, it stores various information in the auxiliary storage device 202. Specifically, the image information acquisition module 211 stores the inspection video, location information, and screen capture data as inspection data in the auxiliary storage device 202 (S1390). The module also issues a video ID, associates the video ID, video title, user ID, and inspection target information, and registers them in the video related information table 600.

[0117] After the real-time distribution ends, the inspection support module 411 of the supporter terminal 104 transmits the recorded additional information data to the inspection support server 102 (S1395). The additional information management module 214 of the inspection support server 102 registers the received additional information data in the additional information management table 700 (S1400). The inspection support sequence 1300 has been described above.

[0118] The sequence 1300 described above enables remote support by an inspection assistant. Furthermore, the inspection assistant can check the progression of damage by comparing inspection videos taken at different times for the same inspection target. In addition, it also enables immediate response to repairs. Furthermore, the inspection assistant can simultaneously view the inspection target with the inspector and have a voice call with the inspector.

[0119] In the above sequence 1300, past inspection videos are streamed to the supporter terminal 104, but the supporter may download and view the past inspection videos. After S1400, a repair necessity determination flow 1200 shown in FIG. 12 may be executed.

[0120] 1-3-5. Map display flow Next, a description will be given of a map display flow executed by the inspection support server 102. An example of this map display flow is shown in Fig. 15. In flow 1500 shown in the figure, pin information added by an inspector or an inspection supporter is displayed on a map.

[0121] First, the map display module 216 of the inspection support server 102 generates a map screen upon receiving a request from the user terminal 105 (S1510). At that time, the module reads pin information associated with the user who uses the user terminal 105 from the additional information management table 700 and plots it on the map. Then, the module displays the generated map screen on the user terminal 105 (S1520).

[0122] An example of a map screen is shown in Fig. 21. The map screen 2100 shown in the figure includes a budget input field 2120, a total repair amount display field 2130, and buttons 2150-2170. Of these display elements, the budget input field 2120 is a field for inputting the repair budget amount. The total repair cost display column 2130 is a column for displaying the calculated total repair cost. Buttons 2150 to 2170 are buttons for controlling the display range and zoom of the map.

[0123] In the map screen 2100 shown in FIG. 21, as an example, five yellow pins (pins that are not shaded) are added to a building 2140.

[0124] On this screen, the user can input any repair budget amount in the budget input field 2120. When the user inputs the repair budget amount (S1530), the repair target extraction module 217 extracts objects that can be repaired within the input budget amount (S1540). This repair target extraction process will be described later.

[0125] Then, the map display module 216 plots pin information corresponding to the extracted repair target on a map screen, and causes the map screen to be displayed on the user terminal 105 (S1550).

[0126] 22 shows an example of an updated map screen. In the map screen 2200 shown in the figure, "50 million yen" has been entered in the budget input field 2120. In addition, "25 million yen" is displayed in the total repair amount display field 2130. Furthermore, on this screen, compared to the map screen 2100 shown in Fig. 21, pin information is not displayed around the bridge 2180 and the road 2190. This indicates that these objects will be excluded from the repair targets if the repair budget amount is set to "50 million yen."

[0127] The map display module 216 waits for the repair budget amount to be updated (S1560), and when the repair budget amount is updated, executes S1540 again. On the other hand, when the repair budget amount is not updated and the user performs an end operation, the map display on the user terminal 105 ends (S1570). This concludes the explanation of the map display flow 1500.

[0128] According to the flow 1500 described above, the user can grasp the positional relationship of the repair target at a glance. In addition, by simply inputting the repair budget, the user can grasp the targets that can be repaired within that budget.

[0129] 1-3-6. Repair target extraction flow Next, we will explain the repair target extraction flow (S1540) executed by the inspection support server 102. Fig. 16 shows an example of this repair target extraction flow. In flow 1600 shown in the same figure, targets that can be repaired within the repair budget amount entered by the user are extracted.

[0130] First, the repair target extraction module 217 extracts red pin information associated with the target user from the additional information management table 700 (S1610). Then, the module refers to the pin information table 900 and calculates the total repair cost corresponding to the extracted red pin information (S1620). At that time, the module multiplies the number of extracted red pin information by the repair cost corresponding to the red pin information to calculate the total repair cost.

[0131] Next, the module compares the calculated total repair amount with the repair budget amount entered by the user (S1630). If the result of this comparison shows that the total repair amount exceeds the repair budget amount, the module identifies targets with low repair priority. In doing so, the module references the additional information management table 700 to identify targets with a small number of corresponding red pin information. The module then excludes the identified targets and calculates the total repair amount corresponding to the red pin information (S1680).

[0132] The module then compares the calculated total repair cost with the repair budget (S1690). If the comparison shows that the total repair cost exceeds the repair budget, the module executes S1680 again. On the other hand, if the comparison shows that the total repair cost does not exceed the repair budget, the module proceeds to S1670 and determines that the items that fall within the repair budget are to be repaired.

[0133] If the comparison in S1630 shows that the total repair cost is less than the repair budget, the module identifies the object with the highest repair priority from among the objects associated with the yellow pin. The module then references the additional information management table 700 to identify the object with the most corresponding yellow pin information. The module then references the pin information table 900 to calculate and add the repair costs for the identified object (S1640), updating the total repair cost (S1650).

[0134] The module then compares the updated total repair cost with the repair budget (S1660). If the result of this comparison shows that the total repair cost exceeds the repair budget, the module excludes the objects identified in S1640 from the list of repair targets, and determines the objects identified up to that point as the repair targets (1670). On the other hand, if the result of this comparison shows that the total repair cost is less than the repair budget, the module executes S1640 again. The repair target extraction flow 1600 has been described above.

[0135] According to the flow 1600 described above, repair targets can be extracted in descending order of repair priority within a range that does not exceed the repair budget amount designated by the user.

[0136] 2. Variations The above embodiment may be modified as follows: The following modifications may be combined with each other.

[0137] 2-1. Displaying the playback bar 19 and 20 show a playback bar. An identifier (e.g., an icon) indicating the timing of displaying additional information may be displayed on this playback bar. This allows the supporter to easily identify frames to which additional information is added.

[0138] In this case, the additional information management module 214 of the inspection support server 102 records the additional information in association with information regarding the timing at which the additional information was added so that information regarding the additional information can be displayed in association with the playback bar of the inspection video.

[0139] The additional information referred to here may be, for example, pin information that indicates the degree of damage using a color, or text or audio comments. When an identifier for an audio comment is displayed on the playback bar, the audio comment may be output when the identifier is selected. The information relating to the timing at which the additional information was added is, for example, the elapsed playback time of the inspection video.

[0140] 2-2. Identifying potential supporters In the above sequences 1100 and 1300, a supporter selection screen for selecting a supporter is generated (see S1125 and S1315). At this time, supporter candidates may be screened based on the type of inspection object. This makes it easier for the inspection requester to select an inspection supporter.

[0141] In this case, the support request module 212 of the inspection support server 102 extracts one or more candidates for inspection supporters based on the type of inspection target. As an example, this module extracts supporter candidates in the following procedure. First, the module refers to the video related information table 600 and extracts predetermined keywords (e.g., "bridge" or "tunnel") that represent the type of inspection object. The module then refers to the supporter information table 800 (particularly the "specialty" column) and extracts supporter candidates associated with the extracted keywords. This makes it possible to present supporter candidates suitable for the inspection object to the support requester.

[0142] 2-3.Determine whether repairs are necessary In the above-described repair necessity determination flow 1200, the degree of repair necessity is calculated based on the color and number of pins (S1220). However, this calculation method is merely an example. As another calculation method, the degree of repair necessity may be calculated based only on the number of pins, regardless of the color of the pins. Alternatively, the degree of repair necessity may be calculated based only on the color of the pins.

[0143] 2-4. How to display the inspection video In the above embodiment, it is assumed that a two-dimensional inspection video is displayed on the supporter terminal 104, but a three-dimensional model may be generated from the inspection video and the generated three-dimensional model may be viewed stereoscopically by the inspection supporter. In this case, the inspection supporter may add three-dimensional additional information to the three-dimensionally displayed three-dimensional model.

[0144] 2-5. Priority of repair targets In the above repair target extraction flow 1600, structures with a large number of corresponding yellow pin information are identified as structures with a high repair priority (S1640). However, this identification method is merely one example. As another identification method, structures with a high repair priority may be identified based on the number of yellow pin information per area occupied by the structure. In this case, structures with a higher density of yellow pin information are identified as structures with a higher repair priority.

[0145] As another method, structures with a high repair priority may be identified based on their distance to the red pin information. In this case, the closer a structure's yellow pin information is to the red pin information, the higher the repair priority of the structure will be.

[0146] 2-6. Map display of pin information On the map screen shown in FIG. 21 or 22, each piece of pin information is displayed. Instead of this display method, only one piece of pin information may be displayed for each type of pin information for each structure. For example, for a structure to which three pieces of red pin information and five pieces of yellow pin information are attached, only one piece of red pin information with the number "3" attached and one piece of yellow pin information with the number "5" attached may be displayed. Even with this display method, the number of pieces of pin information attached to the structure can be grasped.

[0147] 2-7.Pin Information In the above embodiment, the degree of damage is expressed by the color of the pin information. However, this expression method is merely an example. The degree of damage may be expressed by elements other than color (for example, shape, size).

[0148] 2-8.Other The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0149] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0150] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. The above-described embodiments disclose at least the configurations described in the claims. [Explanation of symbols]

[0151] 100... inspection support system, 101... network, 102... inspection support server, 103... inspector terminal, 104... supporter terminal, 105... user terminal

Claims

1. An inspection support system, image information acquisition means for acquiring information about an inspection video obtained by capturing an inspection target; an assistant information providing means for providing information on one or more candidates for inspection assistants who will determine whether or not repairs are required for the inspection target to a first terminal device used by the assistance requester; a selection receiving means for receiving a selection of one of the one or more inspection assistant candidates from the first terminal device; an image information providing means for providing information about the inspection video to a second terminal device used by the selected inspection assistant; additional information management means for receiving additional information about repairs added by the selected inspection assistant to the inspection target from the second terminal device and recording the additional information in association with the inspection video; An inspection support system with

2. 2. The inspection support system according to claim 1, wherein the additional information management means accepts and records additional information added to the inspection video by the selected inspection supporter while the inspection video is being played back.

3. the image information acquisition means acquires an inspection video of the inspection object distributed in real time by an inspector, the image information providing means provides the acquired inspection video to the second terminal device in real time.

2. The inspection support system according to claim 1, wherein:

4. 4. The inspection support system according to claim 3, wherein the additional information management means records, as the additional information, voice information input by the selected inspection supporter during real-time distribution of the inspection video.

5. The inspection support system of claim 1, wherein the additional information management means records the additional information in association with information regarding the timing at which the additional information was added so that information regarding the additional information can be displayed in association with a playback bar of the inspection video.

6. The inspection video is a video taken by an omnidirectional camera, The additional information is added in association with coordinate values ​​in a three-dimensional coordinate system.

2. The inspection support system according to claim 1, wherein:

7. The inspection support system described in claim 1, wherein the image information providing means provides the second terminal device with information regarding an inspection video of the inspection object that was filmed earlier than the inspection video so as to display it alongside the inspection video.

8. 2. The inspection support system according to claim 1, further comprising a supporter candidate extraction unit that extracts the one or more inspection supporter candidates based on the type of the inspection object.

9. 2. The inspection support system according to claim 1, further comprising a repair determination means for determining a repair priority of the inspection target based on the additional information added to the inspection video.

10. the additional information management means records the additional information in association with location information of the inspection target; further comprising a map information providing means for providing information about a map on which the additional information is plotted to the first terminal device; 2. The inspection support system according to claim 1.

11. a budget input receiving means for receiving an input of a repair budget amount; an inspection target extraction means for extracting inspection targets that can be repaired within the input repair budget amount from among the inspection targets to which additional information has been added by the inspection supporter, the inspection target extraction means extracting inspection targets based on repair costs determined according to the type of additional information; Furthermore, The additional information plotted on the map is additional information added to the extracted inspection target. The inspection support system according to claim 10 .

12. 1. An inspection support method executed by a computer, comprising: acquiring information about an inspection video of an inspection target; providing information on one or more candidates for inspection assistants who will determine whether or not repair of the inspection target is necessary to a first terminal device used by the assistance requester; receiving a selection from the first terminal device of one of the one or more inspection assistant candidates; providing information about the inspection video to a second terminal device used by the selected inspection assistant; receiving additional information about repairs added by the selected inspection assistant for the inspection target from the second terminal device and recording the additional information in association with the inspection video; An inspection support method having the above.

13. A program for causing a computer to execute each step of the inspection support method according to claim 12.

Images