Inspection support device and inspection support method
Patent Information
- Application Number
- JP2024139958
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-08-30
- Filing Date
- 2024-08-21
- Publication Date
- 2025-09-02
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
[Technical field]
[0001] The technology disclosed in this specification relates to an inspection support device that supports image capture of multiple inspection points defined in an inspection area. [Background technology]
[0002] In general, when inspecting a structure, images of the inspection location are taken and recorded. By recording images in chronological order, it is possible to check damage and deterioration over time. An example of a support device that supports the inspection of a structure is disclosed in Patent Document 1. The support device of Patent Document 1 can automatically organize captured images by associating them with damage identification information. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2023-57591 A Summary of the Invention [Problem to be solved by the invention]
[0004] When there are many inspection points, the inspector must be careful not to forget to take pictures of any. This specification provides a device (inspection support device) that supports the inspector in taking pictures of all inspection points reliably. [Means for solving the problem]
[0005] The inspection support device disclosed in this specification includes a database, a display device, and a computer. In the database, a plurality of past images of a plurality of inspection points in an inspection area are stored in association with the positions of the plurality of inspection points. In other words, the positions of the inspection points shown in the past images are stored in association with each of the plurality of past images. The computer causes the display device to display the following three pieces of information: (1) a map showing the positions of the plurality of inspection points on the inspection area; (2) a past image corresponding to a specific position designated by a user (inspector) among the plurality of positions (positions of the inspection points) shown on the map; and (3) a current image taken by the user with a camera. For example, when a plurality of inspection points are set on one bridge, the "inspection area" is the entire bridge, and the "map" may be an image showing the positions of the inspection points on a structural diagram (or a plan view, a side view, or a photograph) of the entire bridge.
[0006] When the user presses the capture switch, the computer associates the current image with the specific location and stores it in the database, and adds an indication to the map indicating that the image of the inspection point at the specific location has been photographed. Since the map shows whether all inspection points in the inspection area have been photographed or not, the user can see at a glance whether they have forgotten to take images of any inspection points. The expression "storing in the database" also includes the process of temporarily storing the current image in the computer and uploading it to the database after current images of all inspection points have been taken.
[0007] The computer may calculate the matching rate between the previous image and the current image displayed on the display device and display it on the display device. With such support, it is possible to obtain an image with a similar composition each time. The "matching rate" is a number calculated based on the amount of deviation between the previous image and the current image, with a perfect match being represented as "1.0" and a perfect mismatch being represented as "0.0", and is expressed as a number between "1.0" and "0.0" depending on the degree of match. Using AI makes it possible to obtain a highly accurate matching rate.
[0008] The computer may further store the current image in the database in association with the specific position when the matching rate is above a predetermined threshold and the capture switch is pressed, thereby ensuring the acquisition of images with a high matching rate.
[0009] Details and further improvements of the technology disclosed in this specification are described in the following "Forms for Carrying Out the Invention". [Brief description of the drawings]
[0010] [Figure 1] FIG. 1 is a block diagram of an inspection support device according to an embodiment. [Diagram 2] FIG. 1 is a schematic diagram showing an example of an inspection area and an inspection location (bridge). [Diagram 3] FIG. 2 is a diagram illustrating an example of a database structure. [Figure 4] This is an example of the display on a display device (1). [Diagram 5] This is an example of the display on a display device (2). [Figure 6] This is an example of the display on a display device (3). [Figure 7] This is an example of the display on the display device (4). [Figure 8] This is an example of the display on the display device (5). [Figure 9] 13 is an example of a display on a display device (an example of a screen transition). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0011] An inspection support device 2 according to an embodiment will be described with reference to the drawings. FIG. 1 shows a block diagram of the inspection support device 2. The inspection support device 2 supports a user (inspector) in taking images of multiple inspection points set in an inspection area. The inspection support device 2 will be described using an example in which several inspection points are specified on bridge A shown in FIG. 2. Four inspection points, positions a, b, c, and d, are specified in advance on bridge A. The entire bridge A is referred to as the inspection area. In other words, the inspection area refers to an area that includes multiple inspection points.
[0012] The inspection support device 2 includes a database 21, a computer 11, and a display device 12. The database 21 is stored in a server 20, and the computer 11 and the display device 12 are provided in a terminal 10 carried by a user. The terminal 10 also includes a camera 13, a storage device 14, and a positioning sensor 15 (GPS). The terminal 10 (computer 11) and the server 20 (database 21) can communicate with each other. The server 20 (database 21) can also communicate with an input / output device 30.
[0013] The database 21 stores information related to the inspection area (bridge A). Figure 3 shows an example of the structure of the database 21. One sheet is assigned to each inspection date. In the example of Figure 3, information for the inspection date of 2020 / 08 / 04 is stored in sheet sh1, information for the inspection date of 2021 / 07 / 30 is stored in sheet sh2, and information for the inspection date of 2022 / 08 / 01 is stored in sheet sh3.
[0014] The latest sheet SH3 stores a map 211 of the inspection area, past images (past images 22a-22d) of each inspection point, and the location AD of the inspection point. In the following, the "location of the inspection point" may be simply written as "location." The map 211 of the inspection area illustrates the location AD of the inspection point.
[0015] Each of the past images 22a-22d of the multiple inspection locations is associated with a respective position ad. In record 1, the past image 22a of the inspection location at position a is associated with position a. In record 2 (3, 4), the past image 22b (22c, 22d) of the inspection location at position b (c, d) is associated with position b (c, d). In other words, the location of each inspection location is associated with a past image taken at the inspection location. Each record is provided with a comment field in which the user can write comments about each inspection location. The structure of sheets sh1 and sh2 is the same as that of sheet sh3. In sheets sh1 and sh2, the past images of each inspection location are stored together with the location of the inspection location. In other words, the database 21 stores time-series images of each inspection location. By comparing the time-series images, the progress of deterioration of the inspection location can be confirmed.
[0016] The database 21 stores past images and positions (positions of inspection points) for a plurality of inspection areas other than bridge A. A description of the information on the inspection areas other than bridge A will be omitted.
[0017] Before starting the inspection of bridge A, the user operates the terminal 10 (computer 11) to download data of the latest sheet (sheet sh3) related to the inspection area (bridge A) to be inspected from the database 21 (server 20) to the storage device 14 of the terminal 10. The computer 11 copies sheet sh3 to create a new sheet (sheet sh4). Images of the inspection location to be photographed are stored in the new sheet (sheet sh4). More specifically, the past image stored in sheet sh3 at the time of copying is overwritten with the newly acquired current image.
[0018] When a user who has downloaded the information on the inspection area operates the terminal 10 (computer 11), the computer 11 displays the information on the inspection location on the display device 12. An example of the display is shown in FIG. 4-8.
[0019] First, the layout of the display screen of the display device 12 will be described with reference to Fig. 4. A map 211 of the inspection area stored in the database 21 is displayed in the first display area 121 of the display device 12. The map 211 is a diagram of the entire structure of the inspection area (bridge A), and the positions (positions a-d) of all the inspection points are illustrated thereon. An image (past image) of the inspection point stored in the database 21 is displayed in the second display area 122. As described above, the data of the sheet sh3 of the inspection area (bridge A) has been copied from the database 21 to the storage device 14 in the terminal 10. The computer 11 displays the past image attached to the sheet sh3 stored in the storage device 14 in the second display area 122.
[0020] Of the multiple positions ad displayed on the map 211 in the first display area 121, a past image of a position (specific position) specified by the user is read from the storage device 14 and displayed in the second display area 122. In the example of FIG. 4, when the user specifies position d, a past image 22d of the inspection point at position d is displayed in the second display area 122. In the first display area 121, position d is surrounded by a double rectangle on the map 211 so that it can be seen that the user has specified position d. In other words, position d is highlighted on the map 211 in the first display area 121 so that it can be seen that the user has specified position d.
[0021] The third display area 123 displays an image captured by the camera 13 mounted on the terminal 10. The image captured by the camera 13 is hereinafter referred to as a current image. The fourth display area 124 displays various information. In the example of FIG. 4, the fourth display area 124 displays today's date, that the inspection area is bridge A, that the specific position designated by the user on the map 211 is position d, and the image matching rate. The image matching rate refers to the degree of matching between the displayed past image 22d and the current image (the image displayed in the third display area 123).
[0022] The computer 11 compares the displayed past image 22d with the current image and calculates the matching rate. An example of a method for calculating the matching rate is as follows. The computer 11 extracts a plurality of feature points (past feature points) from the past image 22d. The computer 11 extracts feature points (current feature points) corresponding to the past feature points from the current image. The computer 11 calculates the deviation between each of the past feature points and the corresponding current feature point, and calculates the average value (or integrated value) of the plurality of deviations. If the average value (or integrated value) is zero, the computer 11 outputs "1.0" as the matching rate. If the average value (or integrated value) exceeds a predetermined upper limit, the computer 11 outputs "0.0" as the matching rate. If the average value (or integrated value) of the deviation is smaller than the upper limit, the computer 11 outputs a numerical value between "1.0" and "0.0" according to the average value (or integrated value) of the deviation as the matching rate. The smaller the average value (or integrated value) of the deviation, the greater the matching rate. The matching rate may be calculated based on the size and rotation angle of the inspection spot in the past image and the inspection spot in the current image.
[0023] When the user confirms that the matching rate is greater than a predetermined threshold, he or she presses the displayed capture switch 125a. Then, the screen of the display device 12 changes to the state shown in FIG. 5. In FIG. 5, the switch arrangement under the third display area 123 is different from that shown in FIG. 4. In FIG. 4, the characters "Current Image" and the switch image of the capture switch 125a were displayed, but in FIG. 5, instead of these images, a save switch 125b and a retake switch 125c are displayed. The third display area 123 holds the current image when the capture switch 125a was pressed. If the held current image is acceptable, the user presses the save switch 125b. If the user wants to retake the current image to be captured, the user presses the retake switch 125c. When the retake switch 125c is pressed, the screen returns to that shown in FIG. 4.
[0024] When the save switch 125b is pressed, the computer 11 stores the captured current image in the newly created sheet (sheet sh4). At this time, the computer 11 stores the position of the inspection point shown in the current image (position d in this case) in the sheet sh4 in association with the current image. In other words, the sheet sh4 originally stores the past image of the copy source sheet sh3, but the computer 11 replaces the past image at position d on the sheet sh4 with the captured current image.
[0025] At the same time, the computer 11 adds a mark to the map 211 indicating that the capture of the current image of the inspection location at position d has been completed. Fig. 6 shows the screen of the display device 12 at this time. A mark (completion mark 127) indicating that the capture of the image of the inspection location at position d has been completed is added next to the display of position d on the map 211 in the first display area 121. By displaying the completion mark 127 next to position d, the user can confirm that they have finished taking images of the inspection location at position d. Once displayed, the completion mark 127 will remain displayed until the inspection work is completed.
[0026] 6, the switch display below the third display area 123 has been changed to a continue switch 125d and an end switch 125e. The user presses the continue switch 125d and specifies, on the map 211 in the first display area 121, the location of the inspection point where the next image is to be taken.
[0027] The user repeats the same operation for each of the multiple positions ad. FIG. 7 shows an example of a display screen after images of all inspection points at the positions ad have been taken. A completion mark 127 indicating completion of capture has been added next to each display of the positions ad on the map 211. By looking at the map 211 (the image in the first display area 121) to which the completion mark 127 has been added, the user can prevent forgetting to take an image of an inspection point. Note that the screen in FIG. 7 is a display after the current image of the inspection point at position a has been taken, and position a on the map 211 is highlighted with a double rectangle, and a current location mark 126 indicating the current location of the camera 13 is displayed near position a. The current location mark 126 will be described later.
[0028] When acquisition of current images of all inspection locations has been completed, the user presses the end switch 125e. When the end switch 125e is pressed, the computer 11 adds data of the inspection date to a new sheet sh4 and uploads it to the server 20 (database 21). In other words, when the user presses the import switch 125a, the computer 11 associates the current image with a specific position and stores it in the database 21. A new sheet sh4 is added to the database 21. In this way, time-series data of past images of each inspection location is accumulated.
[0029] Even if there are many inspection points in an inspection area, the user can prevent forgetting to take a current image by checking the display of the positions and the completion mark 127 displayed on the display device 12 (map 211).
[0030] A modified example of the inspection support device will be described with reference to FIG. 8. In the modified example, the computer 11 executes the following process. As described above, the computer 11 displays the matching rate between the past image and the current image on the display device 12. When the matching rate is below a predetermined threshold, the computer 11 deactivates the capture switch 125a. Note that "deactivating the capture switch 125a" means that the computer 11 does not react even if the capture switch 125a is pressed. In other words, when the matching rate is below the threshold, the computer 11 prohibits the capture of the current image. When the matching rate is below the threshold when the capture switch 125a is pressed, the computer 11 does not capture the current image, but instead causes the display device 12 to display a message 128 informing the user that the matching rate is below the threshold. When the matching rate is above the threshold and the capture switch 125a is pressed, the computer 11 associates the current image with a specific position and stores the image in the storage device 14 (i.e., the database 21).
[0031] Also, in FIG. 4-8, the current location mark 126 displayed on the map 211 indicates the current location of the camera 13. As described above, the terminal 10 is equipped with the positioning sensor 15, and the computer 11 displays the positioning position of the positioning sensor 15 (i.e., the current location of the camera 13) on the map 211. The user can ascertain which inspection point he or she is near by checking the current location mark 126 on the map 211. This function can prevent the user from photographing the wrong inspection point when there are multiple inspection points with similar structures.
[0032] An example of screen transitions on the display device 12 will be described with reference to Fig. 9. Fig. 9(A) shows a state in which a map 211 of the inspection area is displayed on the entire screen of the display device 12. Fig. 9(B) shows a four-screen split state similar to Fig. 4. In Fig. 4, a "Zoom in" button 129 has been added to the map 211 displayed in the first display area. Other than the "Zoom in" button 129, Fig. 9(A) is the same as Fig. 4.
[0033] As shown in Fig. 9(A), map 211 may be displayed on the entire screen of display device 12. Then, for example, when the user touches the vicinity of inspection point position d (see finger illustration 301), computer 11 may transition the screen to the state of Fig. 4. That is, computer 11 may transition the screen display from Fig. 9(A) to Fig. 9(B). Furthermore, when the user touches "Enlarge" button 129 (see finger illustration 302) in the display of Fig. 9(B), map 211 may be enlarged and displayed on the entire screen. That is, computer 11 may transition the screen display from Fig. 9(B) to Fig. 9(A).
[0034] Alternatively, when the user touches near another location in the full-screen display of map 211 (FIG. 9(A)), computer 11 may switch the display on display device 12 to a display relating to the location touched by the user (a split-screen display similar to FIG. 9(B) and based on the location touched by the user).
[0035] Furthermore, when the user pinches in (pinch out) on the display screen of the map 211 in Fig. 9(A), the computer 11 may reduce (enlarge) the map 211. The computer 11 may enlarge / reduce the display for each display area in Fig. 9(B) based on the user's operation.
[0036] 9(B), when the user performs a predetermined operation, computer 11 may switch second display area 122 (or another display area) to a full-screen display. Alternatively, computer 11 may hide or redisplay any of the split screens based on another operation by the user.
[0037] Furthermore, the computer 11 may display other information, for example, the information written in the comment field described above, in the fourth display area 124 (or another display area). The information written in the comment field may include the name of the component to be inspected (component name), the name of the inspection element in the component (element number), and, if the inspection point is damaged, the type of damage / the evaluation value of the degree of damage / the magnitude of the damage, etc.
[0038] Although not shown in the figure, the terminal 10 is also provided with a keyboard, and the user can input information about the inspection location that has been photographed using the keyboard. The inspection support device 2 stores the information input from the keyboard in a comment field in the database 21. The keyboard may be a software keyboard that is displayed on the display device 12.
[0039] The input / output device 30 shown in FIG. 1 will be described. The server 20 can communicate with the input / output device 30. The input / output device 30 stores a plurality of formats of inspection papers related to inspection points. The inspection support device 2 can import information of a new inspection area from a past inspection paper into the database 21 using the input / output device 30. The input / output device 30 compares the input past inspection paper with each of a plurality of formats and determines which format the input inspection paper matches. When the format of the input inspection paper is identified, it is possible to determine where and what information is recorded in the inspection paper. The input / output device 30 extracts information required for constructing the database 21 from the input inspection paper, creates a database of a new inspection area, sends it to the server 20, and adds it to the database 21. In other words, the input / output device 30 can convert the format of the input inspection paper into the format of the database 21 and add the information included in the inspection paper to the database 21.
[0040] Furthermore, the inspection support device 2 can output information from the database 21 as an inspection paper using the input / output device 30. As described above, a plurality of formats for inspection papers are stored in the input / output device 30. When the user designates an inspection area and a specific format, the input / output device 30 extracts information for the designated inspection area from the database 21, writes the information from the database 21 in each portion of the designated format, and outputs it. In other words, the input / output device 30 can output information from the database 21 as an inspection paper in a designated format.
[0041] The input / output device 30 may obtain information from a paper medium. In this case, the input / output device 30 is equipped with a scanner and an optical character reader (OCR). The input / output device 30 is equipped with a printer and may print and output the inspection record in a specified format. The input / output device 30 may import the inspection record as a data file, and may output the inspection record in a specified format as a data file.
[0042] Points to note regarding the technology described in the embodiment are described below. In the database 21, a plurality of past images of a plurality of inspection points in an inspection area are stored in association with the positions of the inspection points shown in the past images. In other words, in the database 21, the positions of a plurality of inspection points in an inspection area are stored in association with past images of the respective inspection points. For one inspection point, its position is associated with a past image of the inspection point. The user associates the oldest past image with the position of the inspection point. When the computer processing described above is performed, the computer automatically associates the current image of each inspection point with its position.
[0043] When the inspection support device 2 of the embodiment has taken images of all inspection points in the inspection area, it uploads a new sheet sh4 stored in the storage device 14 of the terminal 10 to the server, and adds the new sheet sh4 to the database 21. During the inspection work, the computer 11 and the server 20 are in constant communication, and each time a current image is captured, the current image and its position (the position of the inspection point) may be associated and stored in the database 21. Even in the case where the current image is uploaded to the database 21 after the current images of all inspection points have been captured, this is essentially equivalent to "when the user presses the capture switch 125a, the computer 11 stores the current image in the database 21 in association with a specific position."
[0044] If the inspection area is a structure such as a bridge or a nuclear reactor, the "map" may be a perspective view, plan view, or side view of the structure including the inspection location. The "map" may also be a photograph of the structure including the inspection location. The "map" may show a schematic or simplified representation of the structure such as a bridge or a nuclear reactor. If the inspection location is a structure along a specific road (such as a guardrail, road sign, or utility pole), the "map" may be a map including the specific road. In the "map," the specific road may be represented by a schematic line. The locations of multiple inspection locations are illustrated on the "map."
[0045] 4 to 8 are examples of displays. The display format does not matter as long as at least the following three pieces of information are displayed on the display device 12: (1) A map showing the locations of multiple inspection points on the inspection area; (2) A past image associated with a specific location designated by the user (inspector) among the multiple locations (locations of inspection points) shown on the map; and (3) A current image being captured by the user with a camera.
[0046] 9 is also an example of the display. The computer 11 may switch the screen in various ways based on the user's operation.
[0047] In the embodiment, the computer 11 and the camera 13 are provided in a single terminal 10. The computer and the camera may be separate devices.
[0048] Although specific examples of the present invention have been described above in detail, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and variations of the specific examples exemplified above. The technical elements described in this specification or drawings exhibit technical utility alone or in various combinations, and are not limited to the combinations described in the claims at the time of filing. In addition, the technology exemplified in this specification or drawings can achieve multiple objectives simultaneously, and achieving one of these objectives is itself technically useful. [Explanation of symbols]
[0049] 2: Inspection support device 10: Terminal 11: Computer 12: Display device 13: Camera 14: Storage device 15: GPS 20: Server 21: Database 22a-22d: Previous images 30: Input / output device 121: First display area 122: Second display area 123: Third display area 124: Fourth display area 125a: Import switch 125b: Save switch 125c: Retake switch 125d: Continue switch 125e: Exit switch 126: Current location mark 127: Completion mark 128: Message 211: Map
Claims
1. a database in which a plurality of past images of a plurality of inspection locations in an inspection area are stored in association with the inspection locations shown in the past images; a display device; A computer, It is equipped with The computer displays on the display device: The past image associated with the inspection location designated by the user; The current image that the user is taking with the camera, on the same screen, Inspection support device.
2. A method for assisting in taking images of inspection points of a structure, comprising: A step of accepting designation of the inspection location; a step of outputting a past image of the designated inspection point taken in the past and a current image being taken by a camera on the same screen; A method for supporting inspection of a structure.
3. The inspection support method according to claim 2 , wherein the computer executes a step of prompting a user to select whether to store the current image in association with the inspection location or to retake the current image.
4. An inspection support method as described in claim 2 or 3, wherein when the computer stores the current image in association with the inspection location, it executes a step of displaying a map showing the location of the inspection location, the map having an indication at the inspection location indicating that current image capture has been completed.
5. The computer displaying a map showing the locations of the inspection points; a step of reducing a display area displaying the map when the inspection location is selected on the display; The inspection support method according to claim 2 or 3, further comprising the steps of: