In-pipe inspection device, in-pipe inspection method, and program

Abstract

To enable the interior of a pipe to be efficiently inspected.SOLUTION: An in-pipe inspection device according to an embodiment includes: a video alignment unit which performs alignment of associating images that have different imaging times and same imaging position in the individual images respectively included in multiple videos or multiple video segments for the multiple videos captured by a camera moving inside a pipe or the multiple video segments that constitute one video; and an output processing unit which displays the multiple videos or multiple video segments subjected to the alignment side by side on a display device for comparison. The output processing unit simultaneously displays the images that have different imaging times and same imaging position.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] An embodiment of the present invention relates to an inside-pipe inspection device, an inside-pipe inspection method, and a program. [Background technology]

[0002] When sediment or rust accumulates inside pipes such as water pipes after many years of use, various abnormalities occur inside the pipes, including the accumulation and adhesion of foreign matter and deterioration and peeling of the inner surface of the pipe.

[0003] To inspect the inside of a pipe such as a water pipe, for example, a camera is inserted into the pipe and the camera is moved around inside the pipe and stopped at various points to capture images of the entire inside of the pipe. Based on the images obtained, the pipe can be checked for any abnormalities. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6532412 [Patent Document 2] Patent Publication No. 2021-162547 Summary of the Invention [Problem to be solved by the invention]

[0005] During an internal pipe inspection, for example, the deterioration of the pipe may be verified from the images of the pipe after filming, or the difference in the condition of the pipe before and after cleaning may be verified. In such cases, multiple images are often compared. However, since the trajectories of the camera movement are often different for each of the multiple images, it is often difficult to match frames taken in the same shooting position to be compared, and it is often impossible to properly verify the above-mentioned aspects. This can result in time-consuming measures to deal with the problem, making it difficult to proceed with the inspection efficiently.

[0006] An object of the present invention is to provide an inside-pipe inspection device, an inside-pipe inspection method, and a program that enable efficient inspection of the inside of a pipe. [Means for solving the problem]

[0007] The pipe inspection device of the embodiment includes an image alignment unit that aligns multiple images obtained by capturing images using a camera moving inside the pipe, or multiple image portions that make up a single image, by matching up individual images contained in the multiple images or multiple image portions that have different capture times and the same capture positions, and an output processing unit that displays the aligned multiple images or multiple image portions on a display device side-by-side for comparison, and the output processing unit simultaneously displays the images captured at different times but the same capture positions. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of the overall configuration of a system including an inside-pipe inspection device according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of the configuration of an inside-pipe inspection device 1 according to an embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of a state in which a camera 4 is inserted into a water pipe 3. As shown in FIG. [Figure 4] FIG. 4 is a diagram showing an example of an image included in a video actually captured by the camera 4. As shown in FIG. [Diagram 5] FIG. 5 is a diagram showing an example of an image on which information indicating the cable length (that is, position information of the camera 4) is displayed. [Figure 6] FIG. 6 is a diagram showing the relationship between the position of the camera 4 reciprocating inside the pipe and time (frames). [Figure 7] FIG. 7 shows an example (part 1) of a table used for frame correspondence and an example of a table summarizing the results of frame correspondence using the table. [Figure 8]FIG. 8 shows an example (part 2) of a table used for frame correspondence and an example of a table summarizing the results of frame correspondence using the table. [Figure 9] FIG. 9 is a diagram showing an example of a screen transition realized by various functions provided in the output processing unit 80. As shown in FIG. [Figure 10] FIG. 10 is a diagram showing an example of the video list screen 51A. [Figure 11] FIG. 11 is a diagram showing an example of the video analysis result confirmation screen 52A. [Figure 12] FIG. 12 is an enlarged view of an example of the time bar 52c. [Figure 13] FIG. 13 is a diagram showing an example of the outbound / inbound comparison screen 53A. [Figure 14] FIG. 14 is a diagram showing an example of an outbound / inbound comparison list screen 53A'. [Figure 15] FIG. 15 is a diagram showing an example of the difference comparison screen 54A. [Figure 16] FIG. 16 is a diagram showing an example of the difference comparison list screen 54A'. [Figure 17] FIG. 17 is a diagram showing an example of the 2D measurement screen 55A. [Figure 18] FIG. 18 is a diagram showing an example of the 3D measurement screen 56A. [Figure 19] FIG. 19 is a flowchart showing an example of the operation of the inside-pipe inspection device according to the embodiment. [Figure 20] FIG. 20 is a flowchart showing an example of an operation related to the UI by the output processing unit 80. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment will be described with reference to the drawings.

[0010] <System configuration> FIG. 1 is a diagram showing an example of the overall configuration of a system including an inside-pipe inspection device according to an embodiment.

[0011] In this embodiment, an example will be described assuming that inspection of the inside of a water pipe (plumbing pipe), in particular inspection of the inside of the pipe when it is full of water as an inspection without interrupting the water supply, however, this embodiment is not limited to this example and can also be applied to inspection of the inside of various types of pipes including, for example, sewer pipes.

[0012] The pipe inspection device 1 according to this embodiment inspects the inside of a pipe based on images captured by a camera 4 capturing images of the inside of a water pipe 3 installed at a site 2 as shown in FIG. 1. The images captured by the camera 4 are captured into an information terminal (for example, a personal computer, a smartphone, or a tablet). The images are, for example, moving images made up of a plurality of images (frames). The information terminal may also store attribute information indicating attributes associated with the camera. The captured images (i.e., moving images) are, for example, recorded on a recording medium 5 as video data (files) made up of a plurality of successive images (frames), or uploaded to and stored in a storage unit of a predetermined server on a network (on the cloud).

[0013] The inside-pipe inspection device 1 imports video data recorded on the recording medium 5, or imports video data stored in the memory of a specified server, and uses these to inspect the inside of the pipe. The attribute information may be imported into the inside-pipe inspection device 1 via the recording medium 5 or the server together with the video data, or may be imported into the inside-pipe inspection device 1 via a route different from the video data, without going through the recording medium 5 or the server.

[0014] An example of the configuration of the inside of a pipe inspection device 1 according to an embodiment is shown in Fig. 2. Fig. 2 is a diagram showing an example of the configuration of the inside of a pipe inspection device 1.

[0015] 2 includes, as various functions, a video input unit 10, an attribute information input unit 11, a camera position / movement detection unit 20, an abnormality detection unit 30, an integrated judgment unit 40, a video position alignment unit 50, a three-dimensional information acquisition unit 60, an information management unit 70, and an output processing unit 80. Some or all of these functions are realized as a program to be executed by one or more computers.

[0016] 2 is merely an example, and is not limited to this example. For example, the inside of the pipe inspection device 1 is not limited to one device, but may be configured by multiple devices, and these multiple devices may exist in different locations. In addition, the various functions described above may be provided in different devices and may exist in different locations.

[0017] For example, the video input unit 10, attribute information input unit 11, camera position / motion detection unit 20, anomaly detection unit 30, integrated judgment unit 40, video position adjustment unit 50, three-dimensional information acquisition unit 60, and information management unit 70 may be provided in a predetermined server, and the output processing unit 80 may be provided in an arbitrary information terminal (for example, a personal computer, a smartphone, or a tablet). In this case, the output processing unit 80 may be provided in advance to the arbitrary information terminal side from the predetermined server as a program that realizes a UI (User Interface).

[0018] The image input unit 10 inputs image data captured by a camera 4 that captures the inside of a water pipe 3 .

[0019] The attribute information input unit 11 inputs attribute information of the camera 4. The attribute information is supplied to the camera position / movement detection unit 20, the abnormality detection unit 30, and the information management unit 70, and is used as necessary in the processing of each unit.

[0020] The camera position / movement detection unit 20 uses the video data input by the video input unit 10 and the attribute information input by the attribute information input unit 11 to detect the position or movement of the camera 4 within the pipe.

[0021] The abnormality detection unit 30 selects one or more images from the video data of the detection target captured by the camera 4 according to the position and movement of the camera 4 detected by the camera position / movement detection unit 20, and uses the selected images to determine the presence or absence of an abnormality and the level of abnormality of each detection target within the pipe, and generates the determination result as an abnormality detection result for each detection target.

[0022] The objects to be detected here refer to any of the following: (1) rust, (2) deposits, (3) partial adhesion of foreign matter / deterioration of corrosion protection on the pipe surface, (4) film-like dirt adhering to the entire inside of the pipe, and (5) floating objects.

[0023] The integrated determination unit 40 integrates the anomaly detection results generated for each detection target by the anomaly detection unit 30, and generates an integrated determination result indicating the level of anomaly within the pipe.

[0024] The image alignment unit 50 uses the image input by the image input unit 10 and the position and movement information of the camera 4 detected by the camera position / motion detection unit 20 to perform alignment for multiple images captured by a camera moving inside the pipe, or multiple image portions constituting a single image, by matching together individual images (frames) contained in each of the multiple images or multiple image portions that were captured at different times and in the same position.

[0025] The three-dimensional information acquisition unit 60 restores three-dimensional information from the video input by the video input unit 10. The three-dimensional information is information that forms an image with a three-dimensional stereoscopic effect. A method of generating the three-dimensional information will be described later.

[0026] The information management unit 70 manages information on the position and movement of the camera 4 detected by the camera position / movement detection unit 20, information including the abnormality detection results generated for each detection target by the abnormality detection unit 30 and the integrated judgment result generated by the integrated judgment unit 40, information including the video alignment result generated by the video alignment unit 50, information including three-dimensional information restored by the three-dimensional information acquisition unit 60, attribute information input by the attribute information input unit 11, and the like.

[0027] The output processing unit 80 accepts operations from the user, and outputs various pieces of information managed by the information management unit 70 to the display device in response to the user's operations.

[0028] <Details of main components> The main elements constituting the inside of pipe inspection device 1 will be described in detail below.

[0029] Video input section 10 The video input unit 10 acquires and inputs video data captured by a camera 4 capturing an image inside the pipe from a specified information terminal via a recording medium 5 or a server. For example, as shown in Fig. 1, a video data file can be saved in a recording medium 5 such as an SD card, and the file can be read by a reader that reads the recording medium 5, thereby inputting the video data.

[0030] FIG. 3 shows an example of the state in which the camera 4 is inserted into the water pipe 3. As shown in FIG. 3, the camera 4 may be waterproofed as a whole in order to move inside the water pipe 3 at the site 2. The camera 4 may also be equipped with a function to illuminate the inside of the pipe and a rotation mechanism to keep the camera 4 vertical and horizontal at all times. A long cable 6 is connected to the camera 4, and the camera 4 is inserted into the water pipe 3 by pushing the cable 6 from a fire hydrant 7 or the like, and the camera 4 takes pictures of the inside of the water pipe 3. For taking pictures, the camera 4 is inserted both upstream and downstream. To achieve this, the direction in which the camera 4 is inserted can be controlled by a jig placed between the fire hydrant 7 and the water pipe 3.

[0031] Furthermore, in order to obtain information on the orientation and movement of the camera 4, the camera 4 may be equipped with sensors such as a gyro sensor or an acceleration sensor, which may be used to detect the orientation and movement of the camera 4. In this case, any type of sensor may be used as long as it can measure the movement and position of the camera 4. The information on the orientation and movement of the camera 4 can be used when the camera position / movement detection unit 20 detects the movement and position of the camera 4.

[0032] The image from the camera 4 is supplied to an information terminal outside the fire hydrant 7 via a cable 6, and is recorded on the recording medium 5 as described above, or uploaded to a specified server on the cloud and stored in a specified memory unit. The image may be recorded in any format, and the format is not important. For example, the image may be in the analog NTSC format, or in a digital uncompressed format or a digitally encoded format.

[0033] Figure 4 shows an example of an image contained in a video actually captured by camera 4. Normally, almost no foreign objects are found inside the cylindrical pipe, but if an abnormality occurs inside the pipe, the images show foreign objects accumulating and adhering inside the pipe as shown in Figure 4(a), the inner surface of the pipe deteriorating and peeling off, and foreign objects floating in the water inside the pipe as shown in Figure 4(b).

[0034] ·Attribute information input section 11 Next, the attribute information input unit 11 will be described.

[0035] As described above, the attribute information may be imported into the inside pipe inspection device 1 together with the video data via the recording medium 5 or server, or may be imported into the inside pipe inspection device 1 via a route different from that of the video data, without going through the recording medium 5 or server, or may be input from the camera 4 or an input device such as a keyboard or mouse. In this case, information obtained from a sensor or the like in the camera 4 may be used as the attribute information. The attribute information generated in either form may be stored as a data file on a recording medium such as an SD card, and the attribute information may be input by reading the file with a reader that reads the medium.

[0036] The attribute information input by the attribute information input unit 11 includes, for example, information indicating the length of the cable 6 when the camera 4 is inserted into the water pipe 3 (cable length), which becomes the position information of the camera 4 within the pipe. Fig. 5 shows an example of an image on which information indicating the cable length (i.e., the position information of the camera 4) is displayed.

[0037] The information indicating the cable length can be used in the camera position / motion detection unit 20, which will be described later, by superimposing the information on a predetermined position of the video data and recording it. If the video is recorded in a digital format, the information indicating the cable length may be recorded in synchronization with the video data, rather than being superimposed on the video data.

[0038] Other attribute information can also be input in the attribute information input unit 11. For example, attribute information such as the location of the water pipe (place name, latitude and longitude, address, etc.), the type of water pipe, the diameter of the water pipe, the year the water pipe was installed, the date and time of the previous inspection and the diagnosis result at that time, and the model and lens parameters of the camera can be input in the attribute information input unit 11 by recording the information in association with the recording medium that records the video.

[0039] Furthermore, if the camera 4 is equipped with sensors such as a gyro sensor or an acceleration sensor, information indicating the position and movement of the camera 4 based on these sensors may be used as attribute information, and the attribute information may be used in the camera position / movement detection unit 20 described below.

[0040] Camera position and motion detection section 20 The camera position / motion detection unit 20 detects the current position of the camera 4 within the pipe, and also detects whether the camera 4 is moving or not. In this example, it is assumed that the video data and attribute information are input to the camera position / motion detection unit 20 via the video input unit 10 and attribute information input unit 11, and the camera position / motion detection unit 20 detects the position and movement of the camera 4 using the video data and attribute information.

[0041] A method for detecting the position and movement of the camera 4 using video data will be described below.

[0042] (1) Detecting the camera position First, a description will be given of detection of the position of camera 4. The simplest method for detecting the position of camera 4 is to read information indicating the cable length (i.e., position information of camera 4) from the attribute information input by attribute information input unit 11. If the video data is in digital format, this can be achieved by simply embedding information indicating the cable length in association with each frame in the video data and reading the value.

[0043] If information indicating the cable length is superimposed and recorded in the video data, the information can be read using optical character recognition (OCR). In this case, recognition is possible by finding areas where characters are lined up horizontally from each image, without specifying a character display area in the camera 4. For example, the digits 0 to 9 and the decimal point "·" can be registered as templates, and the image can be searched using template matching to find character areas by using areas where characters that match any of the characters are lined up horizontally, and at the same time, the numerical value can be read from the template information of the number that matches the most.

[0044] In the example image of Figure 5, the templates "1", "6", "·", and "4" have a high matching score in the upper right area of ​​the screen, and since they are lined up left and right with almost the same position in the Y-axis direction, it is possible to read the value as 16.4. For character recognition in this case, for example, the technology in Reference 1 below may be used.

[0045] ·References 1 “O.Hori, “A video text extraction method for character recognition”, Proceedings of the Fifth International Conference on Document Analysis and Recognition. ICDAR '99 (Cat. No.PR00318)” Also, by previously including information designating the model of the camera 4 in the attribute information, an area for displaying information indicating the cable length may be set according to the model.

[0046] In the case of a cable push-in type camera, the cable 6 bends when pushed in and stretches when pulled back, so the length of the cable 6 tends to be displayed differently even at the same point. For this reason, the accuracy of the position may be improved by comparing images of the round trip when pushed in and pulled back, and correcting the cable length value at the time of pulling back as the official value for similar images. In this case, a method of comparing similar images may be used, or a more accurate position may be obtained by using a technique for reconstructing 3D information using a monocular camera as shown in Reference 2 below.

[0047] ·Reference 2 "Seki, "3D Shape Reconstruction and Self-Localization from Moving Camera Images (SLAM) and Dense 3D Shape Reconstruction", Information Processing Society of Japan Research Report Computer Vision and Image Media (CVIM), 2014-CVIM-190, 2014" Furthermore, if the type of water pipe 3 is known, the position of each connection between the pipes can be known. Therefore, if the position of each connection appears as a concentric circle with a certain or greater brightness change (edge) in the image, the position of camera 4 may be found by counting the number of times the concentric circle edge is detected.

[0048] (2) Camera movement detection Next, we will explain how to detect the movement of the camera 4. A simple method is to find the optical flow for the entire image, and when the vanishing point is found from the flow direction, if it is found that many flow vectors have the vanishing point as their starting point or end point, it can be determined that the camera 4 is moving.

[0049] However, when photographing the inside of a water pipe 3, if there is any floating matter in the water pipe 3, the floating matter will flow toward the back of the pipe or from the back of the pipe to the front, and a vector in the vanishing point direction may be generated even if the camera 4 is stationary. Therefore, the presence or absence of movement of the camera 4 may be determined by detecting that when the camera 4 moves, the image changes during the exposure time, causing a blur (motion blur) and blurring the entire image. For example, the edge strength in the image may be calculated and the value may be accumulated to determine that the state where the edge is stationary is reached when the accumulated strength is greater than a certain threshold, and that the state where the edge is less than a certain threshold is determined to be moving due to blurring.

[0050] To reduce the effect of floating objects, the area of ​​pixels to be accumulated may be limited, for example, to the periphery of the tube (excluding the center). Also, edge strength may be calculated by accumulating the power of high-frequency components by applying FFT to the entire image, rather than to edges with adjacent pixels. For example, the frequency generated by floating objects may be calculated in advance, and then the frequency may be excluded, allowing accurate detection of the presence or absence of movement.

[0051] In addition, this method is limited to images saved in the analog signal format of NTSC (National Television Standards Committee), but in this case, the images are recorded alternately line by line in the vertical direction by the interlace method, so that striped noise called combing noise occurs when the camera 4 moves. This is because the shooting times of odd-numbered and even-numbered rows are different, so that striped luminance changes occur where changes in brightness occur. Therefore, for this striped portion, when Y(x,y) is the luminance value Y of the coordinates x,y, with a predetermined coefficient α, if there are a predetermined number of pixels that satisfy the following formula or more, it may be determined that combing noise has occurred and that the camera 4 is moving.

[0052] |Y(x,y)-Y(x,y-1)| > |(x,y)-y(x,y-2)|+α According to the above formula, normally the luminance changes smoothly in space, so the luminance difference between the two rows above is larger than the luminance difference between the row above, but when combing noise occurs, the luminance difference between the row above is larger than the luminance difference between the row above. This method makes use of this phenomenon.

[0053] If information from sensors such as a gyro sensor and an acceleration sensor mounted on the camera 4 is input from the attribute information input unit 11 to the camera position / movement detection unit 20, the movement of the camera 4 can be easily detected by reading the information. In this case, it is possible to accurately detect the movement of the camera 4 regardless of the underwater conditions.

[0054] In this way, it is possible to detect the current position of the camera 4 and whether or not the camera 4 is moving.

[0055] Anomaly detection unit 30 The abnormality detection unit 30 detects the presence or absence of an abnormality in the water pipe 3 for each detection object from the video data and determines the level of the abnormality. In this embodiment, the detection objects are assumed to be (1) rust, (2) deposits, (3) partial adhesion of foreign matter / deterioration of corrosion protection on the pipe surface, (4) dirt adhering in a film form to the entire inside of the pipe, and (5) floating matter. The method of detecting each detection object will be described below.

[0056] (1) Chorus Rust occurs when the inner surface of a tube corrodes and rusts, and often appears reddish brown in color. Since the inner surface of a tube does not normally turn reddish brown, it is possible to detect rust by finding reddish brown areas through color detection. The level of abnormality of the chorus may be determined based on, for example, the shape of the inside of the pipe or the blockage rate from the video data.In addition to the method using the blockage rate as an index, other values ​​may be used as the index, such as the total area (number of pixels) of the chorus on the screen, the size of the chorus chunks by labeling, or the area of ​​the chorus in a predetermined area to be determined.

[0057] (2) Sediment In deposit detection, foreign matter accumulated at the bottom of the pipe is detected as deposit. The level of abnormality of the deposit may be determined based on, for example, the area or volume. In addition to the method using the area or volume as an index, the size of the area where the deposit exists or the number of lumps obtained by labeling may also be used as an index.

[0058] (3) Partial adhesion of foreign matter and deterioration of the corrosion protection of the pipe surface A normal water pipe has a uniform surface color, so there are no edge-like features. On the other hand, when abnormalities occur such as partial adhesion of foreign matter or deterioration of the corrosion protection of the pipe surface (hereinafter referred to as foreign matter adhesion and inner surface deterioration), edge-like features tend to appear in the image. Therefore, by using the edges detected by edge processing, it is possible to detect foreign matter adhesion and inner surface deterioration and to determine the level of abnormality. For example, as shown in the example image of Fig. 4(a), when an edge is detected, a circumscribing rectangle of the connected range is obtained, and the abnormality level may be determined based on the area of ​​the circumscribing rectangle. In addition to the method using the area of ​​the circumscribing rectangle as an index, the number of abnormal regions or the size of the distribution range (the size of the circumscribing rectangle of the entire cluster of abnormalities) may also be used as an index.

[0059] (4) Dirt that has formed a film-like coating on the entire inside of the pipe It is difficult to detect film-like dirt that has adhered to the entire inside of a pipe (hereinafter referred to as film-like adhesions) using the method described in (3) above. Therefore, by calculating the average brightness and color of the inner surface of the pipe and calculating the difference from the normal state, it is possible to detect film-like adhesions and determine their abnormality level. The level of abnormality of the film-like deposit may be determined based on, for example, the amount of change in brightness from the normal state. In addition to the method using the amount of change in brightness as an index, the size of the line traced by the camera 4, for example, the thickness, length, area, or cumulative value of edge strength, may also be used as an index.

[0060] (5) Floating objects Floating objects are foreign objects contained in the water flowing inside the pipe, and are reflected in the form of optical flow, as shown in the example image of Figure 4(b). Floating objects can be detected from the optical flow. The abnormality level of the floating matter may be determined, for example, such that a state in which there are no floating matter is normal (eg, level S), and the abnormality level is increased (worsened) as the amount of floating matter increases.

[0061] Integrated Judgment Section 40 The integrated judgment unit 40 detects anomalies individually for the five types of detection targets described above, and then uses the mutual relationships between those results to output a final result. Note that there is no problem if the results are output as a final result without using the mutual relationships.

[0062] When obtaining the integrated anomaly judgment result, for example, the calculation method (weighted addition method, etc.) for generating the integrated judgment result may be changed (for example, merging or excluding) depending on the mutual relationship between the anomaly detection results for each detection target. For example, it is possible to use only rust when judging rust, but use both rust and deposits when judging deposits.

[0063] As a method for calculating the integrated abnormality determination result, for example, Y(1) to Y(5) may be the final results of each of the five types of abnormalities, X1 to X5 may be the anomaly detection results calculated individually, and a to e may be the respective weighting coefficients, and the final result may be calculated by weighting each anomaly detection result by multiplying it by a predetermined coefficient as shown in the following formula.

[0064] Y(i) = a*X1 + b*X2 + c*X3 + d*X4 + e*X5 By adjusting each coefficient in the formula, appropriate action can be taken when the individual dependencies are clear. For example, when determining the abnormal level of deposits, the result of ORing the abnormal detection results of rust and deposits can be determined as the abnormality detection result (this can also be achieved by using the coefficient of the above Y formula), and if a certain abnormality is found, it can be excluded from other abnormality detection results using the above formula.

[0065] Also, a predetermined abnormality may be detected first, and the result may be used to adjust the result of another abnormality detection. For example, the abnormality level of a predetermined detection target may be adjusted depending on whether or not there is an influence between the detection targets (for example, since sediments are easily confused with floating matter, the abnormality level may be adjusted depending on whether or not there is floating matter).

[0066] For example, since floating objects occur across the entire area of ​​the image, the presence or absence of anomalies in the floating objects and the level of the anomalies may be determined first, and when there are a lot of floating objects, the criteria for the detection results of other anomaly detection targets may be corrected. When there are a lot of floating objects, it becomes difficult to see abnormalities in the image, so that an abnormality that actually exists may be missed, or conversely, it may be easier to determine an abnormality when there is no abnormality. It is also possible to lower or raise the level of the abnormality at a predetermined level, and the length of the area (section) in which the abnormality is determined may be changed depending on the amount of floating objects. For example, when there are a lot of floating objects, the influence of noise can be reduced by performing the level determination from the anomaly detection results obtained over a long range.

[0067] In addition, instead of being limited to five types of abnormality detection, one overall judgment result may be output as the overall judgment result for the entire pipe, regardless of the type of abnormality detection. This can be realized by calculating the worst value, average value, weighted addition, etc. of the five types of abnormality level judgment results.

[0068] Image alignment unit 50 As described above, the image alignment unit 50 uses the images input by the image input unit 10 and the position and movement information of the camera 4 detected by the camera position / motion detection unit 20 to perform alignment for multiple images captured by the camera 4 moving inside the pipe, or multiple image portions constituting a single image, by matching up individual images contained in each of the multiple images or multiple image portions that were captured at different times and in the same position.

[0069] In this example, the following two types of alignment are assumed: [A] and [B]. The implementation methods for each are explained below.

[0070] [A] A method for matching frames from the same camera position at different times (frames) in a single video First, we explain how to match frames at different times in the first video.

[0071] First, the position information of each frame in the video can be obtained by reading the position information in the video, which is implemented in the camera position and motion detection unit 20. Specifically, as described in FIG. 5, in the camera video of the water pipe 3, text information according to the insertion amount of the cable connected to push the camera 4 in is embedded in the video, and the distance from the entrance of the water pipe 3 can be obtained by reading this value. This text information is displayed at a fixed position (different depending on the type of camera 4) in the same video, so the current camera position can be obtained by reading the text information in this area using OCR technology or the like. Note that the cable is prone to bending on the way in which it is pushed in, so the cable length appears longer than the actual camera position, and so it is possible to improve accuracy by carrying out a correction process that takes this into account (such as reducing the length by a certain ratio).

[0072] FIG. 6 shows the relationship between the position of the camera 4 reciprocating inside the pipe and time (frames).

[0073] In the case of an inspection video of a water pipe 3, video is taken of the outward journey by pushing the camera 4 in, and video is taken of the return journey by pulling back the cable when the camera 4 reaches the end.

[0074] As shown in Fig. 6, the first half of the image corresponds to the outbound journey, and the second half of the image corresponds to the return journey. In this case, the first and second halves of the image pass through the same point inside the water pipe 3. Therefore, a process is performed to match each image of the outbound journey with the image taken on the return journey. This process is premised on resetting the OCR read value in advance (resetting the reference position immediately after insertion into the pipe to 0.0 m), and also on the premise that the beginning and end positions of the image are 0.0 m.

[0075] FIG. 7(a) shows an example (part 1) of a table used for frame correspondence.

[0076] The vertical axis indicates the frame numbers of the video corresponding to the forward path (the direction moving from the beginning to the second half of the video) and the respective camera positions (OCR reading values), and the horizontal axis indicates the frame numbers of the video corresponding to the reverse path (the direction moving in the opposite direction from the end to the first half of the video) and the respective camera positions (OCR reading values).

[0077] (Step 1) First, obtain the turning point of Camera 4 from the video. To obtain the turning point, find the maximum value of the OCR reading value and, from the detection result of the camera position and movement detection unit 20, find the location where the movement of Camera 4 stops. In Fig. 7(a), among the 9 frames of the entire video, 3.0 m is the maximum value of the OCR reading value, so the turning point is set as the 5th frame.

[0078] (Step 2) Set the frame that becomes the reference 0.0 m at the beginning of the video as FA, and the frame that becomes 0.0 m at the end of the video as FB. In Fig. 7(a), FA = 1 and FB = 9 (the numerical values represent frame numbers).

[0079] (Step 3) Compare the OCR reading values in the images of the FA / FB frames as OCR(FA) / OCR(FB) respectively.

[0080] When OCR(FA) = OCR(FB): Add 1 frame to FA and subtract 1 frame from FB.

[0081] When OCR(FA) > OCR(FB): Keep FA as it is and subtract 1 frame from FB.

[0082] When OCR(FA) < OCR(FB): Add 1 frame to FA and keep FB as it is.

[0083] By doing this, the frames of the forward path and the reverse path can be associated so that the OCR reading value is always the same value.

[0084] Repeat this process until both FA and FB reach the turning point.

[0085] A specific processing example based on the table in FIG.

[0086] Start from position FA=1,FB=9 marked “START” in the table ((1) in the table).

[0087] Next, since the reading value of FA (frame=1) is 0.0 m and the reading value of FB (frame=9) is 0.0 m, FA is increased by one frame and FB is decreased by one frame, resulting in FA=2 and FB=8 ((2) in the table).

[0088] Next, since the reading value of FA (frame=2) is 1.0 m and the reading value of FB (frame=8) is 1.0 m, similarly FA is added by 1 frame and FB is subtracted by 1 frame, and the correspondence is set as FA = 3 and FB = 7 ((3) in the table).

[0089] Next, since the reading value of FA (frame = 3) is 1.0 m and the reading value of FB (frame = 7) is 2.0 m, FA is added by one frame while FB remains the same, and the correspondence is made as FA = 4 and FB = 7 ((4) in the table).

[0090] Next, since the reading value of FA (frame=4) is 2.0 m and the reading value of FB (frame=7) is 2.0 m, FA and FB are each added by one frame, and the correspondence is set as FA=5 and FB=6 ((6) in the table).

[0091] Finally, since the reading value of FA (frame=5) is 3.0 m and the reading value of FB (frame=6) is 2.0 m, one frame is subtracted only from FB, and the correspondence is established as FA=5 and FB=5 ((7) in the table).

[0092] At this point, both FA and FB have reached the turning point, and the association process is completed.

[0093] (Step 4) The matching results are managed in a table as image alignment information.

[0094] (Step 5) Next, the results of associating frames using the table in FIG. 7(a) are managed together in another table.

[0095] FIG. 7(b) shows an example of a table summarizing the results of associating frames using the table in FIG. 7(a).

[0096] Figure 7(b) is an example of a table that allows you to check the corresponding frames from either the forward or backward direction. The horizontal axis shows the frame number of the image that applies to either the forward or backward direction and the respective camera position (OCR read value). The vertical axis has two items, forward and backward.

[0097] The first row for the outbound leg lists the frame numbers of the video on the return leg that correspond to frame numbers 1 to 5 of the video on the outbound leg, along with the matching results. The second row for the return leg lists the frame numbers of the video on the outbound leg that correspond to frame numbers 5 to 9 of the video on the return leg, along with the matching results.

[0098] Between the outbound and inbound routes, one frame may correspond to multiple frames, so the correspondence is not necessarily 1:1.

[0099] By managing a table like that shown in FIG. 7(b), it is possible to instantly identify associated frames from either the forward or backward direction.

[0100] [B] A method for matching frames from the same camera position between two or more videos. Next, a method for associating a plurality of videos will be described.

[0101] Here, we will explain an example of a method of matching the second and subsequent images by using the frame of the first image as a reference. Even if there are three or more images, matching can be done regardless of the number of images by using the first image as a reference for matching the third and subsequent images in the same way as the second image is matched to the first image as described here.

[0102] FIG. 8(a) shows an example (part 2) of a table used for frame correspondence.

[0103] The horizontal axis indicates the frame number of the first image (Image 1) and the respective camera positions (OCR reading values), and the vertical axis indicates the frame number of the second image (Image 2) and the respective camera positions (OCR reading values).

[0104] (Step 1) First, the turning point of camera 4 is found in each video. To find the turning point, the maximum OCR read value is found, and the location where the movement of camera 4 stops is found from the detection results of camera position and motion detection unit 20. In Fig. 6, since the maximum OCR read value is 3.0m in both video 1 and video 2, the turning point is the 7th frame where the OCR value is 3.0m in video 1, and the 5th frame where the OCR value is similarly 3.0m in video 2.

[0105] (Step 2) The frames at the beginning of video 1 and video 2 that are the reference 0.0m are set as FA and FB, respectively. In the example in Figure 6, the initial values ​​are both 0.0m, so FA=1 and FB=1.

[0106] Also, since the direction in which the OCR read value increments differs between the forward and return passes, processing is first carried out in the forward pass mode (the direction in which the OCR value increases).

[0107] (Step 3) The OCR readings in the images of the FA / FB frames are compared as OCR(FA) / OCR(FB), respectively.

[0108] ·When OCR(FA) = OCR(FB): Add one frame to both FA and FB.

[0109] ·When OCR(FA) > OCR(FB): Keep FA as it is and add one frame to FB.

[0110] ·When OCR(FA) < OCR(FB): Add one frame to FA and keep FB as it is.

[0111] By doing this, the first and second frames can be associated so that the OCR reading values are always the same. Repeat this process for both FA and FB until the folding point is reached.

[0112] The following shows a specific processing example based on the table in Fig. 8(a).

[0113] Start from the position where FA = 1 and FB = 1 marked as "Start" in the table. Next, since the reading value of FA (frame = 1) = 0.0m and the reading value of FB (frame = 1) = 0.0m, add one frame to both FA and FB and associate them as FA = 2 and FB = 8 (item (2) in the table).

[0114] Next, since the reading value of FA (frame = 2) = 1.0m and the reading value of FB (frame = 2) = 1.0m, also add one frame to both FA and FB and associate them as FA = 3 and FB = 3 (item (3) in the table).

[0115] Next, since the reading value of FA (frame = 3) = 1.0m and the reading value of FB (frame = 3) = 1.0m, also add one frame to both FA and FB and associate them as FA = 4 and FB = 4 (item (4) in the table).

[0116] Next, since the read value of FA (frame = 4) is 2.0 m and the read value of FB (frame = 4) is 2.0 m, add one frame to each of FA and FB, and associate them as FA = 5 and FB = 5 (item (5) in the table).

[0117] Next, since the read value of FA (frame = 5) is 3.0 m and the read value of FB (frame = 5) is 2.0 m, add one frame only to FB, and associate them as FA = 5 and FB = 6 (item (6) in the table).

[0118] Finally, since the read value of FA (frame = 5) is 3.0 m and the read value of FB (frame = 6) is 2.0 m, add one frame only to FB, and associate them as FA = 5 and FB = 7 (item (7) in the table).

[0119] Here, since both FA and FB have reached the folding point, switch the association process to the return path.

[0120] (Step 4) Next, in the forward path mode, advance the frame while aligning in the direction where the OCR value becomes smaller in the opposite direction to the forward path.

[0121] · When OCR(FA) = OCR(FB): Add one frame to both FA and FB.

[0122] · When OCR(FA) > OCR(FB): Add one frame to FA and keep FB as it is.

[0123] · When OCR(FA) < OCR(FB): Keep FA as it is and add one frame to FB.

[0124] (Step 5) Manage the associated results in a table as video alignment information.

[0125] (Step 6) Next, summarize and manage the results of associating the frames using the table in Fig. 8(a) in another table.

[0126] FIG. 8B shows an example of a table summarizing the results of associating frames using the table in FIG. 8A.

[0127] Figure 8(b) is an example of a table that allows you to check the corresponding frames from both video 1 and video 2. The horizontal axis, like Figure 8(a), shows the frame numbers of video 1 and the respective camera positions (OCR read values).

[0128] The vertical axis has only the item of frame numbers of video 2. In this row, the frame numbers of video 2 that correspond to frame numbers 1 to 11 of video 1 are written together with the results of the correspondence.

[0129] Here, an example of a table is shown in which the first video (video 1) is used as the reference to match the second video (video 2). However, if there are three or more videos, matching can be performed regardless of the number of videos by adding and managing information on the frame number of the third video and the frame number of the fourth video in the row direction of Figure 8(b).

[0130] By managing a table like that shown in FIG. 8(b), it is possible to instantly identify associated frames from any of multiple videos.

[0131] ·3D information acquisition unit 60 As described above, the three-dimensional information acquisition unit 60 restores three-dimensional information from the video input by the video input unit 10. For example, many techniques have been developed to obtain three-dimensional information from two-dimensional video captured by a monocular camera.

[0132] For example, this can be achieved by using the technology in the following reference documents:

[0133] ·Reference 3 Masako Kashiwagi et al., “Deep Depth From Aberration Map”, Proceedings of the IEEE / CVF International Conference on Computer Vision (ICCV), 2019, pp. 4070-4079(https: / / openaccess.thecvf.com / content_ICCV_2019 / html / Kashiwagi_Deep_Depth_From_Aberration_Map_ICCV_2019_paper.html) ·Reference 4 N. Mishima et al., “Absolute Scale from Varifocal Monocular Camera > through SfM and Defocus Combined,” BMVC, 2021.(https: / / www.bmvc2021-virtualconference.com / assets / papers / 0287.pdf) The three-dimensional information obtained by these techniques may be managed as RGB-D (RGB color information and distance value) that assigns a distance value from the camera to each pixel of a still image in each frame of the video, or the information may be managed as three-dimensional point cloud information. Either method can hold color information (RGB value) of a specific location in the pipe and three-dimensional distance information (relative positional relationship). The obtained three-dimensional information is managed by the information management unit 70.

[0134] ·Information Management Department 70 As described above, the information management unit 70 manages various pieces of information supplied from each of the camera position / motion detection unit 20, the anomaly detection unit 30, the integrated judgment unit 40, the image alignment unit 50, and the 3D information acquisition unit 60, thereby enabling the realization of various screen displays and UI functions in the output processing unit 80.

[0135] The information management unit 70 manages, for example, in a table format, information obtained from the camera position / motion detection unit 20, such as OCR read values ​​(camera position) for each frame.

[0136] Furthermore, the information management unit 70 manages the anomaly detection result for each detection target and one overall judgment result for the entire video as information obtained from the anomaly detection unit 30 and the integrated judgment unit 40. In this case, the input image, the processed image (an image in which the presence or absence of anomalies is displayed pixel by pixel), and the rank judgment result for each anomaly are managed in table format on a frame by frame basis.

[0137] Furthermore, the information management unit 70 manages the image alignment results and the like as information obtained from the image alignment unit 50. The image alignment results vary depending on the number of images specified. In the case of one image, only the outward and return journeys are associated. In the case of two or more images, both the outward and return journeys within each image and the images are associated with each other.

[0138] Furthermore, the information management unit 70 manages three-dimensional information and the like as information obtained from the three-dimensional information acquisition unit 60. Examples of the three-dimensional information include the RGBD values ​​of each frame and three-dimensional point cloud information of the entire water pipe.

[0139] Output processing unit 80 As described above, the output processing unit 80 accepts operations by the user, and outputs various pieces of information managed by the information management unit 70 to the display device in response to the user's operations.

[0140] For example, the output processing unit 80 has a function of displaying a plurality of images or a plurality of image portions that have been aligned by the image alignment unit 50 side-by-side on a display device so that they can be compared.

[0141] In addition, the output processing unit 80 has a function of displaying, on a display device, multiple image portions, namely, an image portion of the outbound journey captured by the camera 4 as it moves along an outbound path within the pipe, and an image portion of the return journey captured as the camera moves along a return path in the opposite direction to the outbound path, side-by-side for comparison.

[0142] The output processing unit 80 also has a function of displaying one or more representative images selected as representative images from each of the individual images contained in the outbound video portion and the return video portion, the representative images being taken at different times and in the same location, side-by-side for comparison.

[0143] The output processing unit 80 also has a function of displaying, as a plurality of images, a plurality of images captured by the same tube at different shooting time periods, side by side on the display device so that they can be compared.

[0144] The output processing unit 80 also has a function of displaying one or more representative images selected as representative images from among the individual images contained in each of a plurality of videos captured by the same tube at different shooting times, side by side for comparison, the representative images being different shooting times and the same shooting locations.

[0145] In addition, the output processing unit 80 has a function of measuring at least one of the length, area, and volume of an object specified on the image by the user based on the three-dimensional information obtained by the three-dimensional information acquisition unit 60 from the image inside the pipe, and displaying the measurement results.

[0146] The output processing unit 80 also has a function of displaying an image that enables the user to specify a measurement range of an object.

[0147] The output processing unit 80 also has a function of displaying an image based on the above-mentioned three-dimensional information that enables the user to specify a measurement range of the object.

[0148] The output processing unit 80 is capable of displaying a screen that allows the user to efficiently check the content of an abnormality by comparing the input video and the abnormality detection results, as well as a screen for managing a list of videos, a screen for comparatively displaying videos, and a screen for performing various measurements of the videos.

[0149] The hardware configuration capable of mounting the function of the output processing unit 80 may be, for example, a personal computer (PC) and a display device such as a liquid crystal display. In this case, the PC manages the video captured by the video input unit 10, the attribute information input by the attribute information input unit 11, and the results detected and judged by the anomaly detection unit 30 and the integrated judgment unit 40, and displays the necessary screens on the display according to the operation. A typical liquid crystal display is used as the display device, but a tablet or smartphone may be used, and a display of a different type from the liquid crystal display may be used. In addition, a UI for screen operation such as a mouse or a touch panel is provided to enable operations for controlling the screen, designating an area, playing and stopping, etc. A keyboard for character input and for improving operation efficiency by shortcut keys may also be connected.

[0150] (Various display screens) In this embodiment, an example in which the displayed screen is configured from the following eight types of screens is shown.

[0151] FIG. 9 shows an example of screen transitions realized by various functions provided in the output processing unit 80. In FIG.

[0152] The following eight types of examples are shown as basic image configurations, but the same functions can be realized by dividing the screen into smaller parts or integrating it into fewer screens. Also, when switching between screens, a separate window can be opened, or the screen layout of the same window can be changed.

[0153] The output processing unit 80 performs a process of displaying various display screens on a display device, as described below, and various other processes in response to input operations by an operator.

[0154] 1) Video list screen 51A for inputting multiple videos and managing them as a list 2) A video analysis result confirmation screen 52A that displays the input video and anomaly detection results (video analysis results) for the video 3) An outward / return comparison screen 53A that displays the images corresponding to the outward and return journeys of the image displayed on the image analysis result confirmation screen 52A side by side and allows comparison while playing back each image. 4) An outward / inward comparison list screen 53A' that displays representative images of each video associated with the outward and inward journeys for the video displayed on the video analysis result confirmation screen 52A side by side and allows comparison of the representative images of each video 5) A difference comparison screen 54A that displays at least two selected images taken at different times side by side and allows comparison while playing the images. 6) A difference comparison list screen 54A' for displaying at least two representative images of selected images taken at different times side by side and comparing the representative images. 7) A 2D measurement screen 55A that can measure the length, area, and volume of an object in a specified image displayed on the screen. 8) A 3D measurement screen 56A that displays a 3D point cloud corresponding to the displayed image in a screen-displayed image of a specified video, and can measure the length, area, and volume of an object in the image. The output processing unit 80 can also appropriately display screens other than the above-mentioned display screens in response to operations.

[0155] (Screen transition overview) In this embodiment, when the system or program is started, a video list screen 51A is displayed. In the video list screen 51A, a list of videos for investigation is displayed, and editing, searching, adding, deleting, and the like can be performed on this screen. When one video is selected from the video list, a video analysis result confirmation screen 52A can be opened by pressing "To video analysis screen" described later. In addition, when multiple videos are selected, a difference comparison screen 54A or a difference comparison list screen 54A' can be opened by pressing "To difference comparison screen" or "To difference comparison list screen" described later. In addition, in the video analysis result confirmation screen 52A, a difference comparison screen 53A or a difference comparison list screen 53A' can be opened by pressing a "Outward and Inward Comparison Display" button or a "Outward and Inward Comparison List Display" button described later, respectively, and a 2D measurement screen 55A or a 3D measurement screen 56A can be opened by pressing a "2D measurement" button or a "3D measurement" button described later, respectively.

[0156] (Configuration of various display screens) The video list screen 51A shown in FIG. 9 displays a video list 51a, video file information 51b, status information 51c, an analysis execution button 51d, a video analysis result confirmation display button 51e, a report display button 51f, a difference comparison display button 51g, and a difference comparison list display button 51h.

[0157] In addition, the video analysis result confirmation screen 52A displays input video 52a, processed result video 52b, time bar 52c, abnormality type information 52d, partial judgment bar 52e, capture icon area 52f, video playback function 52g, screen enlargement buttons 52h, 52i, capture execution buttons 52j, 52k, overall judgment result information 52l, partial judgment result information 52m, color switch button 52n, thumbnail display area 52o, report display button 52p, 2D measurement button 52r, 3D measurement button 52s, outbound / inbound comparison display button 52t, outbound / inbound comparison list display button 52u, and the like.

[0158] An outbound journey image 53a, an inbound journey image 53b, and the like are displayed on the outbound journey comparison screen 53A.

[0159] An outbound journey representative image list 53a', a return journey representative image list 53b', and the like are displayed on the outbound journey / return journey comparison list screen 53A'.

[0160] The difference comparison screen 54A displays multiple images 54a, 54b to be compared, image file information 54c, a time bar 54d, a image playback function 54e, a snapshot display area 54f, a snapshot execution button 54g, a report display button 54h, an image alignment display playback checkbox 54i, and an image alignment display playback function 54j.

[0161] A difference comparison list screen 54A' displays a list 54a' of a plurality of representative images with different dates.

[0162] The 2D measurement screen 55A displays a 2D image 55a, a measurement result display area 55b, a measurement mode display area 55c, and the like.

[0163] The 3D measurement screen 56A displays a 3D image 56a, a measurement result display area 56b, an operation mode display area 56c, measurement mode information 56d, and the like.

[0164] The details of each piece of information displayed on these screens will be explained later.

[0165] (Details of each screen) Below, with reference to Figures 10 to 18, the video list screen 51A, the video analysis result confirmation screen 52A, the outbound / inbound comparison screen 53A, the outbound / inbound comparison list screen 53A', the difference comparison screen 54A, the difference comparison list screen 54A', the 2D measurement screen 55A, and the 3D measurement screen 56A will be described in detail.

[0166] Note that the screen examples shown in Figures 10 to 18 are merely examples and are not limited to these examples. Individual pieces of information displayed in Figures 10 to 18 may be added, changed, or deleted as appropriate.

[0167] (Video list screen 51A) FIG. 10 shows an example of the video list screen 51A.

[0168] A video list 51a is displayed on the video list screen 51A. In the video list 51a, each video is managed in units of video file information 51b. Each video managed in the video file information 51b is treated as video data in a file format (video file).

[0169] For example, by pressing a button marked "Import Data" in the upper right of the screen (e.g., by clicking or tapping with an input device), it becomes possible to specify a video file to be added to the video list 51a from among a plurality of pieces of video data (video files) already stored in the storage device, and to add information about the specified video file to the video list 51a. For example, when the button marked "Import Data" is pressed, another window may be displayed, in which the desired video file may be specified. In this window, the desired video file may be specified by narrowing down the search by inputting a keyword (e.g., a specific shooting date and time, shooting location, etc.), or the desired video file may be specified from a plurality of options (i.e., a plurality of video files) provided.

[0170] When an operator designates a video file to be added to the video list 51a, the designated video file is read out from a plurality of video files already stored in the storage device, and the information of the video file is imported and added to the video list 51a.

[0171] In this example, a video file is specified and information about the video file is imported into video list 51a from a storage device, but the video file information may also be imported into video list 51a by transferring (uploading) the video file information from another terminal via a network, or the video file information may be read from a memory card or other medium on which the video file is stored and imported into video list 51a by inserting the memory card or other medium into a card reader or the like.

[0172] A video file added to the video list 51a may include information (attribute information) associated with the video. The attribute information may be included in the video file, but the attribute information may be stored in a separate file (attribute file) and the attribute file may be managed in association with the video file. In this case, when a video file is specified, information on the associated attribute file may be imported into the video list 51a together with information on the video file. At this time, the information on the video file and the information on the attribute file may be combined and added to the video list 51a as information on a single file.

[0173] The information displayed in each video file information 51b includes information related to the corresponding video. This may include attribute information associated with the video. The contents of the video file information 51b can be freely edited by the operator, as described later.

[0174] In this example, the video file information 51b is displayed in a list format, but is not limited to this example. For example, the video file information 51b may be displayed in a format in which icons are arranged.

[0175] In the example of Fig. 10, the video list 51a displays one line of video file information 51b for each video. Items displayed in the video list 51a include the file name, shooting date and time (time period), shooting location, pipe information (diameter, material, etc.), and installation year of the video files managed by each video file information 51b. These may also include information equivalent to attribute information. Note that the numbers lined up in the column direction are not limited to numbers, and may be any information that can identify the video, such as letters and symbols.

[0176] In the video file information 51b, the content of each item may be directly edited by the worker, or editing may be performed only when the video file information 51b to be edited is selected and a button marked, for example, "Information Input" is pressed. In addition, during editing, characters or the like may be input, or one of a number of options prepared in advance may be selected (for example, for the pipe diameter (cm), one of the options "100", "150", "200", etc. may be selected).

[0177] Additionally, at the top of the screen, various controls that can be used to search for video files (controls that enable the specification of conditions such as shooting location, shooting date and time, free keywords, etc.) are provided. After specifying conditions such as a specific shooting location and shooting date and time using these controls, pressing the button marked "Search" displays information on video files that match the conditions in the video list 51a as video file information 51b.

[0178] For example, when a shooting location is specified, if multiple pieces of video file information 51b with the same shooting location but different shooting dates and times are displayed, the worker can compare them. Also, when a shooting date and time is specified, if multiple pieces of video file information 51b with the same shooting date and time but different shooting locations are displayed, the worker can compare them.

[0179] When searching for a video file, instead of using the above controls, a separate window may be opened and search conditions may be specified in the window. In this case, one of a number of options may be selected, or a keyword may be entered to perform a search.

[0180] Also, as shown on the right side of the screen, status information 51c may be displayed for each video file information 51b as an item showing the processing status of each video. The status information 51c displays one of three types of status: "Unanalyzed" if the video has not yet been analyzed, "Analyzed" if the video has been analyzed (but a report has not yet been created), and "Report Created" if the video has been analyzed and a report has been created. Also, such statuses may be included in the search targets for searches.

[0181] If the status is "Analyzed", the abnormality level indicated in the overall judgment result (for example, one of "S", "A", "B", "C", or "D") may be displayed as one of the items.

[0182] If the status is "unanalyzed", video analysis can be performed by selecting the video to be analyzed and pressing the analysis execution button 51d. When the video analysis is completed and the video analysis results are generated, the status changes to "analyzed".

[0183] Also, at the bottom right of the screen, there are a video analysis result confirmation display button 51e, a report display button 51f, and a difference comparison display button 51g.

[0184] When one piece of video file information 51b with a status of "unanalyzed" is selected, for example, by pressing a video analysis result confirmation display button 51e labeled "Go to video analysis results", a video analysis result confirmation screen 52A for the corresponding video can be opened. Instead of pressing the video analysis result confirmation display button 51e, the list may be opened by double-clicking (double-tapping) or the like. The status display is assumed to be "analyzed", but when the video analysis result confirmation display button 51e is pressed in an unanalyzed state, a message such as "video analysis not yet performed" may be displayed to prompt the operator to perform the video analysis.

[0185] In addition, when one piece of video file information 51b with a status of “report created” is selected, by pressing a report display button 51f labeled, for example, “Go to report screen,” a report screen where a report can be created can be directly opened.

[0186] Furthermore, with two pieces of video file information 51b selected, by pressing a difference comparison display button 51g labeled, for example, "Go to difference comparison screen," a difference comparison screen 54A can be opened in which the two corresponding videos are lined up.

[0187] Furthermore, when multiple pieces of video file information 51b with different dates are selected, by pressing a difference comparison list display button 51h labeled, for example, "Go to difference comparison list screen," a difference comparison list screen 54A' can be opened in which representative images with multiple corresponding different dates are arranged.

[0188] The screen layout, whether to open a separate window, and the like are design considerations and may be changed as appropriate. The same can be said for the various display screens described below.

[0189] (Video analysis result confirmation screen 52A) FIG. 11 shows an example of the video analysis result confirmation screen 52A.

[0190] On the video analysis result confirmation screen 52A, the input video 52a and the processing result video 52b are displayed side by side for easy comparison. On the input video 52a, for example, one frame of the video designated on the video list screen 51A is displayed. On the processing result video 52b, an image is displayed on which the processing result of performing video analysis on the image is superimposed. Specifically, the area indicating an abnormality is displayed in a colored state.

[0191] More specifically, the area indicating an abnormality is displayed with a different color corresponding to the type of abnormality being detected (here, "rust," "deposits," "internal adhesion," "foreign matter adhesion," "internal corrosion prevention," and "floating matter").

[0192] The color display has two modes: a first mode in which a different color is applied to each type of abnormality so that the type of abnormality can be identified (here, pink for "rust," green for "deposits," purple for "internal adhesion," persimmon for "foreign matter adhesion," blue for "internal corrosion protection," and light blue for "floating matter"), and a second mode in which a different color is applied to each level of abnormality so that the level of abnormality can be identified (here, green for "A," yellow-green for "B," orange for "C," reddish-brown for "D," and no color for "S"). Either mode can be selected using color switch button 52n, which will be described later.

[0193] In the example of FIG. 11, an example is shown in which the portion in the processing result image 52b in which rust is detected is colored reddish brown, indicating an abnormality level D of the partial determination result.

[0194] In this example, the difference between the abnormality types and the difference between the abnormality levels are expressed by different colors, but the present invention is not limited to this example. For example, instead of different colors, the difference between the abnormality types and the abnormality levels may be expressed by different color tones, different color transparency, etc.

[0195] Below the input image 52a and the processed image 52b is a time bar 52c. The time bar 52c includes a graph showing the relationship between the image capture time from the start to the end of image capture by the camera 4 and the camera position in the longitudinal direction inside the pipe.

[0196] FIG. 12 shows an enlarged example of the time bar 52c. In the graph in the time bar 52c, the horizontal axis indicates the time when the image was captured, and the vertical axis indicates the camera position, and the movement T of the camera 4 is shown in the graph. In addition, a vertical pointer P indicates the relationship between the position on the time axis of the image displayed in the input image 52a and the processed result image 52b and the camera position in the longitudinal direction inside the pipe. Under the time bar 52c, information on the camera position obtained from the camera position / movement detection unit 20 for each frame in the image is displayed. In the example of FIG. 12, the camera position indicating the vertical axis is 0 m at the top, and the further down it is, the deeper the camera has progressed into the pipe. From this graph, not only can the camera position of the displayed image be confirmed, but also the movement of the camera and whether the camera is on the outward or return path can be confirmed.

[0197] When the pointer P is moved left or right by dragging or the like, the input video 52a and the processed result video 52b change accordingly. For example, by moving the pointer P left or right, a frame (image) at a given shooting time or at a given camera position in the video can be searched for (seeked), and the image at the given shooting time or at the given camera position can be displayed in the input video 52a and the processed result video 52b. At this time, information indicating the camera position is displayed in the upper right corner of the input video 52a and the processed result video 52b. The shooting time may also be displayed.

[0198] Below the time bar 52c, there are anomaly type information 52d and a partial judgment bar 52e. In the example of FIG. 11, in the anomaly type information 52d, "rust", "deposits", "inner surface adhesion", "foreign matter adhesion", and "inner surface corrosion prevention" are arranged in order from the top, and each color is defined (however, "floating matter" is omitted). In correspondence with this arrangement, multiple bars corresponding to "rust", "deposits", "inner surface adhesion", "foreign matter adhesion", and "inner surface corrosion prevention" are arranged in the partial judgment bar 52e. The multiple bars arranged for each of these anomaly types are each colored in a different color for each anomaly level at the position on the time axis where the anomaly appears. The vertically long pointer P described above is also movable left and right in the partial judgment bar 52e.

[0199] In the example of Figure 11, pointer P is placed on the bar corresponding to "rust", and the bar is displayed in reddish brown, which indicates that the abnormality level of "rust" at that image time and camera position is "D".

[0200] Therefore, the worker can easily understand which abnormality has occurred at what abnormality level in which frame (image) at which shooting time and camera position in the video by referring to the time bar 52c, abnormality type information 52d, partial judgment bar 52e, and pointer P. Furthermore, the worker can display a frame (image) at any shooting time and camera position in the input video 52a and the processed result video 52b by moving the pointer P by a dragging operation or the like, and visually confirm the abnormality status.

[0201] Below the partial judgment bar 52e is a video playback function 52g. The video playback function 52g is provided with various control buttons related to video playback, and the operator can operate these buttons to play, stop, fast forward, rewind, advance frame, and rewind the input video 52a and the processed result video 52b. The pointer P described above moves left and right in response to video play, stop, fast forward, rewind, advance frame, and rewind.

[0202] Above the input image 52a and the processing result image 52b are overall judgment result information 52l and partial judgment result information 52m. The overall judgment result information 52l is information on the overall judgment result that indicates the abnormality level of the entire image. The partial judgment result information 52m is information that allows the selection of whether or not to display the abnormality rank for each abnormality type, and allows the corresponding area in the processing result image 52b to be colored for the selected abnormality type.

[0203] Each time the screen enlargement buttons 52h and 52i are pressed, the corresponding image is switched between an enlarged state and a normal size state. For example, if the user wishes to check the input image 52a in more detail out of the input image 52a and the processed result image 52b, pressing the screen enlargement button 52h causes the input image 52a to be displayed in an enlarged state. On the other hand, if the user wishes to check the processed result image 52b in more detail out of the input image 52a and the processed result image 52b, pressing the screen enlargement button 52i causes the processed result image 52b to be displayed in an enlarged state.

[0204] The capture execution buttons 52j, 52k are buttons that enable capturing the currently displayed image from the input video 52a or the processed result video 52b and storing it in a specified memory area, as well as reducing the image and displaying it, for example, as a thumbnail image in the thumbnail display area 52o on the right side of the screen shown in FIG. 11.

[0205] For example, if one wishes to check again later an image (frame) in which a typical abnormality is observed in the images 52a, 52b, one can press the capture execution button 52j, 52k to capture the image, record the captured image in a storage device, and check the image later.

[0206] When the capture execution button 52j, 52k is pressed, the currently displayed image is captured, and the captured image is associated with information such as the abnormality types contained in the image, the abnormality levels for each abnormality type, the camera position, and the shooting time (including information on the corresponding video analysis results), and is recorded in a storage device and added to the thumbnail display area 52o on the right side of the screen shown in Figure 11.

[0207] The items additionally displayed in the thumbnail display area 52o include thumbnail images and at least one of the following information: the abnormality type associated with the thumbnail image, the abnormality level for each abnormality type, the camera position, and the image capture time.

[0208] When there are multiple thumbnail images in the thumbnail display area 52o, they are displayed in a list so that they are easy to view. When multiple images of the same video taken at different times are displayed in the thumbnail display area 52o, the images may be arranged in chronological order, in the order in which they were captured, or in the order of the camera position when they were taken. The order of the thumbnail images may be freely changed by the worker. In this case, the thumbnail display area 52o may be provided with a function for freely performing various sorting operations. In this way, the worker can rearrange the order of the thumbnail images in a desired order, making it easy to use.

[0209] When the capture execution button 52j or 52k is pressed to capture an image in the video, a capture icon indicating that the image has been captured is displayed as a mark or symbol in the capture icon area 52f below the partial judgment bar 52e in FIG. 11. The position where the capture icon for the image is displayed indicates the time when the image was taken. A bar-shaped line that passes over the partial judgment bar 52e and extends vertically toward the time bar 52c is displayed above the capture icon. The horizontal axis of the graph in the time bar 52c indicates the time when the image was taken, so that the bar-shaped line and the position of the capture icon can be used to determine the time when the captured image was taken, and each bar of the partial judgment bar 52e can be used to determine what abnormality types are included in the image and what abnormality level each abnormality type indicates.

[0210] In addition, the video playback function 52g is provided with a function that enables the operator to skip through multiple captured frames (images) by referring to the capture icon and bar-shaped line in the capture icon area 52f, and the pointer P.

[0211] In addition to including buttons for play, stop, fast forward, and rewind, the video playback function 52g further includes a button for skipping captured frames (images), and each time a button is pressed, it skips multiple captured frames (images) in the fast forward or rewind direction.

[0212] Further, on the video analysis result confirmation screen 52A, a 2D measurement button 52r, a 3D measurement button 52s, an outward / inward comparison display button 52t, and an outward / inward comparison list display button 52u are arranged.

[0213] Pressing the outward / return comparison display button 52t opens an outward / return comparison screen 53A which uses the currently displayed image (frame) as a reference and displays side-by-side images of the same point on the outward and return routes associated in the image positioning unit 50 for comparison. Pressing the outward / return comparison list display button 52u opens an outward / return comparison list screen 53A' which includes a representative image for the outward route and a representative image for the return route.

[0214] In addition, when the 2D measurement button 52r or the 3D measurement button 52s is pressed, a 2D measurement screen 55A or a 3D measurement screen 56A is opened as a measurement screen for measuring the length, area, and volume of the image displayed on the image analysis result confirmation screen 52A.

[0215] (Outbound / return comparison screen 53A) FIG. 13 shows an example of the outbound / inbound comparison screen 53A.

[0216] In the outward / inward comparison screen 53A, it is possible to display the outward and inward positions of the same image.

[0217] In addition to outbound video 53a and inbound video 53b to be compared, video file information 53c, time bar 53d, video playback function (playback control) 53e, etc. corresponding to each of outbound video 53a and inbound video 53b are displayed on outbound / inbound comparison screen 53A. Time bar 53d and video playback function (playback control) 53e have the same functions as time bar 52c and video playback function 52g in video analysis result confirmation screen 52A described above.

[0218] As shown in FIG. 13, an outbound video 53a is displayed on the left side, and an inbound video 53b is displayed on the right side, with playback controls 53e disposed below each video.

[0219] Since frames corresponding to the outbound journey image 53a and the return journey image 53b are stored in advance in the image positioning unit 50, for example, when the playback control 53e is played on the outbound journey display side on the left, a frame of the outbound journey image 53a is displayed and at the same time a frame of the return journey image 53b corresponding to the frame of the outbound journey image 53a is displayed.

[0220] In addition, when playback control 53e is used to play back on the return road display side on the right side, a frame of the return road side image 53b is displayed, and at the same time, a frame of the outbound road side image 53a corresponding to the frame of the return road side image 53b is displayed.

[0221] In some cases, multiple frames of images may correspond to one frame. In such cases, the multiple frames are played back in frame order.

[0222] (Outbound / return comparison list screen 53A') FIG. 14 shows an example of an outbound / inbound comparison list screen 53A'.

[0223] The outbound journey comparison list screen 53A' makes it possible to display at least one representative image selected from each of the outbound journey image 53a and the inbound journey image 53b side by side for comparison.

[0224] That is, instead of playing back the outbound side image 53a and the return side image 53a on the outbound side / return side comparison screen 53A, a list of several representative images can be displayed on the outbound side / return side comparison list screen 53A'. In this case, several representative images selected from the outbound side image and several representative images selected from the return side image are displayed as an outbound side representative image list 53a' and a return side representative image list 53b', respectively, as shown in FIG.

[0225] For example, in the outbound representative image list 53a', the individual representative images are arranged in the upper row from left to right in the direction in which the camera moves (the direction in which the camera position value increases), and in the inbound representative image list 53b', the individual representative images are arranged in the lower row from right to left in the direction in which the camera moves (the direction in which the camera position value decreases). At this time, the individual representative images in the outbound representative image list 53a' and the individual representative images in the inbound representative image list 53b' are displayed in a list aligned vertically for easy comparison. This makes it easy to compare the associated representative images vertically and to confirm the differences between them.

[0226] FIG. 14 shows examples in which the presence of an anomaly 53p or a pipe joint 53q can be confirmed among some representative images in the return trip representative image list 53b'.

[0227] A mode selection item 53c' is provided at the top right of the outward / return comparison list screen 53A'. The mode selection item 53c' is used to select the representative image display standard (mode). Various modes are provided: "Specified frame", "Manual alignment", "Thinning out alignment", "Prescribed alignment part", and "Detection result of abnormality in prescribed alignment part".

[0228] The representative image list 54a' of each video is arranged and displayed according to a preset representative image display criterion (mode), which determines the selection rule for the representative image. By selecting one of the radio buttons provided in the mode selection item 53c' as a selection criterion for a representative image to be selected from each video, it is possible to select one of the following modes.

[0229] "Specified frame" It is possible to thin out the specified number of frames and arrange them (however, they will not be aligned). The number of frames to thin out can be specified on the screen that opens by pressing the "Detailed Settings" button.

[0230] "Manual alignment" It is possible to display a frame manually selected by a human being for each set of frames associated between multiple videos by the video alignment unit 50. For example, it is possible to align and display a location selected using a capture function on the video analysis result confirmation screen 52A.

[0231] "Alignment thinning" It is possible to display each set of frames, which have been associated between multiple images by the image alignment unit 50, by thinning out the frames by a specified number of frames after aligning them so that their positions inside the water pipe are the same by the image alignment unit 50. The number of frames to be thinned out can be specified on the setting screen that opens by pressing the "Detailed Settings" button.

[0232] "Prescribed alignment area" After aligning the positions inside the water pipe so that they are the same, characteristic locations such as joints (the joints between pipes), the middle of a straight pipe (near the center of the straight part of the pipe), and perforations (the part with holes in the middle of a straight pipe) are selected and displayed. For the specific parts to be displayed, a screen opens when "Detailed Settings" is pressed, and it is possible to set whether or not to display them using check boxes, etc.

[0233] "Alignment specific part abnormality detection result" In addition to the display method of the "predetermined alignment portion," it is possible to select a frame of a location where an abnormality detected in a water pipe exists as a representative image. The type of abnormality to be selected as the representative image, the threshold value of the judgment rank, etc. may be selected on a screen that opens when "Detailed Settings" is pressed.

[0234] (Difference comparison screen 54A) FIG. 15 shows an example of the difference comparison screen 54A.

[0235] The difference comparison screen 54A is a screen that enables the worker to compare multiple images 54a, 54b selected from multiple images managed in the image list 51a of the image list screen 51 A. Here, an example in which two images 54a, 54b are displayed will be described.

[0236] In addition to a plurality of videos 54a and 54b to be compared, the difference comparison screen 54A displays video file information 54c, a time bar 54d, a video playback function 54e, a snapshot display area 54f, a snapshot execution button 54g, a report display button 54h, etc. The time bar 54d and the video playback function 54e have the same functions as the time bar 52c and the video playback function 52g described above.

[0237] The video file information 54c displays the following: [Water pipe name], [Shooting date and time], [Overall judgment] (corresponding to the abnormality rank of the overall judgment result), and [Distance] (corresponding to the camera position). However, it is not necessary to display all of this information, and some of it may not be displayed.

[0238] The two arranged images 54a, 54b can be played back or sought independently. Playback, seeking, etc. can be performed using a time bar 54d and an image playback function 54e. Basically, the images can be displayed side by side for comparison, and the camera position can also be displayed. However, if the image is one for which image analysis has been completed, the overall judgment result and the abnormality detection result may be displayed on the screen. In this case, the screen configuration may be realized in the same manner as the processing result image and partial judgment display on the image analysis result confirmation screen 52A.

[0239] In addition, since the two images 54a and 54b were taken with different camera movements, they do not need to be played back in sync and can be played back separately. The two images 54a and 54b may be displayed as images taken from the same camera position according to the camera position information.

[0240] The difference comparison screen 54A also has a snapshot execution button 54g marked "screenshot." When the snapshot execution button 54g is pressed while images of any shooting time in the two videos 54a, 54b are displayed, a snapshot of the corresponding image is taken and stored, and the image is displayed as a snapshot in the snapshot display area 54f in thumbnail display format. By capturing the two images side by side, the images may be displayed on the report screen, or may be saved as individual image files.

[0241] In addition, when a snapshot image is present, for example, by pressing a button 54h labeled "Go to Report Screen," a report screen is opened and displayed. In the report screen, the snapshot image can be used to create a report.

[0242] In this way, the worker can freely compare multiple images selected from the multiple images managed in the image list 51a on the image list screen 51A as necessary. This makes it easy to perform, for example, time-lapse observation or confirmation work before and after pipe cleaning. Furthermore, the comparison results can be used to create reports.

[0243] Furthermore, on the difference comparison screen 54A, it is possible to simultaneously play back and display a plurality of videos arbitrarily selected from the video list 51a on the video list screen 51A with the individual frames aligned.

[0244] In this example, two videos are displayed side by side, but since the video positioning unit 50 can perform video positioning for three or more videos, not just two, it is also possible to display three or more videos side by side. In this case, it is sufficient to increase the number of time bars 54d and video playback functions (playback controls) 54e according to the number of videos to be displayed.

[0245] Furthermore, on the difference comparison screen 54A, in order to enable the two videos 54a, 54b to be played back and displayed simultaneously, not only are there provided a time bar 54d and a video playback function (playback control) 54e corresponding to each of the two videos 54a, 54b, but also a video alignment display playback checkbox 54i and a video alignment display playback function (playback control) 54j.

[0246] The playback method changes depending on whether the image alignment display playback check box 54i is checked or not, that is, whether it is in the on state or off state. For example, it is possible to switch between the following two.

[0247] On state: video alignment playback is enabled, and the two videos 54a, 54b are simultaneously played back with the individual frames aligned by the video alignment unit 50.

[0248] Off state: Video alignment playback is disabled, and the two videos 54a, 54b are simply played back separately with the number of frames of the videos matched.

[0249] The video alignment display playback function (playback control) 54j can be used when the video alignment display playback checkbox 54i is checked, and the two videos 54a, 54b can be played back simultaneously. Stop, fast forward, rewind, frame forward, and frame reverse are also performed with the frames of the two videos 54a, 54b aligned. In this case, there is no need to use the two video playback functions (playback control) 54e.

[0250] (Difference comparison list screen 54A') FIG. 16 shows an example of the difference comparison list screen 54A'.

[0251] The difference comparison list screen 54A' makes it possible to display at least one or more representative images selected from multiple videos side by side for comparison. In this case, several representative images selected from each video are displayed as a representative image list 54a' for each video, in accordance with a selection rule for the representative images, which will be described later.

[0252] Although it is similar to the difference comparison screen 54A in that each video is displayed with the individual frames aligned, the difference comparison list screen 54A' arranges representative images of each frame for each video horizontally in the time direction, and arranges multiple selected videos vertically, as shown in Fig. 16. In this example, when multiple videos taken at different dates and times of the same water pipe are specified, one or more representative images showing the same position from each video are arranged side by side for comparison.

[0253] FIG. 16 shows an example in which three videos with different shooting dates are selected and arranged from top to bottom in order of shooting date.

[0254] FIG. 16 also shows examples in which the presence of various anomalies 53p, 53r and pipe joints 54q can be confirmed among several representative images in the representative image list 54a' of the video dated September 21, 2021.

[0255] A mode selection item 54b' is provided at the top right of the difference comparison list screen 54A'. The mode selection item 54b' is used to select a representative image display standard (mode). Various modes are provided: "Specified frame", "Manual alignment", "Thinning out alignment", "Prescribed alignment part", and "Detection result of abnormality in prescribed alignment part".

[0256] The representative image list 54a' of each video is arranged and displayed according to a preset representative image display criterion (mode), which determines the selection rule for the representative image.

[0257] By selecting one of the radio buttons provided in mode selection item 54b' as a selection criterion for a representative image to be selected from each video, it is possible to select one of the following modes. Note that the functions of each mode are the same as those explained in the explanation of mode selection item 53c' on outward / inward comparison list screen 53A' in Fig. 14, so explanations thereof will be omitted here.

[0258] "Specified frame" "Manual alignment" "Alignment thinning" "Prescribed alignment area" "Alignment specific part abnormality detection result" (2D measurement screen 55A) FIG. 17 shows an example of the 2D measurement screen 55A.

[0259] The 2D measurement screen 55A enables three-dimensional measurement to be performed from a two-dimensional image of the water pipe 3.

[0260] The 2D measurement screen 55A displays a 2D image 55a, a measurement result display area 55b, a measurement mode display area 55c, an image playback function (playback control) 55d, and the like.

[0261] When the abnormality of the measurement target is displayed on the video analysis result confirmation screen 52A, pressing the 2D measurement button 52r opens the 2D measurement screen 55A shown in FIG. 17 and enables measurement to be performed.

[0262] Since the three-dimensional information acquisition unit 60 has been able to restore the three-dimensional information from the video, the three-dimensional information corresponding to each pixel displayed on the screen has already been stored in a predetermined storage area.

[0263] When the "measurement mode" is set to "length" in the measurement mode display area 55c, the "length" can be measured by calculating the distance between the three-dimensional positions corresponding to the pixels at both ends of the line segment 55e specified by dragging the mouse on the screen, for example.

[0264] When "Measurement mode" is set to "Area" in the measurement mode display area 55c, specify a rectangular area by dragging the mouse, or specify a polygon area by clicking the mouse. For example, you can specify the initial starting point by clicking, then the next click to connect the line segment 55e down to the location below to form a polygon, and end the line when you have specified the vicinity of the starting point to specify the end point. You can also specify the end point by double-clicking or right-clicking. If the end point is different from the starting point, the end point and starting point can be automatically connected.

[0265] The area of ​​a solid object existing within a rectangle or polygon may be calculated by calculating the area on a 2D image, or the largest area on a 3D point cloud. Accumulation can also be measured using the same method as for area.

[0266] The measurement results of the area specified by these means are displayed under "Measurement results" in the measurement result display area 55b.

[0267] In addition, a playback control may be provided below the image so that the user can move to the previous or next frame without returning to the video analysis result confirmation screen 52A.

[0268] As a result, it becomes possible to measure the object in the displayed image. Note that, since the contents of the 2D measurement screen 55A are also displayed on the video analysis result confirmation screen 52A, the control and mode selection parts may be displayed in the video analysis result confirmation screen 52A without opening a separate window.

[0269] (3D measurement screen 56A) FIG. 18 shows an example of the 3D measurement screen 56A.

[0270] The 3D measurement screen 56A enables three-dimensional measurement to be performed from a three-dimensional point cloud image of the water pipe 3.

[0271] The 3D measurement screen 56A displays a 3D image 56a, a measurement result display area 56b, an operation mode display area 56c, measurement mode information 56d, and the like.

[0272] When an abnormality to be measured is displayed on the video analysis result confirmation screen 52A, pressing the "3D measurement" button opens a 3D measurement screen 56A shown in FIG. 18, making it possible to perform measurement.

[0273] Since the three-dimensional information acquisition unit 60 has been able to restore the three-dimensional information from the video, the three-dimensional information corresponding to each pixel displayed on the screen has already been stored in a predetermined storage area.

[0274] In addition, when changing the view of the 3D point cloud, similar mouse operations are required for measuring length, etc., so a control to switch the "operation mode" between "measurement" and "view change" can also be placed on the screen so that the mouse operation can be switched between view change and measurement.

[0275] Here, an example of operation when the "operation mode" is set to "view change" in the operation mode display area 56c will be described.

[0276] Point clouds can be translated, rotated, and scaled using the mouse. Also, by changing the "Operation Mode" to "Measurement," measurements can be made using 3D point clouds.

[0277] When the “measurement mode” is specified as “length” in the measurement mode information 56d, the “length” can be measured by calculating the distance between the three-dimensional positions corresponding to the pixels at both ends of the line segment 56e specified by dragging the mouse on the screen, for example.

[0278] When "measurement mode" is set to "area" in the measurement mode information 56d, a rectangular area is specified by dragging the mouse, or a polygon area is specified by clicking the mouse. For example, the first starting point is a click, the next click is connected to the lower location of the line segment 56e to form a polygon, and the end point can be specified by ending when the vicinity of the starting point is specified. The end point can also be specified by double-clicking or right-clicking. If the end point is different from the starting point, the end point and the starting point can be automatically connected.

[0279] The area of ​​a solid object existing within a rectangle or polygon may be calculated by calculating the area on a 2D image, or the largest area on a 3D point cloud. Accumulation can also be measured using the same method as for area.

[0280] The measurement results of the area specified by these means are displayed under "Measurement Results" in the measurement result display area 56b.

[0281] <Basic operation> Next, an example of the operation of the inside of a pipe inspection device according to the embodiment will be described with reference to the flowchart in Fig. 19. Note that the individual processes shown in Fig. 19 are not necessarily performed in the order of the step numbers, and can be changed as appropriate.

[0282] The video input unit 10 inputs video data captured by the camera 4 while moving inside the water pipe 3, and the attribute information input unit 11 inputs attribute information indicating attributes associated with the camera 4 (step S1).

[0283] Next, the camera position / movement detection unit 20 detects the position and movement of the camera 4 within the pipe using the video data input by the video input unit 10 and the attribute information input by the attribute information input unit 11 (step S2).

[0284] Next, the abnormality detection unit 30 selects one or more images from the video data of the detection targets captured by the camera 4 according to the position and movement of the camera 4 detected by the camera position / movement detection unit 20, and judges the presence or absence and the abnormality level of each detection target in the pipe using the selected images, and generates the judgment result as an abnormality detection result for each detection target (step S3). The integrated judgment unit 40 integrates the abnormality detection results generated for each detection target by the abnormality detection unit 30, and generates an integrated judgment result indicating the level of abnormality in the pipe (step S4).

[0285] On the other hand, the image alignment unit 50 uses the image input by the image input unit 10 and the position and movement information of the camera 4 detected by the camera position and movement detection unit 20 to perform alignment for a plurality of images taken by a camera moving inside the pipe, or a plurality of images constituting one image, by matching images (frames) included in each of the plurality of images that were taken at different times and the same positions (step S5). The three-dimensional information acquisition unit 60 restores three-dimensional information from the image input by the image input unit 10 (step S6).

[0286] The information management unit 70 manages information on the position and movement of the camera 4 detected by the camera position / movement detection unit 20, information including the abnormality detection result generated for each detection target by the abnormality detection unit 30 and the integrated judgment result generated by the integrated judgment unit 40, information including the video alignment result generated by the video alignment unit 50, information including the three-dimensional information restored by the three-dimensional information acquisition unit 60, and attribute information input by the attribute information input unit 11 (step S7).

[0287] The output processing unit 80 receives the operations of the user, and outputs various types of information managed by the information management unit 70 to the display device according to the operations of the user (step S8). The output processing unit 80 not only outputs the abnormality detection results and integrated determination results corresponding to the inspection results, but also outputs various types of information related to various display screens (video list screen 51A, video analysis result confirmation screen 52A, forward and return path comparison screen 53A, forward and return path comparison list screen 53A', difference comparison screen 54A, difference comparison list screen 54A', 2D measurement screen 55A, 3D measurement screen 56A, etc.) realized by the UI according to the operations, as shown in the flowchart of FIG. 20 described later.

[0288] <Operations related to the UI> Next, with reference to the flowchart of FIG. 20, an example of the operations related to the UI by the output processing unit 80 will be described. Here, it is described assuming that the video analysis process has already been completed.

[0289] First, a video list 51a for managing various videos obtained by photographing the inside of the pipe with the camera 4 is displayed on the video list screen 51A. When there is any operation from the operator on the displayed video list screen 51A, the process corresponding to the operation is performed (step S11).

[0290] Here, for example, when an operation (for example, a click or tap operation by the input device) of pressing a video analysis display button 51e marked "To the video analysis screen" is performed in a state where one video is selected from the video list 51a on the video list screen 51A (YES in step S12), the process proceeds to step S19.

[0291] On the other hand, for example, when multiple videos are selected on the video list screen 51A and an operation is performed to press the difference comparison display button 51g marked "To difference comparison screen" (NO in step S12, YES in step S13), the difference comparison screen 54A is opened and displayed, making it possible to check the differences while comparing the selected multiple input videos 54a, 54b. If the operator performs some operation on the displayed difference comparison screen 54A, a process corresponding to the operation is performed (step S14).

[0292] Furthermore, when multiple videos are selected on the video list screen 51A and an operation is performed to press the difference comparison display button 51g marked "To difference comparison list screen" (NO in step S13, YES in step S15), the difference comparison list screen 54A' is opened and displayed, and representative images of the selected multiple input videos 54a, 54b are displayed side by side, making it possible to compare them. If the operator performs some operation on the displayed difference comparison list screen 54A', a process corresponding to the operation is performed (step S16).

[0293] Moreover, when an operation other than the above is performed on the video list screen 51A (NO in step S15, YES in step S17), a process corresponding to the operation is performed (step S18).

[0294] In step S12, when one video is selected from the video list 51a on the video list screen 51A, for example, if an operation of pressing the video analysis display button 51e labeled "To video analysis screen" (for example, a click or tap operation using an input device) is performed (YES in step S12), the video analysis result confirmation screen 52A is opened and displayed, and it becomes possible to perform confirmation work by comparing the input video 52a with the processing result video 52b. If the operator performs some operation on the displayed video analysis result confirmation screen 52A, processing according to the operation is performed (step S19).

[0295] When the outward / inward comparison display button 52t is pressed on the video analysis result confirmation screen 52 (YES in step S20), the inward / inward comparison screen 53A is opened and displayed, and images associated with the outward and inward journeys are displayed side by side for comparison with respect to the video displayed on the video analysis result confirmation screen 52A. When the operator performs some operation on the displayed inward / inward comparison screen 53A, a process corresponding to the operation is performed (step S21).

[0296] Furthermore, when an operation of pressing the outward journey / return journey comparison list display button 52u is performed on the video analysis result confirmation screen 52 (NO in step S20, YES in step S22), an outward journey / return journey comparison list screen 53A' is opened and displayed, and representative images of each video associated with the outward journey and the return journey for each video displayed on the video analysis result confirmation screen 52A are displayed side by side, making it possible to compare them. When an operator performs some operation on the displayed outward journey / return journey comparison list screen 53A', a process corresponding to the operation is performed (step S23).

[0297] Furthermore, when the 2D measurement button 52r is pressed on the video analysis result confirmation screen 52 (NO in step S22, YES in step S24), the 2D measurement screen 55A is opened and displayed, making it possible to measure the length, area, and volume of an object in the image. When the operator performs some operation on the displayed 2D measurement screen 55A, a process corresponding to the operation is performed (step S25).

[0298] Furthermore, when the 3D measurement button 52s is pressed on the video analysis result confirmation screen 52 (NO in step S24, YES in step S26), the 3D measurement screen 56A is opened and displayed, making it possible to measure the length, area, and volume of an object in the displayed 3D point cloud image. When the operator performs some operation on the displayed 3D measurement screen 56A, processing according to the operation is performed (step S27).

[0299] Moreover, if an operation other than the above is performed on video analysis result confirmation screen 52 (NO in step S26, YES in step S28), a process corresponding to that operation is performed (step S29).

[0300] The above-described processing enables the efficiency of work to be improved by performing appropriate processing according to the operator's operations on the various display screens displayed by the inside-pipe inspection device.

[0301] According to the embodiment, by aligning the individual frames that make up the video, it is possible to efficiently check the deterioration state of the target pipe, compare before and after cleaning, and check the movement of foreign objects by moving the camera to efficiently check the behavior of accumulated objects, etc. In addition, the measurement function makes it possible to efficiently measure the length, area, and volume of objects in the water pipe 3 through the UI without using special equipment, placing a reference object within the field of view of the camera, or separately preparing a model made of a known size in advance.

[0302] As described above in detail, according to the embodiment, it is possible to efficiently inspect the inside of a pipe. [Explanation of symbols]

[0303] 1...pipe inspection equipment, 2...site, 3...water pipe, 4...camera, 5...recording medium, 10...video input section, 11...attribute information input section, 12...setting section, 20...camera position / movement detection section, 30...abnormality detection section, 40...integrated judgment section, 50...video alignment section, 60...3D information acquisition section, 70...information management section, 80...output processing section, 51A...video list screen, 52A...video analysis result confirmation screen, 53A...outbound comparison screen, 53A'...outbound comparison list screen, 54A...difference comparison screen, 54A'...difference comparison list screen, 55A...2D measurement screen, 56A...3D measurement screen.

Claims

1. an image registration unit that performs registration for a plurality of images captured by a camera moving inside a pipe, or a plurality of image portions constituting one image, by associating images that are captured at different times and at the same positions among individual images included in the plurality of images or the plurality of image portions; an output processing unit that displays the plurality of images that have been subjected to the position alignment side by side on a display device so that the images can be compared; the output processing unit simultaneously displays the images captured at different times and at the same position. Pipe inspection equipment.

2. The output processing unit displays, as the plurality of image portions, an image portion of an outward path taken by the camera while moving along an outward path in the pipe and an image portion of a return path taken by the camera while moving along a return path in the opposite direction to the outward path, side by side on a display device so as to be compared. The pipe inspection device according to claim 1 .

3. the output processing unit displays one or more representative images selected as representative images from among the individual images included in each of the outbound journey video portion and the return journey video portion, the representative images being taken at different times and at the same positions, side by side so that they can be compared with each other; The pipe inspection device according to claim 2.

4. The output processing unit displays, as the plurality of images, a plurality of images captured by the same tube at different shooting time periods, side by side on a display device so that the images can be compared. The pipe inspection device according to claim 1 .

5. the output processing unit displays one or more representative images selected as representatives from among individual images included in each of the multiple images captured by the same tube at different shooting times, the representative images being captured at different shooting positions at different times, side by side so that they can be compared; The pipe inspection device according to claim 4.

6. Further comprising a three-dimensional information acquisition unit that restores three-dimensional information from an image of the inside of the pipe, the output processing unit measures at least one of a length, an area, and a volume of an object designated by a user on an image based on the three-dimensional information, and displays a measurement result. The pipe inspection device according to claim 1 .

7. The output processing unit includes: displaying an image that allows a user to specify a measurement range of the object; The pipe inspection device according to claim 6.

8. The output processing unit includes: displaying an image based on the three-dimensional information that enables a user to specify a measurement range of the object; The pipe inspection device according to claim 6.

9. A video positioning unit performs positioning for a plurality of videos captured by a camera moving inside the pipe, or a plurality of video portions constituting one video, by associating images captured at different times and at the same positions among individual images included in the plurality of videos or the plurality of video portions; an output processing unit that displays the aligned images or image portions on a display device in a line for comparison, and simultaneously displays the images captured at different times and at the same positions; An in-pipe inspection method comprising:

10. On one or more computers, A function for performing registration of multiple images captured by a camera moving inside a pipe, or multiple images constituting a single image, by matching images captured at different times and in the same positions among the individual images contained in each of the multiple images or multiple image portions; a function of displaying the plurality of images or the plurality of image portions that have been subjected to the positional alignment on a display device in a line for comparison, and simultaneously displaying the images that were taken at different times and at the same positions; A program to achieve this.

Images