Image management system and image management method

Abstract

To provide an image management technique capable of easily managing each image in a case of imaging the same subject with a plurality of cameras having different field angles.SOLUTION: An image management system 1 comprises a management computer 3 which manages each image captured by a first camera 4 and a second camera 5 that can perform imaging in a wider field angle than the first camera 4. The management computer 3 acquires a first image 13 captured by the first camera 4 and a second image 14 captured by the second camera 5, records the first image 13 in association with a coordinate position being the coordinate position in the second image 14 and matching with the imaging direction of the first camera 4 when the imaging direction of the first camera 4 is included in an imaging range of the second camera 5, and displays the first image 13 recorded in association with the selected coordinate position when an arbitrary coordinate position of the second image 14 is selected.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] FIELD OF THE INVENTION Embodiments of the present invention relate to image management techniques. [Background technology]

[0002] At power plants and other facilities, periodic patrol inspections are conducted to manage construction progress or detect abnormalities. During patrol inspections, cameras are used to capture images of the inspection locations. When using a camera with a fisheye lens, it is possible to capture a wide range of images, making it possible to fit multiple inspection locations into a single image. It is also possible to use multiple cameras with fisheye lenses to capture images in all directions. On the other hand, when capturing detailed images of inspection locations, the camera is often positioned close to the inspection location or the camera's zoom function is used. The photographed inspection locations are managed by recording them on paper or electronically on a tablet PC or other device. Inspectors analyze the captured images to detect abnormalities such as damage, deformation, cracks, discoloration, and internal fluid leaks. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-192057 Summary of the Invention [Problem to be solved by the invention]

[0004] When detecting abnormalities at inspection points from captured images, it is desirable to zoom in on the inspection points to obtain more detailed images. However, when multiple inspection points have similar shapes, it can be difficult to identify which inspection point appears in the captured image. In particular, when the same subject is photographed using a wide-angle image taken with a camera having a wide-angle or fisheye lens and a detailed image taken with a camera having a zoom function, there is a demand for easier management of each image.

[0005] The embodiment of the present invention has been made in consideration of these circumstances, and aims to provide an image management technology that makes it easy to manage each image when photographing the same subject with multiple cameras with different angles of view. [Means for solving the problem]

[0006] An image management system according to an embodiment of the present invention includes a patrolling robot equipped with a first camera and a second camera capable of capturing images at a wider angle of view than the first camera, and a management computer that manages images captured by the first camera and the second camera, the relative positions of the first camera and the second camera being fixed at a certain distance apart, and the management computer Multiple times at different angles a first image captured by the first camera and a second image captured by the second camera, and when the shooting direction of the first camera is included in the shooting range of the second camera, the first image is recorded in association with a coordinate position in the second image that coincides with the shooting direction of the first camera; In front of Coordinate position The coordinate position where the mark is displayed is selected, the first image recorded in association with the selected coordinate position is displayed. [Effects of the Invention]

[0007] According to an embodiment of the present invention, an image management technique is provided that allows easy management of images taken of the same subject with multiple cameras having different angles of view. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing the overall configuration of an image management system. [Figure 2] FIG. 2 is a block diagram showing a management computer. [Figure 3] FIG. [Figure 4] FIG. 2 is an explanatory diagram showing the positional relationship of detailed images in an omnidirectional image. [Figure 5] FIG. 10 is an explanatory diagram showing a manner in which corresponding points are set on an omnidirectional image. [Figure 6] FIG. 2 is an explanatory diagram showing the relationship between the pan-tilt angle of the camera and the coordinates of the omnidirectional image. [Figure 7] FIG. 2 is a screen diagram showing the display mode of an image in the image management system. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of an image management system and an image management method will be described in detail with reference to the drawings.

[0010] Reference numeral 1 in FIG. 1 denotes an image management system according to this embodiment. This image management system 1 is used, for example, to photograph subjects such as equipment or structures installed at a specified inspection site and to perform various inspections based on the photographed images. The image management system 1 mainly photographs images at the inspection site and manages the large number of photographed images. Examples of inspection sites include power plants, chemical plants, and factories. These plants are home to a large number of equipment or structures to be inspected.

[0011] The image management system 1 includes a patrol inspection robot 2 and a management computer 3. The patrol inspection robot 2 includes a pan-tilt-zoom camera 4 as a first camera, an omnidirectional camera 5 as a second camera, a shooting direction detection unit 6, a distance measurement unit 7, a position detection unit 8, a traveling device 9, and a control computer 10 that controls these components in an integrated manner. The control computer 10 is connected to the management computer 3.

[0012] The control computer 10 and the management computer 3 have hardware resources such as a CPU, ROM, RAM, and HDD, and the CPU executes various programs to realize software-based information processing using the hardware resources. Furthermore, the image management method of this embodiment is realized by having the computer execute various programs.

[0013] The patrol inspection robot 2 travels within the inspection site along a preset route and takes photographs of the inspection site. The images taken by this patrol inspection robot 2 are sent to the management computer 3. An inspection worker (user) can inspect the equipment or structures at the inspection site by checking the images stored in the management computer 3.

[0014] Still images are exemplified as images managed by the management computer 3. Note that the images may include moving images, or may be a mixture of both still images and moving images.

[0015] This patrol inspection robot 2 will be described with reference to Fig. 3. Note that the left side of the paper in Fig. 3 will be described as the front side (forward side), which is the direction in which the patrol inspection robot 2 moves.

[0016] The patrol inspection robot 2 is capable of moving on the ground by a traveling device 9 having wheels 11. The patrol inspection robot 2 moves, for example, by autonomous control. Note that the patrol inspection robot 2 may also move by remote control by an inspection worker.

[0017] A pan-tilt-zoom camera 4 serving as a first camera is provided on top of the patrol inspection robot 2. This pan-tilt-zoom camera 4 has a pan function that allows it to swivel horizontally, a tilt function that allows it to swivel vertically, and a zoom function that allows it to zoom in (telephoto) and out (wide-angle). This pan-tilt-zoom camera 4 can capture detailed images of subjects such as equipment or structures at the inspection site. A detailed image 13 (FIG. 7) captured by this pan-tilt-zoom camera 4 serves as the first image in this embodiment.

[0018] The shooting direction detection unit 6 (FIG. 1) detects the shooting direction of the pan-tilt-zoom camera 4. The shooting direction is the orientation of the pan-tilt-zoom camera 4, and corresponds to the center position of the detailed image 13.

[0019] This shooting direction detection unit 6 is composed of, for example, a plurality of sensors provided on the movable parts of the pan-tilt-zoom camera 4. Note that the shooting direction detection unit 6 may also be composed of a plurality of motors that move the pan-tilt-zoom camera 4. For example, the orientation of the pan-tilt-zoom camera 4 is determined by the number of rotations (rotation angle) of the motor. The shooting direction of the pan-tilt-zoom camera 4 detected by the shooting direction detection unit 6 is input to the control computer 10.

[0020] Shooting direction information 15 (FIG. 2) indicating the shooting direction of pan-tilt-zoom camera 4 is sent from control computer 10 to management computer 3. This shooting direction information 15 may be, for example, a detection signal output from a sensor (shooting direction detection unit 6) that detects the shooting direction, or a control signal output from control computer 10 that controls the shooting direction. In this way, even if the shooting direction of pan-tilt-zoom camera 4 changes, management computer 3 can identify the new shooting direction.

[0021] An omnidirectional camera 5 serving as a second camera is provided near the pan-tilt-zoom camera 4 (first camera). The relative positions of the pan-tilt-zoom camera 4 and the omnidirectional camera 5 are fixed with a certain distance between them. The pan-tilt-zoom camera 4 is also provided at a slightly higher position than the omnidirectional camera 5. In this way, the shooting range of the pan-tilt-zoom camera 4 does not include the omnidirectional camera 5, and it does not get in the way.

[0022] The omnidirectional camera 5 (second camera) is capable of capturing images at a wider angle of view than the pan-tilt-zoom camera 4 (first camera). For example, the angle of view of the omnidirectional camera 5 is wider than that of the pan-tilt-zoom camera 4 when it is zoomed out.

[0023] This omnidirectional camera 5 has a wide-angle lens or a fisheye lens and is capable of capturing images in all directions. In this way, the omnidirectional camera 5 can capture images of a wide range. The omnidirectional camera 5 can then determine the location of the subject captured by the pan-tilt-zoom camera 4.

[0024] For example, the omnidirectional camera 5 is equipped with two wide-angle cameras 12, each having an image sensor with a fisheye lens. These wide-angle cameras 12 can capture a wide range around the omnidirectional camera 5.

[0025] The two wide-angle cameras 12 are arranged back to back, so that they can simultaneously capture the scenery in front and behind. In other words, the omnidirectional cameras 5 can be used to simultaneously capture an omnidirectional image 14 ( FIG. 7 ), which is a 360-degree panoramic image capturing the patrol inspection robot 2 in all directions, up, down, left, and right. The omnidirectional image 14 captured by the omnidirectional cameras 5 serves as the second image in this embodiment. Note that the second image does not have to be the omnidirectional image 14, and may simply be a wide-angle image with a wider angle of view than the detailed image 13.

[0026] In this embodiment, two wide-angle cameras 12 are exemplified as the omnidirectional camera 5 (second camera), but a single camera may be used to configure the omnidirectional camera 5. For example, a convex mirror or the like may be used to guide the surrounding scenery to one camera, and the omnidirectional image 14 may be captured. This article explains:

[0027] With an omnidirectional camera 5, it is possible to capture omnidirectional images with a single omnidirectional camera 5 without changing its orientation at the inspection site. Alternatively, two or more cameras each having one image sensor may be used, and the omnidirectional image 14 may be captured by combining images captured by these cameras.

[0028] The omnidirectional image 14 is, for example, a spherical image. Note that the omnidirectional image 14 does not have to be a spherical image, and may be an image that covers at least a 360-degree range in the horizontal direction (in all directions).

[0029] A distance measurement unit 7 is provided near the omnidirectional camera 5. This distance measurement unit 7 measures the distance from the omnidirectional camera 5 to the subject. If the relative positional relationship between the pan-tilt-zoom camera 4 (first camera) and the omnidirectional camera 5 (second camera) is determined in advance, the distance from the pan-tilt-zoom camera 4 to the subject can be estimated based on the distance from the omnidirectional camera 5 to the subject. In other words, the distance measurement unit 7 measures the distance from at least one of the pan-tilt-zoom camera 4 and the omnidirectional camera 5 to the subject.

[0030] Distance measurement unit 7 is composed of a laser rangefinder or the like that can change direction in conjunction with the pan-tilt mechanism (pan-tilt function) of pan-tilt-zoom camera 4. Distance measurement unit 7 may also be a device that measures the distance to the subject based on focus information from a predetermined camera that has a pan-tilt mechanism.

[0031] The distance measurement unit 7 may measure the distance to the object by laser scanning the inspection site using an infrared sensor or a laser sensor such as LiDAR. For example, the distance measurement unit 7 can measure the distance to the object by projecting a laser beam onto the object and receiving the reflected light with a light receiving element. The distance measurement unit 7 can also measure the distance to surrounding objects using a ToF (Time of Flight) method, which converts the delay time between a projected light pulse and a received light pulse into distance.

[0032] Furthermore, a position detection unit 8 is provided on the top of the patrol inspection robot 2. This position detection unit 8 detects the current position (photographing position) of the patrol inspection robot 2. This position detection unit 8 detects its own position using, for example, a satellite positioning system, radar, or the like.

[0033] The position detection unit 8 may detect its own position using radio signals used for transmitting and receiving information. For example, the position detection unit 8 may acquire the position information 16 using wireless communication. The position detection unit 8 may also acquire the position information 16 using a PDR that uses an acceleration sensor or the like.

[0034] The position detection unit 8 may also detect its own position using known techniques such as SLAM (Simultaneous Localization and Mapping), SfM (Structure from Motion), etc. Furthermore, the position detection unit 8 may also detect its own position using predetermined position markers placed on the ground.

[0035] The omnidirectional camera 5 (second camera) can capture a wide range of images, making it easy to determine the location of an inspection target (subject), such as equipment or a structure. However, it has the drawback of being unable to capture detailed images of the inspection target. On the other hand, the pan-tilt-zoom camera 4 (first camera) can capture only the inspection target using a zoom function when facing the direction of the inspection target. This provides high-resolution images, but the resulting images are enlarged, making it difficult to determine the location of the inspection target at the inspection site (e.g., inside a building), placing a burden on the inspection worker (user) who manages the images. In particular, it is desirable to associate and manage the positional relationship of the inspection target captured in the omnidirectional image 14 (second image) and the detailed image 13 (first image), thereby making it easier to manage the inspection target. This embodiment can solve this problem.

[0036] 4, when the shooting direction of pan-tilt-zoom camera 4 (first camera) is included in the shooting range of omnidirectional camera 5 (second camera), management computer 3 records (registers) detailed image 13 (first image) in association with a coordinate position in omnidirectional image 14 (second image) that coincides with the shooting direction of pan-tilt-zoom camera 4. When any coordinate position in omnidirectional image 14 is selected, management computer 3 displays detailed image 13 that is recorded in association with the selected coordinate position.

[0037] For example, as shown in Fig. 7, assume that there are many meters 28 and valves 29 at an inspection site. Here, the omnidirectional image 14 is a photograph of a wide area in which many meters 28 and valves 29 are visible. On the other hand, the detailed image 13 is a photograph of a narrow area in which each meter 28 and valve 29 is visible.

[0038] When the management computer 3 displays the omnidirectional image 14 on the display, it displays predetermined marks 27 near the meters 28 and valves 29 to be inspected. The inspection worker (user) selects any mark 27 using the mouse cursor 30. The management computer 3 displays the detailed image 13 corresponding to the selected mark 27 on the display. For example, even if the value on a meter 28 cannot be read in the omnidirectional image 14, the inspection worker can select the mark 27 corresponding to that meter 28 to display the detailed image 13 in which the meter 28 is enlarged. In this way, the value on the meter 28 becomes readable.

[0039] Next, the system configuration of the management computer 3 will be described with reference to the block diagram shown in Figure 2. Note that the arrows in Figure 2 are an example of the process flow, and there may be other process flows besides the arrows. Furthermore, the order of each process is not necessarily fixed, and the order of some processes may be reversed. Furthermore, some processes may be executed in parallel with other processes. Furthermore, the management computer 3 may include components other than those shown in Figure 2, or some of the components shown in Figure 2 may be omitted.

[0040] The management computer 3 receives the detailed image 13 (first image) and the omnidirectional image 14 (second image), as well as the imaging direction information 15, the position information 16, and the distance information 17.

[0041] The shooting direction information 15 is information indicating the shooting direction of the pan-tilt-zoom camera 4 (first camera) detected by the shooting direction detection unit 6 (FIG. 1). For example, the shooting direction information 15 includes angle information indicating the orientation of the pan-tilt-zoom camera 4, such as the pan angle or tilt angle.

[0042] It should be noted that the shooting direction information 15 does not necessarily have to be input from the control computer 10 to the management computer 3. For example, the management computer 3 may estimate the pan angle or tilt angle based on the operation of the pan-tilt mechanism of the pan-tilt-zoom camera 4 that appears in the omnidirectional image 14 captured by the omnidirectional camera 5. In this manner, the shooting direction information 15 may be acquired.

[0043] Position information 16 (FIG. 2) is the position of the patrol inspection robot 2 detected by the position detection unit 8, and is information indicating the photographing position. Distance information 17 (FIG. 2) is the distance to the subject measured by the distance measurement unit 7, and is information indicating the distance from at least one of the pan-tilt-zoom camera 4 and the omnidirectional camera 5 to the subject.

[0044] The management computer 3 includes a first image acquisition unit 18, a second image acquisition unit 19, a distortion correction unit 20, a shooting position estimation unit 21, and a display control unit 22. These are realized by the CPU executing programs stored in the memory or HDD.

[0045] Furthermore, the management computer 3 includes an image database 23. This image database 23 is a collection of information stored in a memory, a HDD, or the cloud, and organized so that it can be searched or accumulated.

[0046] The components of the management computer 3 do not necessarily have to be installed on a single computer. For example, one management computer 3 may be realized by multiple computers connected to each other via a network. For example, the image database 23 may be installed on a separate computer.

[0047] Although not specifically shown, the management computer 3 includes an input unit and an output unit. Predetermined information is input into the input unit in response to user operations. This input unit includes input devices such as a mouse or keyboard. In other words, predetermined information is input into the input unit in response to operations of these input devices.

[0048] The output unit outputs predetermined information. The management computer 3 includes a device for displaying images, such as a display. The display may be separate from the computer main body or may be integrated with the computer. Additionally or alternatively, the management computer 3 may control images displayed on displays of other computers connected via a network.

[0049] The first image acquisition unit 18 acquires, from the control computer 10 (FIG. 1), a detailed image 13 (first image) captured by the pan-tilt-zoom camera 4 (first camera), and also acquires shooting direction information 15 indicating the shooting direction of the pan-tilt-zoom camera 4 when the detailed image 13 was captured. In this way, the shooting direction of the pan-tilt-zoom camera 4 can be acquired, and based on this, the corresponding coordinate position in the omnidirectional image 14 can be identified.

[0050] Note that if the shooting direction of the pan-tilt-zoom camera 4 can be estimated based on the subject appearing in the detailed image 13, the management computer 3 does not need to acquire the shooting direction information 15 from the control computer 10. For example, the management computer 3 may generate the shooting direction information 15 based on the detailed image 13.

[0051] The second image acquisition unit 19 acquires the omnidirectional image 14 (second image) captured by the omnidirectional camera 5 (second camera) from the control computer 10 (FIG. 1).

[0052] The distortion correction unit 20 performs image processing to correct distortion in the omnidirectional image 14 acquired by the second image acquisition unit 19. For example, the distortion correction unit 20 performs processing to correct an image distorted by a fisheye lens or the like into an image without distortion similar to the detailed image 13.

[0053] The photographing position estimation unit 21 acquires the detailed image 13 , the omnidirectional image 14 , the position information 16 , and the distance information 17 .

[0054] The photographing position estimation unit 21 performs processing to estimate which coordinate position in the omnidirectional image 14 (second image) photographed by the omnidirectional camera 5 (second camera) corresponds to a detailed image 13 (first image) photographed in a predetermined direction by the pan-tilt-zoom camera 4 (first camera). In other words, when the photographing direction of the pan-tilt-zoom camera 4 (for example, the center position of the photographed image) is included in the photographing range of the omnidirectional camera 5, the photographing position estimation unit 21 performs processing to associate the detailed image 13 with a coordinate position in the omnidirectional image 14 that matches the photographing direction of the pan-tilt-zoom camera 4.

[0055] For example, a pan-tilt-zoom camera 4 is used in advance to take multiple images at different pan and tilt angles. Then, detailed images 13 acquired by these images are associated with coordinate positions in an omnidirectional image 14 taken by the omnidirectional camera 5. This association process may be performed automatically by the management computer 3 or manually by the user.

[0056] Using FIG. 5, an aspect of setting by manual operation by a user will be described. First, the user selects, for example, a partial range 24 included in the shooting range of the omnidirectional image 14 and estimated to include the shooting direction of the pan-tilt-zoom camera 4. The user manually sets corresponding points 25 on the omnidirectional image 14 while checking the center position (shooting direction) of the detailed image 13 captured by the pan-tilt-zoom camera 4. These corresponding points 25 are set, for example, at positions shifted by one degree each in the pan angle and tilt angle of the pan-tilt-zoom camera 4. Furthermore, grid lines 26 are set that connect the corresponding points 25 and interpolate the coordinates between the corresponding points 25. Coordinates can be converted based on these corresponding points 25 and grid lines 26.

[0057] By performing such processing in advance, it becomes possible to convert the coordinate position of the omnidirectional image 14 from the pan angle and tilt angle of the pan-tilt-zoom camera 4. It also becomes possible to convert the pan angle and tilt angle of the pan-tilt-zoom camera 4 from the coordinate position of the omnidirectional image 14. A conversion table or conversion formula that enables these conversions is generated.

[0058] For example, as shown in FIG. 6, there is a first coordinate system (coordinate system on the left side of the paper in FIG. 6) with the pan angle of the pan-tilt-zoom camera 4 on the horizontal axis (p-axis) and the tilt angle on the vertical axis (t-axis), and there is a second coordinate system (coordinate system on the right side of the paper in FIG. 6) with the horizontal direction of the omnidirectional image 14 on the horizontal axis (x-axis) and the vertical direction of the omnidirectional image 14 on the vertical axis (y-axis).

[0059] Here, suppose there are four known corresponding points d that correspond to the corners of any square in the first coordinate system. Four known corresponding points b that correspond to these known corresponding points d also exist in the second coordinate system. The respective known corresponding points b and d are associated in advance. Then, once any corresponding point d is determined in the first coordinate system, one corresponding point b is identified in the second coordinate system. Furthermore, once any corresponding point b is determined in the second coordinate system, one corresponding point d is identified in the first coordinate system. A conversion table or conversion formula that allows such coordinate conversion is generated in advance.

[0060] The shooting position estimation unit 21 performs processing to convert the shooting direction information 15 indicating the shooting direction of the pan-tilt-zoom camera 4 into the coordinate position of the omnidirectional image 14 based on at least one of a conversion table and a conversion formula. In this way, the processing to convert the shooting direction of the pan-tilt-zoom camera 4 into the coordinate position of the omnidirectional image 14 can be easily performed.

[0061] If the distances from the pan-tilt-zoom camera 4 (first camera) and the omnidirectional camera 5 (second camera) to the subject are different, the spatial coordinates centered on the pan-tilt-zoom camera 4 will differ from the spatial coordinates centered on the omnidirectional camera 5. For this reason, the conversion table or conversion formula is configured to enable conversion even under conditions where the distance to the subject is different.

[0062] The image database 23 records (stores) the detailed images 13 (first images) and the omnidirectional images 14 (second images) that have been associated by the photographing position estimation unit 21. The management computer 3 may analyze the detailed images 13 stored in the image database 23 and automatically perform processing to detect abnormalities in the inspection target based on the analysis results.

[0063] When a user selects any coordinate position of the omnidirectional image 14 (second image) displayed on the display, the display control unit 22 performs a process of displaying the detailed image 13 (first image) recorded in the image database 23 on the display in association with the selected coordinate position.

[0064] For example, the display control unit 22 performs processing to display a mark 27 whose coordinates can be selected in the omnidirectional image 14 captured by the omnidirectional camera 5, based on the coordinate information estimated by the photographing position estimation unit 21. When the inspection worker (user) selects the mark 27 in the omnidirectional image 14 displayed on the display, the display control unit 22 performs control to display the corresponding detailed image 13.

[0065] If the center (physical center of the device) of the pan-tilt-zoom camera 4 (first camera) and the center of the omnidirectional camera 5 (second camera) are at the same position, the pan-tilt angle of the pan-tilt-zoom camera 4 and the coordinate position of the omnidirectional image 14 can be converted by a simple calculation. However, due to physical constraints when attaching them to the patrol inspection robot 2, the pan-tilt-zoom camera 4 and the omnidirectional camera 5 are fixed at a certain distance from each other. For this reason, in this embodiment, a process is performed to obtain in advance the relationship between the pan-tilt angle of the pan-tilt-zoom camera 4 and the coordinate position of the omnidirectional image 14.

[0066] Furthermore, based on the acquired distance information 17, the shooting position estimation unit 21 estimates a coordinate position in the omnidirectional image 14 that coincides with the shooting direction of the pan-tilt-zoom camera 4. In this way, it is possible to improve the accuracy of identifying the correspondence between the detailed image 13 and the omnidirectional image 14 from the distance from the pan-tilt-zoom camera 4 or omnidirectional camera 5 to the subject.

[0067] As shown in FIG. 7, the shooting position estimation unit 21 (FIG. 2) estimates the shooting position based on the input shooting direction information 15 of the pan-tilt-zoom camera 4. The display control unit 22 (FIG. 2) displays a mark 27 at the corresponding coordinate position in the omnidirectional image 14. When the inspection worker (user) selects the mark 27 with the mouse cursor 30, the display control unit 22 extracts the detailed image 13 corresponding to the selected coordinate position from the image database 23 and displays it on the display. In this way, it is possible to link the detailed image 13 captured by the pan-tilt-zoom camera 4 with the omnidirectional image 14 captured by the omnidirectional camera 5, making it easy to manage images that show the inspection target.

[0068] When a coordinate position other than the mark 27 is selected with the mouse cursor 30, the detailed image 13 corresponding to this coordinate position may be displayed on the display. Also, the mark 27 does not have to be displayed on the omnidirectional image 14 as long as any coordinate position can be selected. Also, the coordinate position may be selected by a method other than the mouse cursor 30. For example, if the display is a touch panel, the coordinate position may be selected when the display is touched.

[0069] In this embodiment, when an arbitrary coordinate position is selected while a still omnidirectional image 14 is being displayed on the display, a detailed image 13 of the still image is displayed. Note that this detailed image 13 may be displayed as a moving image. Also, the omnidirectional image 14 displayed on the display may be a moving image.

[0070] Additionally or alternatively, when the detailed image 13 is acquired, the shooting position estimation unit 21 compares the detailed image 13 with the distortion-corrected omnidirectional image 14. Next, the shooting position estimation unit 21 extracts a similar area in the distortion-corrected omnidirectional image 14 that is similar to the detailed image 13. Next, the shooting position estimation unit 21 identifies the extracted similar area as one that includes a coordinate position that coincides with the shooting direction of the pan-tilt-zoom camera 4. In this way, the similar area corresponding to the detailed image 13 can be extracted from the omnidirectional image 14, and the accuracy of identifying the correspondence between the detailed image 13 and the omnidirectional image 14 can be improved.

[0071] The shooting position estimation unit 21 may use an analysis technique based on learning of artificial intelligence when extracting a similar range in the omnidirectional image 14 that is similar to the detailed image 13. For example, the shooting position estimation unit 21 may be equipped with artificial intelligence (AI) that performs machine learning.

[0072] The system of this embodiment includes a control device with a highly integrated processor such as a dedicated chip, FPGA (Field Programmable Gate Array), GPU (Graphics Processing Unit), or CPU (Central Processing Unit), a storage device such as ROM (Read Only Memory) or RAM (Random Access Memory), an external storage device such as HDD (Hard Disk Drive) or SSD (Solid State Drive), a display device such as a monitor, an input device such as a mouse or keyboard, and a communication interface. This system can be realized with a hardware configuration using a normal computer.

[0073] The program executed by the system of this embodiment is provided in advance in a ROM, etc. Alternatively, the program may be provided in the form of an installable or executable file stored on a computer-readable non-transitory storage medium such as a CD-ROM, CD-R, memory card, DVD, or flexible disk (FD).

[0074] The programs executed by this system may be stored on a computer connected to a network such as the Internet and provided by downloading them via the network. This system may also be configured by combining separate modules that independently perform the functions of the components and interconnect them via a network or dedicated lines.

[0075] In this embodiment, the pan-tilt-zoom camera 4 (first camera) itself has a pan function, but other configurations are also possible. For example, the pan function of the pan-tilt-zoom camera 4 may be realized by the rotation of the patrol inspection robot 2. In other words, the functions of the patrol inspection robot 2 may be included as functions of the pan-tilt-zoom camera 4. Furthermore, the shooting direction detection unit 6 may detect the direction in which the patrol inspection robot 2 is facing (rotation direction).

[0076] In this embodiment, the pan-tilt-zoom camera 4 (first camera) and the omnidirectional camera 5 (second camera) are mounted on the patrol inspection robot 2, but other configurations are also possible. For example, an inspection worker may carry the pan-tilt-zoom camera 4 and the omnidirectional camera 5, or the pan-tilt-zoom camera 4 and the omnidirectional camera 5 may be fixedly positioned at the inspection site using a fixing device such as a tripod to take images.

[0077] According to the embodiment described above, when the shooting direction of the first camera is included in the shooting range of the second camera, the first image is recorded in association with a coordinate position in the second image that matches the shooting direction of the first camera, making it easy to manage each image when the same subject is photographed with multiple cameras with different angles of view.

[0078] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations can be made without departing from the spirit of the invention. These embodiments and modifications thereof are intended to be included within the scope and spirit of the invention, as well as within the scope of the invention described in the claims and their equivalents. [Explanation of symbols]

[0079] 1...Image management system, 2...Patrol inspection robot, 3...Management computer, 4...Pan-tilt-zoom camera, 5...Omnidirectional camera, 6...Shooting direction detection unit, 7...Distance measurement unit, 8...Position detection unit, 9...Traveling device, 10...Control computer, 11...Wheels, 12...Wide-angle camera, 13...Detailed image, 14...Omnidirectional image, 15...Shooting direction information, 16...Position information, 17...Distance information, 18...First image acquisition unit, 19...Second image acquisition unit, 20...Distortion correction unit, 21...Shooting position estimation unit, 22...Display control unit, 23...Image database, 24...Partial range, 25...Corresponding points, 26...Grid lines, 27...Mark, 28...Meter, 29...Valve, 30...Mouse cursor.

Claims

1. a patrolling robot equipped with a first camera and a second camera capable of capturing images with a wider angle of view than the first camera; a management computer that manages the images captured by the first camera and the second camera; Equipped with The relative positions of the first camera and the second camera are fixed with a certain distance between them, The management computer acquiring a first image captured multiple times at different angles by the first camera and a second image captured by the second camera; When the shooting direction of the first camera is included in the shooting range of the second camera, the first image is recorded in association with a coordinate position in the second image that coincides with the shooting direction of the first camera; When a coordinate position on which a mark is displayed is selected from among the coordinate positions in the second image, the first image recorded in association with the selected coordinate position is displayed. It is configured as follows: Image management system.

2. The second camera is capable of capturing images in all directions. The image management system according to claim 1 .

3. the management computer is configured to acquire information indicating the photographing direction of the first camera together with the acquisition of the first image; 3. The image management system according to claim 1.

4. the first camera has at least a pan-tilt function, the information indicating the photographing direction of the first camera is at least one of a control signal output from a control computer that controls the photographing direction and a detection signal output from a sensor that detects the photographing direction; The image management system according to claim 3 .

5. the management computer is configured to convert information indicating the shooting direction of the first camera into the coordinate position of the second image based on at least one of a conversion table and a conversion formula.

5. The image management system according to claim 3 or 4.

6. The management computer performing image processing to correct distortion of the second image; comparing the first image with the distortion-corrected second image when the first image is acquired; extracting a similar range in the second image that is similar to the first image; identifying the similar range as including the coordinate position that coincides with the shooting direction of the first camera; It is configured as follows: The image management system according to any one of claims 1 to 5.

7. The management computer acquiring information indicating a distance from at least one of the first camera and the second camera to a subject from a distance measurement unit that measures a distance from at least one of the first camera and the second camera to a subject; estimating the coordinate position in the second image that coincides with the shooting direction of the first camera based on a distance from at least one of the first camera and the second camera to the subject; It is configured as follows: The image management system according to any one of claims 1 to 6.

8. a patrolling robot equipped with a first camera and a second camera capable of capturing images with a wider angle of view than the first camera; a management computer that manages the images captured by the first camera and the second camera; This is a method using The relative positions of the first camera and the second camera are fixed with a certain distance between them, the management computer acquires a first image taken a plurality of times at different angles by the first camera and a second image taken by the second camera; When the shooting direction of the first camera is included in the shooting range of the second camera, the management computer records the first image in association with a coordinate position in the second image that coincides with the shooting direction of the first camera; when a coordinate position on which a mark is displayed is selected from among the coordinate positions in the second image, the management computer displays the first image recorded in association with the selected coordinate position. Image management methods.

Images