Image processing device, method, and program

Abstract

To provide an image processing device, method, and program facilitating a check of the imaging state of each camera in a case of imaging with a plurality of cameras.SOLUTION: The image processing device which processes first images captured with a plurality of cameras in time series comprises a processor. The processor acquires the first images, acquires first information relating to capturing states of the first images, arranges the first images of the latest N images, N being 2 or more, in chronological order, generates a second image including second information indicating defective capturing of the first images based on the first information, and outputs the second image to a display destination.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] The present invention relates to an image processing device, method, and program, and more particularly to an image processing device, method, and program for processing images captured by a plurality of cameras. [Background technology]

[0002] Patent Document 1 describes a technology in which, for multiple images taken continuously while automatically moving an imaging device, the shooting position of a failed image is determined, and the shooting position is automatically reproduced to take the image again.

[0003] Patent Documents 2 and 3 describe techniques for displaying images captured in time series by a plurality of cameras, arranged in chronological order (in the order of the time of capture) for each camera. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-134182 [Patent Document 2] Japanese Patent Application Laid-Open No. 2003-259342 [Patent Document 3] Japanese Patent Application Laid-Open No. 2009-187388 Summary of the Invention

[0005] One embodiment of the technique of the present disclosure provides an image processing device, method, and program that allows easy confirmation of the shooting status of each camera when shooting with multiple cameras. [Means for solving the problem]

[0006] (1) An image processing device that processes first images captured in chronological order by multiple cameras, comprising a processor, which acquires the first images, acquires first information regarding the capture conditions of the first images, arranges the most recent N first images from the multiple cameras in chronological order, where N≧2, and generates a second image based on the first information, including second information indicating poor capture of the first images, and outputs the second image to a display destination.

[0007] (2) An image processing device described in (1), in which the processor arranges first images from multiple cameras in a first direction and generates a second image in which the most recent N first images from the multiple cameras are arranged in chronological order in a second direction.

[0008] (3) An image processing device described in (2), in which the multiple cameras are arranged in a predetermined arrangement, and the processor generates a second image in which the latest first images of the multiple cameras are arranged in a first direction in an arrangement corresponding to the arrangement of the multiple cameras.

[0009] (4) The image processing device according to any one of (1) to (3), wherein the plurality of cameras are mounted on a moving body and move integrally, and capture the first image as the moving body moves.

[0010] (5) The image processing device described in (4), wherein the processor acquires third information regarding the position of the moving object and generates a second image further including the third information.

[0011] (6) The image processing device according to (5), wherein the third information is information about the distance from the starting point.

[0012] (7) The image processing device according to (5) or (6), wherein the second information includes information about the position of the moving object at the time of capturing the first image determined to be poorly captured.

[0013] (8) The image processing device according to any one of (1) to (7), wherein the processor generates a second image that further includes fourth information relating to the positional relationship of the multiple cameras.

[0014] (9) The image processing device described in (8), wherein the fourth information is information of a diagram in which first symbols or first figures representing individual cameras are arranged in accordance with the arrangement of multiple cameras.

[0015] (10) The image processing device according to (8) or (9), wherein the fourth information further includes information of a diagram showing the outline of the subject to be photographed.

[0016] (11) The image processing device according to (9), wherein the processor highlights and displays the first symbol or the first figure of the camera in which the poor image capture occurred.

[0017] (12) An image processing device described in any one of (1) to (11), in which the processor arranges a second symbol or a second figure representing each camera, associates the second symbol or the second figure with the corresponding camera, and generates a second image in which the most recent N first images of the corresponding camera are arranged in chronological order.

[0018] (13) The image processing device according to (12), wherein the processor highlights and displays the second symbol or second figure of the camera in which the poor image capture occurred.

[0019] (14) An image processing device described in any one of (1) to (13), wherein the processor acquires fifth information regarding the status of the multiple cameras and generates a second image further including the fifth information of the multiple cameras.

[0020] (15) The image processing device according to (14), wherein the fifth information includes information on the remaining battery charge.

[0021] (16) An image processing device described in (14) or (15), in which the processor generates a second image in which the most recent N first images from the corresponding cameras are arranged in chronological order, associated with the fifth information of the multiple cameras.

[0022] (17) An image processing device described in any one of (1) to (16), in which the processor generates a second image in which the most recent N first images from multiple cameras are arranged in chronological order with at least one of the size and color tone changed.

[0023] (18) The image processing device according to any one of (1) to (17), wherein the processor analyzes the first image to obtain the first information.

[0024] (19) An image processing device described in any one of (1) to (18), wherein the processor estimates the cause of the poor imaging based on the first information and generates a second image further including the estimated result of the cause of the poor imaging.

[0025] (20) An image processing method for processing first images captured in chronological order by multiple cameras, the image processing method comprising: acquiring the first images; acquiring first information relating to the capture conditions of the first images; arranging the most recent N first images from the multiple cameras in chronological order, where N≧2; generating a second image based on the first information, the second image including second information indicating poor capture of the first images; and outputting the second image to a display destination.

[0026] (21) An image processing program that processes first images captured in chronological order by multiple cameras, the image processing program causing a computer to perform the following functions: acquiring the first images; acquiring first information regarding the capture status of the first images; arranging the most recent N first images from the multiple cameras in chronological order, where N≧2, and generating a second image that includes second information indicating poor capture of the first images based on the first information; and outputting the second images to a display destination. [Brief explanation of the drawings]

[0027] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a photography system. [Figure 2] FIG. 1 is a perspective view showing the configuration of a multi-eye photographing device; [Figure 3] A front view showing the configuration of a multi-eye photographing device [Figure 4] A side view showing the configuration of a multi-eye photographing device. [Figure 5] A front view showing the camera and lighting device attached to the front panel. [Figure 6] Rear view showing the camera and lighting device attached to the front panel [Figure 7] A front view showing the camera and lighting device attached to the rear panel. [Figure 8] Rear view showing the camera and lighting device attached to the rear panel [Figure 9] Block diagram showing the system configuration of the imaging system [Figure 10] FIG. 1 is a diagram illustrating an example of a hardware configuration of a control device. [Figure 11] Block diagram of main functions related to shooting control [Figure 12] Block diagram of the main functions related to the presentation of shooting conditions [Figure 13] Block diagram of the main functions of the image judgment unit [Figure 14] FIG. 10 is a diagram showing an example of an output image. [Figure 15] FIG. 10 is a diagram showing an example of an output image when a malfunction occurs in the camera. [Figure 16] Enlarged image display area [Figure 17] FIG. 10 is a diagram showing an example of a display in the image display area when an imaging error occurs. [Figure 18] FIG. 10 is a diagram showing how the captured image display area changes as the image capture progresses. [Figure 19] FIG. 10 is a diagram showing how the captured image display area changes as the image capture progresses. [Figure 20] A diagram showing an example of an enlarged display [Figure 21] A flowchart showing an example of a processing procedure of a control device when automatically stopping photography. [Figure 22] FIG. 10 is a diagram showing another example of displaying a captured image in the captured image display area; [Figure 23] FIG. 10 is a diagram showing another example of displaying a captured image in the captured image display area; [Figure 24] FIG. 10 is a diagram showing another example of displaying a captured image in the captured image display area; [Figure 25] Figure showing an example of a screen display showing the cause of an error DETAILED DESCRIPTION OF THE INVENTION

[0028] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] Here, an example will be described in which the present invention is applied to an imaging system for photographing the inner wall surface of a tunnel structure for the purpose of inspection.

[0030] Tunnel structures (hereinafter referred to as "tunnels") such as water conduits for hydroelectric power plants and subway tunnels are inspected periodically to ensure their safety. Visual inspection was the norm in the past, but in recent years this has been increasingly replaced by inspection using images taken with cameras. Image inspection involves taking photographs of the tunnel's inner wall surface with a camera and then visually inspecting or processing the images to detect damage such as cracks.

[0031] A well-known imaging device suitable for photographing the inner walls of tunnels is a device (multi-lens imaging device) that mounts multiple cameras on a moving object and photographs the tunnel while moving inside. The multiple cameras are installed so that the imaging ranges of adjacent cameras partially overlap so that the images can be synthesized into a panorama after shooting.

[0032] When inspecting using images, it is necessary to take a high-quality photograph of the object (for example, a quality that allows for the detection of cracks 0.2 mm wide) without missing anything, and if a photograph fails, it must be taken again. However, tunnels are dark, and when retaking photographs, it is often difficult to immediately determine which point to return to and take the photograph again.

[0033] In this embodiment, when taking pictures with multiple cameras while moving inside a tunnel, a photography system is provided that allows the photography status of each camera to be easily confirmed and allows easy retaking of a photograph even if photography fails.

[0034] [Shooting system configuration] FIG. 1 is a diagram showing a schematic configuration of an imaging system.

[0035] As described above, the imaging system 1 of this embodiment is configured as a system that images the inner wall surface of the tunnel 3 while moving inside the tunnel 3. Fig. 1 shows an example in which the tunnel 3 to be imaged is a so-called horseshoe-shaped tunnel (a tunnel with a horseshoe-shaped cross section).

[0036] As shown in FIG. 1, the photography system 1 includes a multi-eye photography device 10 equipped with multiple cameras and lighting devices, and a control device 100 that controls the multi-eye photography device 10 and processes images captured by the multi-eye photography device 10.

[0037] The multi-eye camera 10 is mounted on, for example, a hand-pushed cart 2 and moves through the tunnel 3. This allows the multiple cameras to move together. FIG. 1 shows an example in which rails 4 are laid inside the tunnel 3. In this case, for example, the cart 2 travels on the rails 4 and moves through the tunnel 3.

[0038] The bogie 2 is equipped with a distance meter 5. The distance meter 5 measures the distance from the starting point to the current position (distance along the route). When the bogie 2 travels on the rails 4, the distance is measured along the rails 4. The distance meter 5 measures the distance, for example, by utilizing the rotation of the wheels of the bogie 2.

[0039] Information about the distance measured by the rangefinder 5 is transmitted to the control device 100 via the relay device 20. Therefore, the rangefinder 5 has a communication function.

[0040] [Multi-eye camera] FIG. 2 is a perspective view showing the configuration of the multi-eye photography device. FIG. 3 is a front view showing the configuration of the multi-eye photography device. FIG. 4 is a side view showing the configuration of the multi-eye photography device. In FIGS. 2 to 4, x, y, and z are three mutually perpendicular axes. The plane including the x-axis and y-axis is the horizontal plane, and the direction of the z-axis is the vertical direction. The direction of the x-axis is the traveling direction of the dolly 2, and the + direction of the x-axis (rightward in FIG. 4) is the traveling direction during photography. Therefore, the + direction of the x-axis (leftward in FIG. 4) is the forward direction (forward direction) of the dolly 2 and the multi-eye photography device 10, and the - direction (leftward in FIG. 4) is the backward direction (rearward direction) of the dolly 2 and the multi-eye photography device 10.

[0041] Multi-eye photographing device 10 is configured by combining multiple cameras and multiple lighting devices. The number of cameras and lighting devices can be increased or decreased as needed depending on the subject being photographed. Here, we will explain an example in which multi-eye photographing device 10 is configured using nine cameras C1 to C9 and nine lighting devices L1 to L9.

[0042] The multi-eye photographing device 10 has a frame 11 for mounting a plurality of cameras C1 to C9 and lighting devices L1 to L9. The frame 11 is made up of a base 12, a front column 13F, a rear column 13R, a front panel 14F, and a rear panel 14R.

[0043] The base 12 has a rectangular, flat plate shape. A front column 13F and a rear column 13R are installed on the base 12.

[0044] The front column 13F and the rear column 13R have a rectangular column shape. The front column 13F and the rear column 13R are arranged with a predetermined distance between them in the front-to-rear direction (the direction of the x-axis) relative to the base 12. The front column 13F and the rear column 13R are also installed perpendicular to the base 12. A front panel 14F is attached to the front column 13F, and a rear panel 14R is attached to the rear column 13R.

[0045] The front panel 14F and the rear panel 14R are disk-shaped and arranged orthogonal to the front-rear direction (the direction of the x-axis) of the base 12 and are arranged coaxially. An axis passing through the centers of the front panel 14F and the rear panel 14R and parallel to the x-axis is defined as the axis Ax of the camera array 10.

[0046] Cameras C1 to C9 and lighting devices L1 to L9 are attached to front panel 14F or rear panel 14R via brackets B1 to B9. Hereinafter, cameras C1 to C9 will be distinguished from one another as necessary by referring to camera C1 as the "first camera C1," camera C2 as the "second camera C2," camera C3 as the "third camera C3," camera C4 as the "fourth camera C4," camera C5 as the "fifth camera C5," camera C6 as the "sixth camera C6," camera C7 as the "seventh camera C7," camera C8 as the "eighth camera C8," and camera C9 as the "ninth camera C9." Furthermore, the lighting devices L1 to L9 are distinguished from one another by referring to lighting device L1 as the "first lighting device L1," lighting device L2 as the "second lighting device L2," lighting device L3 as the "third lighting device L3," lighting device L4 as the "fourth lighting device L4," lighting device L5 as the "fifth lighting device L5," lighting device L6 as the "sixth lighting device L6," lighting device L7 as the "seventh lighting device L7," lighting device L8 as the "eighth lighting device L8," and lighting device L9 as the "ninth lighting device L9." Additionally, bracket B1 will be referred to as the "first bracket B1," bracket B2 as the "second bracket B2," bracket B3 as the "third bracket B3," bracket B4 as the "fourth bracket B4," bracket B5 as the "fifth bracket B5," bracket B6 as the "sixth bracket B6," bracket B7 as the "seventh bracket B7," bracket B8 as the "eighth bracket B8," and bracket B9 as the "ninth bracket B9" to distinguish between brackets B1 to B9.

[0047] The first camera C1 and the first lighting device L1 are attached to the front panel 14F via a first bracket B1. The second camera C2 and the second lighting device L2 are attached to the rear panel 14R via a second bracket B2. The third camera C3 and the third lighting device L3 are attached to the front panel 14F via a third bracket B3. The fourth camera C4 and the fourth lighting device L4 are attached to the rear panel 14R via a fourth bracket B4. The fifth camera C5 and the fifth lighting device L5 are attached to the front panel 14F via a fifth bracket B5. The sixth camera C6 and the sixth lighting device L6 are attached to the rear panel 14R via a sixth bracket B6. The seventh camera C7 and the seventh lighting device L7 are attached to the front panel 14F via a seventh bracket B7. The eighth camera C8 and the eighth lighting device L8 are attached to the rear panel 14R via an eighth bracket B8. The ninth camera C9 and the ninth lighting device L9 are attached to the front panel 14F via a ninth bracket B9. That is, the odd-numbered cameras C1, C3, C5, C7, and C9 and the lighting devices L1, L3, L5, L7, and L9 are attached to the front panel 14F, and the even-numbered cameras C2, C4, C6, and C8 and the lighting devices L2, L4, L6, and L8 are attached to the rear panel 14R.

[0048] The sets of cameras C1 to C9 and lighting devices L1 to L9 attached to the brackets B1 to B9 respectively constitute photographing units U1 to U9. Hereinafter, as necessary, the set of the first camera C1 and the first lighting device L1 will be referred to as the "first photographing unit U1," the set of the second camera C2 and the second lighting device L2 will be referred to as the "second photographing unit U2," the set of the third camera C3 and the third lighting device L3 will be referred to as the "third photographing unit U3," the set of the fourth camera C4 and the fourth lighting device L4 will be referred to as the "fourth photographing unit U4," the set of the fifth camera C5 and the fifth lighting device L5 will be referred to as the "fifth photographing unit U5," the set of the sixth camera C6 and the sixth lighting device L6 will be referred to as the "sixth photographing unit U6," the set of the seventh camera C7 and the seventh lighting device L7 will be referred to as the "seventh photographing unit U7," the set of the eighth camera C8 and the eighth lighting device L8 will be referred to as the "eighth photographing unit U8," and the set of the ninth camera C9 and the ninth lighting device L9 will be referred to as the "ninth photographing unit U9," in order to distinguish between the photographing units U1 to U9.

[0049] Fig. 5 is a front view showing how the camera and lighting device are attached to the front panel, and Fig. 6 is a rear view showing how the camera and lighting device are attached to the front panel.

[0050] Brackets B1, B3, B5, B7, and B9 are arranged on the same circumference with respect to the front panel 14F. Each bracket B1, B3, B5, B7, and B9 is attached to the front panel 14F so that it can move circumferentially within a predetermined angular range (e.g., 30°). Each bracket B1, B3, B5, B7, and B9 is fixed to the front panel 14F with a clamp (e.g., a toggle clamp) CL. Therefore, the position can be easily adjusted by loosening the clamp CL.

[0051] The cameras C1, C3, C5, C7, and C9 are attached to the camera mounting portions provided on the brackets B1, B3, B5, B7, and B9. The lighting devices L1, L3, L5, L7, and L9 are attached to the lighting mounting portions provided on the brackets B1, B3, B5, B7, and B9. The cameras C1, C3, C5, C7, and C9 are attached to the camera mounting portions using, for example, tripod screw holes. The lighting devices L1, L3, L5, L7, and L9 are attached to the lighting mounting portions by fixing the arm portions with bolts.

[0052] The cameras C1, C3, C5, C7, and C9 and the lighting devices L1, L3, L5, L7, and L9 are attached to the front panel 14F via brackets B1, B3, B5, B7, and B9 and are disposed on the frame 11 in a predetermined orientation. Specifically, the cameras are disposed within a plane (zy plane) perpendicular to the axis Ax of the camera array 10, facing outward in a radial direction (normal direction) centered on the axis Ax of the camera array 10. More specifically, the cameras C1, C3, C5, C7, and C9 are disposed with their imaging optical axes facing outward in a radial direction (normal direction) centered on the axis Ax of the camera array 10. The cameras C1, C3, C5, C7, and C9 are attached such that the bottom surfaces of their camera bodies are parallel to the front panel 14F (parallel to the zy plane) (the bottom sides of the image sensors are attached parallel to the zy plane). As a result, the cameras C1, C3, C5, C7, and C9 are arranged at predetermined intervals in the circumferential direction within the zy plane, centered on the axis Ax of the multiple-eye imaging device 10. The illumination devices L1, L3, L5, L7, and L9 are arranged with their irradiation directions directed radially outward (normally) from the axis Ax of the multiple-eye imaging device 10. As a result, the cameras C1, C3, C5, C7, and C9 and the illumination devices L1, L3, L5, L7, and L9 are arranged radially within the zy plane, centered on the axis Ax of the multiple-eye imaging device 10.

[0053] As described above, the brackets B1, B3, B5, B7, and B9 are attached to the front panel 14F so as to be movable in the circumferential direction within a predetermined angular range. FIGS. 5 and 6 show the brackets B1, B3, B5, B7, and B9 fixed at their reference positions. By fixing the brackets B1, B3, B5, B7, and B9 at their reference positions, the first camera C1 and the first lighting device L1 are positioned at 330° (-30°) in a front view (FIG. 5). The third camera C3 and the third lighting device L3 are positioned at 30°. The fifth camera C5 and the fifth lighting device L5 are positioned at 90°. The seventh camera C7 and the seventh lighting device L7 are positioned at 150°. The ninth camera C9 and the ninth lighting device L9 are positioned at 210°.

[0054] Each bracket B1, B3, B5, B7, and B9 is attached so as to be movable in the circumferential direction within a range of ±15° from a reference position. Therefore, the positions of each camera C1, C3, C5, C7, and C9 and lighting device L1, L3, L5, L7, and L9 can be adjusted in the circumferential direction within a range of ±15° from the reference position.

[0055] Fig. 7 is a front view showing how the camera and lighting device are attached to the rear panel, and Fig. 8 is a rear view showing how the camera and lighting device are attached to the rear panel.

[0056] Brackets B2, B4, B6, and B8 are arranged on the same circumference with respect to the rear panel 14R. Each bracket B2, B4, B6, and B8 is attached to the rear panel 14R so that it can move circumferentially within a predetermined angular range (for example, 30°). Each bracket B2, B4, B6, and B8 is fixed to the rear panel 14R with a clamp CL. Therefore, the position can be easily adjusted by loosening the clamp CL.

[0057] Cameras C2, C4, C6, and C8 are attached to camera mounting portions provided on brackets B2, B4, B6, and B8. Lighting devices L2, L4, L6, and L8 are attached to lighting mounting portions provided on brackets B2, B4, B6, and B8. Cameras C2, C4, C6, and C8 are attached to the camera mounting portions using, for example, tripod screw holes. Lighting devices L2, L4, L6, and L8 are attached to the lighting mounting portions by fixing their arm portions with bolts.

[0058] The cameras C2, C4, C6, and C8 and the lighting devices L2, L4, L6, and L8 are attached to the rear panel 14R via brackets B2, B4, B6, and B8 and are disposed on the frame 11 in a predetermined orientation. Specifically, the cameras are disposed within a plane (Zy plane) perpendicular to the axis Ax of the multiple-eye imaging device 10, facing outward in a radial direction (normal direction) centered on the axis Ax of the multiple-eye imaging device 10. More specifically, the cameras C2, C4, C6, and C8 are disposed with their imaging optical axes facing outward in a radial direction (normal direction) centered on the axis Ax of the multiple-eye imaging device 10. The cameras C2, C4, C6, and C8 are attached with the bottom surfaces of their camera bodies parallel to the rear panel 14R (parallel to the zy plane) (the bottom sides of the image sensors are attached parallel to the zy plane). As a result, the cameras C2, C4, C6, and C8 are arranged at predetermined intervals in the circumferential direction within the zy plane, centered on the axis Ax of the multiple-eye imaging device 10. The illumination devices L2, L4, L6, and L8 are arranged with their irradiation directions facing outward in the radial direction (normal direction) centered on the axis Ax of the multiple-eye imaging device 10. As a result, the cameras C2, C4, C6, and C8 and the illumination devices L2, L4, L6, and L8 are arranged radially within the zy plane, centered on the axis Ax of the multiple-eye imaging device 10.

[0059] As described above, the brackets B2, B4, B6, and B8 are attached to the rear panel 14R so as to be movable in the circumferential direction within a predetermined angular range. FIGS. 7 and 8 show the brackets B2, B4, B6, and B8 fixed at their reference positions. By fixing the brackets B2, B4, B6, and B8 at their reference positions, the second camera C2 and the second lighting device L2 are positioned at a 0° angle in a front view (FIG. 7). The fourth camera C4 and the fourth lighting device L4 are positioned at a 60° angle. The sixth camera C6 and the sixth lighting device L6 are positioned at a 120° angle. The eighth camera C8 and the eighth lighting device L8 are positioned at a 180° angle. Therefore, the second camera C2 is positioned between the first camera C1 and the third camera C3 in the circumferential direction. The fourth camera C4 is positioned between the third camera C3 and the fifth camera C5 in the circumferential direction. The sixth camera C6 is disposed between the fifth camera C5 and the seventh camera C7 in the circumferential direction. The eighth camera C8 is disposed between the seventh camera C7 and the ninth camera C9 in the circumferential direction. Similarly, the second lighting device L2 is disposed between the first lighting device L1 and the third lighting device L3 in the circumferential direction. The fourth lighting device L4 is disposed between the third lighting device L3 and the fifth lighting device L5 in the circumferential direction. The sixth lighting device L6 is disposed between the fifth lighting device L5 and the seventh lighting device L7 in the circumferential direction. The eighth lighting device L8 is disposed between the seventh lighting device L7 and the ninth lighting device L9 in the circumferential direction.

[0060] Each bracket B2, B4, B6, and B8 is attached so as to be movable within a range of ±15° in the circumferential direction from a reference position. Therefore, the positions of each camera C2, C4, C6, and C8 and lighting device L2, L4, L6, and L8 can be adjusted within a range of ±15° in the circumferential direction from the reference position.

[0061] In the multi-eye photographing device 10 configured as described above, nine cameras C1-C9 and lighting devices L1-L9 are arranged at predetermined intervals on a circle centered on the axis Ax of the device. When the brackets B1-B9 are fixed at reference positions, the cameras C1-C9 and lighting devices L1-L are arranged at 30° intervals. Furthermore, the cameras C1-C9 and lighting devices L1-L are attached so that their positions can be adjusted within a range of ±15° in the circumferential direction. The positions of the cameras C1-C9 are adjusted so that the shooting ranges of adjacent cameras overlap. More specifically, the positions are adjusted so that the shooting ranges overlap at a predetermined overlap rate (also called side overlap rate). As an example, the positions are adjusted so that the shooting ranges overlap at an overlap rate of 20% or more.

[0062] The cameras C1 to C9 used are digital cameras. There are no particular limitations on the type of digital camera, as long as it has the function of electrically recording images (still images and / or moving images). As an example, a digital camera with interchangeable lenses is used. In this embodiment, the cameras C1 to C9 each have storage (storage media) and store captured images in the storage. The storage may be a built-in memory or an exchangeable memory (so-called memory card).

[0063] The lighting devices L1 to L9 used are not particularly limited. As an example, a halogen lamp is used. Other than this, for example, an LED (light emitting diode) lamp, a xenon lamp, etc. can also be used. In this embodiment, lighting devices having a function for adjusting the irradiation angle (irradiation direction) are used. Each of the lighting devices L1 to L9 rotates (swings back and forth) around an axis perpendicular to the optical axis of the cameras C1 to C9 to adjust the irradiation angle. The lighting devices L1 to L9 have an irradiation range that can cover the shooting range of the cameras C1 to C9.

[0064] [Repeater] FIG. 9 is a block diagram showing the system configuration of the imaging system.

[0065] 9, the multi-eye photographing device 10 and the rangefinder 5 are communicably connected to the control device 100 via a relay device 20. The relay device 20 is, for example, configured as a computer with a communication function. The relay device 20 is mounted on a dolly 2 together with the multi-eye photographing device 10.

[0066] The cameras C1 to C9 and lighting devices L1 to L9 included in the multi-eye imaging device 10 are connected to a relay device 20. The connection between each of the cameras C1 to C9 and the relay device 20, and the connection between each of the lighting devices L1 to L9 and the relay device 20 are not particularly limited. They may be connected by wire or wirelessly. As an example, in this embodiment, each of the cameras C1 to C9 and each of the lighting devices L1 to L9 are connected to the relay device 20 by wire.

[0067] The connection between the rangefinder 5 and the relay device 20 is not particularly limited. It may be a wired connection or a wireless connection. As an example, in this embodiment, the rangefinder 5 is connected to the relay device 20 wirelessly. For example, the connection may be via short-range wireless communication such as Bluetooth (registered trademark).

[0068] The connection between the control device 100 and the relay device 20 is not particularly limited. The connection may be wired or wireless. As an example, in this embodiment, the control device 100 and the relay device 20 are connected wirelessly. For example, they are connected via a wireless LAN (Local Area Network) such as Wi-Fi (registered trademark).

[0069] [Control device] The control device 100 is configured by a computer equipped with a communication function. As an example, in this embodiment, the control device 100 is configured by a tablet computer.

[0070] FIG. 10 is a diagram illustrating an example of a hardware configuration of the control device.

[0071] The control device 100 includes a processor 111, a main memory unit 112, an auxiliary memory unit 113, a display unit 114, an operation unit 115, a communication unit 116, and the like.

[0072] The processor 111 executes programs and functions as various processing units. As an example, in this embodiment, the processor 111 is configured as a CPU (Central Processing Unit). The programs (image processing programs, etc.) executed by the processor 111, as well as various data necessary for control, calculation, etc., are stored in the main memory unit 112 and / or the auxiliary memory unit 113.

[0073] The main storage unit 112 includes a RAM (Random Access Memory) and a ROM (Read Only Memory). The RAM is used as a work area for the processor 111. The ROM stores a basic input / output program and the like.

[0074] The auxiliary storage unit 113 is configured, for example, with an EEPROM (Electrically Erasable and Programmable ROM), an SSD (Solid State Drive), or the like.

[0075] The display unit 114 is configured by, for example, a liquid crystal display, an organic electroluminescence diode display, or the like.

[0076] The operation unit 115 includes a touch panel and various operation buttons, etc. The touch panel detects a touch operation on the screen of the display unit 114.

[0077] The communication unit 116 connects the control device 100 and an external device (such as the relay device 20) so that they can communicate with each other. As described above, in this embodiment, the control device 100 and the relay device 20 are connected via a wireless LAN (for example, Wi-Fi). Therefore, the communication unit 116 has at least a communication function via the wireless LAN.

[0078] [Controller function] The control device 100 has a function for controlling the shooting by the camera array imaging device 10 (shooting control function), a function for presenting the shooting conditions of each of the cameras C1 to C9 to the user (monitoring function), and the like.

[0079] [Shooting control function] FIG. 11 is a block diagram of the main functions related to shooting control.

[0080] 11, the control device 100 has functions related to photography control, such as a camera control unit 111A, an illumination control unit 111B, and a rangefinder control unit 111C. The functions of each control unit are realized by the processor 111 executing a predetermined program.

[0081] The camera control unit 111A controls the operation of each of the cameras C1 to C9 mounted on the multi-eye photographing device 10. Based on operation input from the operation unit 115, the camera control unit 111A controls each of the cameras C1 to C9 to perform predetermined photographing. In other words, it causes each of the cameras C1 to C9 to photograph still images or moving images. The photographing of still images includes so-called interval photographing. Interval photographing is a function that repeatedly photographs still images at regular time intervals.

[0082] When a still image is to be captured, camera control unit 111A, in response to a capture instruction from operation unit 115, causes each of cameras C1 to C9 to capture a still image.

[0083] When interval shooting is performed, camera control unit 111A causes each of cameras C1 to C9 to start interval shooting in response to a shooting start instruction from operation unit 115. That is, camera control unit 111A causes each of cameras C1 to C9 to shoot still images at regular time intervals. Furthermore, camera control unit 111A causes each of cameras C1 to C9 to end shooting in response to a shooting end instruction from operation unit 115.

[0084] When capturing a moving image, camera control unit 111A causes each of cameras C1 to C9 to start capturing a moving image in response to a capture start instruction from operation unit 115. Furthermore, camera control unit 111A causes each of cameras C1 to C9 to stop capturing a moving image in response to a capture end instruction from operation unit 115.

[0085] The illumination control unit 111B controls the operation of each of the illumination devices L1 to L9 installed in the multi-eye imaging device 10. The illumination control unit 111B controls each of the illumination devices L1 to L9 and controls the emission of illumination light based on operation input from the operation unit 115. The control of emission of illumination light includes control of brightness as well as control of on / off of the illumination light.

[0086] The rangefinder control unit 111C controls the operation of the rangefinder 5 mounted on the carriage 2. The rangefinder control unit 111C controls the rangefinder 5 based on an operation input from the operation unit 115. For example, it performs processing such as resetting the measurement.

[0087] [Monitor function] FIG. 12 is a block diagram of the main functions related to the presentation of the shooting situation.

[0088] 12, the control device 100 has functions related to the presentation of the shooting situation, such as a data acquisition unit 111E, a distance information acquisition unit 111F, an image determination unit 111G, a status determination unit 111H, an output image generation unit 111J, and an output control unit 111K. The functions of each unit are realized by the processor 111 executing a predetermined program. In this embodiment, the control device 100 that provides the monitor function is an example of an image processing device. Furthermore, the program that causes the processor 111 to realize the monitor function is an example of an image processing program.

[0089] (1) Data acquisition section The data acquisition section 111E acquires captured images, status information, and the like from the cameras C1 to C9 that make up the camera array photographing device 10. The data acquisition section 111E has the functions of an image acquisition section 111E1 and a status information acquisition section 111E2, and the like.

[0090] The image acquisition unit 111E1 acquires captured images from each of the cameras C1 to C9. When each of the cameras C1 to C9 performs capturing, it outputs the image obtained by capturing (captured image) to the control device 100. The image acquisition unit 111E1 acquires the captured images output from each of the cameras C1 to C9. The captured images are output from the cameras C1 to C9 in chronological order of capturing (capture order). Therefore, the captured images are acquired in chronological order. In the case of a moving image, the images constituting each frame are output in chronological order. Therefore, in the case of a moving image, the captured images (images constituting a frame) are acquired in chronological order. The captured images acquired by the image acquisition unit 111E1 are added to the image determination unit 111G and the output image generation unit 111J. In this embodiment, the captured images output from each of the cameras C1 to C9 are an example of a first image.

[0091] The status information acquisition unit 111E2 acquires information (status information) relating to the status of each camera C1 to C9 from each of the cameras C1 to C9. The status information includes information such as remaining battery capacity information, remaining memory capacity information, and communication status information. The communication status information is information relating to the communication status (connection status) between each of the cameras C1 to C9 and the control device 100. As an example, in this embodiment, the status information acquired by the status information acquisition unit 111E2 is added to the status determination unit 111H and the output image generation unit 111J. In this embodiment, the status information is an example of fifth information relating to the status of the camera.

[0092] (2) Distance information acquisition unit The distance information acquisition unit 111F acquires distance information from the rangefinder 5. As described above, the distance is the distance from the starting point to the current position. The rangefinder 5 measures the distance to the current position of the carriage 2, setting the starting point as 0, and outputs the distance to the control device 100. The distance information acquired by the distance information acquisition unit 111F is added to the output image generation unit 111J.

[0093] (3) Image Judgment Unit The image determination unit 111G analyzes the images captured by the cameras C1 to C9 and determines whether the images are suitable for inspection (OK or NG). In this embodiment, the image determination unit 111G determines whether the captured images are suitable for inspection from the viewpoints of image quality and overlap rate.

[0094] FIG. 13 is a block diagram of the main functions of the image determination unit.

[0095] As shown in FIG. 13, the image determination unit 111G has the functions of an image quality determination unit 111G1, an overlap rate determination unit 111G2, and a comprehensive determination unit 111G3.

[0096] (3-1) Image quality assessment section The image quality determination unit 111G1 determines whether a captured image is suitable for use in inspection from the perspective of image quality. The image quality determination unit 111G1 analyzes the captured image and determines whether it meets the required image quality. That is, it determines whether the image quality is sufficient for inspection purposes (for example, image quality sufficient to detect a crack with a width of 0.2 mm). The image quality determination unit 111G1 checks the image quality from the perspective of, for example, brightness, blur, image blur, cloudiness, etc., and determines whether the image quality is sufficient for inspection purposes. Well-known techniques can be used to check the image quality. For example, a trained model that has been machine-learned to check images for inspection purposes can be used to check the image quality.

[0097] If the image quality requirements are met, image quality judgment unit 111G1 outputs the result as an OK image to overall judgment unit 111G3. If the image quality requirements are not met, image quality judgment unit 111G1 outputs the result as an NG image (poorly captured image) to overall judgment unit 111G3.

[0098] (3-2) Overlap rate determination section The overlap rate determination unit 111G2 determines whether the captured images are suitable for use as inspection images from the perspective of overlap rate. The captured images are synthesized into a panorama for later use. If the overlap rate of the captured images between adjacent cameras is insufficient, problems will occur in the synthesis. For this reason, the acceptability of the captured images is determined from the perspective of overlap rate.

[0099] As shown in FIG. 13, the overlapping rate determining unit 111G2 has functions of an overlapping range detecting unit 111G2a, an overlapping rate calculating unit 111G2b, an overlapping rate pass / fail determining unit 111G2c, and the like.

[0100] The overlap area detection unit 111G2a processes the images acquired from each of the cameras C1 to C9 and detects the overlap area of ​​the captured images between adjacent cameras. Specifically, it detects the overlap area of ​​the captured images between the first camera C1 and the second camera C2, between the second camera C2 and the third camera C3, between the third camera C3 and the fourth camera C4, between the fourth camera C4 and the fifth camera C5, between the fifth camera C5 and the sixth camera C6, between the sixth camera C6 and the seventh camera C7, between the seventh camera C7 and the eighth camera C8, and between the eighth camera C8 and the ninth camera C9.

[0101] A known method is used to detect the overlapping area through image processing. As an example, the overlapping area detection unit 111G2a detects feature points of objects in each of the two images and detects the overlapping area between the two images based on the detected feature points. The detection result is output to the overlapping rate calculation unit 111G2b.

[0102] The overlap rate calculation unit 111G2b calculates the overlap rate (also called the side overlap rate) of images captured by adjacent cameras. The overlap rate is calculated as the proportion of overlapping images from adjacent cameras relative to the entire image. Therefore, for example, if the area of ​​the entire image is Sa and the area of ​​the region of the entire image that overlaps with the image of an adjacent camera is Sb, the overlap rate R is calculated as R=Sb / Sa.

[0103] The overlap rate of images is calculated by calculating the overlap rate between the image of the first camera C1 and the image of the second camera C2. Also, the overlap rate between the image of the second camera C2 and the image of the third camera C3 is calculated. That is, the overlap rate between the image of the nth camera and the image of the n+1th camera is calculated (n=1, 2, ..., 8).

[0104] The overlap rate calculation unit 111G2b calculates the overlap rate between each image based on the detection result of the overlap range detection unit 111G2a, and outputs the calculation result to the overlap rate pass / fail determination unit 111G2c.

[0105] The overlap rate pass / fail determination unit 111G2c determines whether the images captured by each of the cameras C1 to C9 pass or fail (OK or NG) based on the overlap rate calculated by the overlap rate calculation unit 111G2b. As described above, the images captured by each of the cameras C1 to C9 are panoramic-combined for later use. To ensure that the images captured by each of the cameras C1 to C9 are panoramic-combined, a certain overlap rate or higher is required between adjacent images. Furthermore, even if panoramic composition is not performed, it is necessary to capture the entire circumference without omissions. The overlap rate pass / fail determination unit 111G2c acquires the overlap rate calculated by the overlap rate calculation unit 111G2b, compares it with a threshold value (e.g., 20%), and determines whether the images captured by each of the cameras C1 to C9 pass or fail. That is, if the overlap rate is equal to or greater than the threshold value, it is determined that panoramic composition is possible or that all images have been captured without omissions, and the result is OK. On the other hand, if the overlap rate is less than the threshold, it is determined that panoramic composition is difficult or that some images have been missed, and the result is NG (poor photography). For example, if the overlap rate between the image of the nth camera and the image of the n+1th camera is less than a threshold, the images captured by the nth camera and the n+1th camera are determined to be NG.

[0106] (3-3) General Judging Section The overall determination unit 111G3 comprehensively determines whether the captured images of the cameras C1 to C9 are acceptable or unacceptable (OK or NG) based on the determination results of the image quality determination unit 111G1 and the overlap rate determination unit 111G2. That is, it comprehensively determines whether the captured images are appropriate for inspection from the viewpoints of image quality and overlap rate.

[0107] When the judgment results of both the image quality judgment section 111G1 and the overlap rate judgment section 111G2 are "OK", the overall judgment section 111G3 judges it as "OK". Therefore, when the judgment result of either one is "NG", it is judged as "NG (poor shooting)".

[0108] The determination result of comprehensive determination section 111G3 is added to output image generation section 111J as the determination result of image determination section 111G. In the present embodiment, information on the determination result of image determination section 111G is an example of first information related to the shooting state of the captured image, which is the first image.

[0109] (4) Status determination section The status determination unit 111H determines the state (status) of each of the cameras C1 to C9 based on the status information of each of the cameras C1 to C9 acquired by the status information acquisition unit 111E2. As described above, in this embodiment, the status information acquired includes information on the remaining battery capacity and information on the communication status from each of the cameras C1 to C9. The status determination unit 111H determines the remaining battery capacity and communication status of each of the cameras C1 to C9. With regard to the remaining battery capacity, for example, it determines whether the remaining battery capacity is equal to or greater than a threshold. With regard to the communication status, it determines whether communication is possible. The determination result of the status determination unit 111H (status determination result) is sent to the output image generation unit 111J.

[0110] (5) Output image generation unit Output image generation unit 111J generates an image (output image) to be output to display unit 114. Output image generation unit 111J generates the output image based on the captured images of each of cameras C1 to C9 acquired by image acquisition unit 111E1, the determination results of each captured image by image determination unit 111G, status information of each of cameras C1 to C9 acquired by status information acquisition unit 111E2, information on the status determination results by status determination unit 111H, and distance information acquired by distance information acquisition unit 111F. In the present embodiment, the output image is an example of a second image.

[0111] Fig. 14 is a diagram showing an example of an output image when the display screen is portrait-oriented.

[0112] 14, output image 200, when output to a display destination, is configured to include an operating status display area 210 in which the operating status of multi-eye photography device 10 is displayed, and a captured image display area 220 in which images captured by multi-eye photography device 10 (captured images) are displayed. As shown in FIG. 14, when the display destination screen is vertically oriented, operating status display area 210 and captured image display area 220 are arranged one above the other.

[0113] (5-1) Operation status display area The operating status of the multiple-eye photographing device 10 is displayed in the operating status display area 210. More specifically, the operating status of each of the cameras C1 to C9 of the multiple-eye photographing device 10 is displayed.

[0114] As shown in Figure 14, in this embodiment, icons representing each camera C1 to C9 (hereinafter referred to as "camera icons") IC1 to IC9 are arranged in a predetermined layout, and the operating status of each camera C1 to C9 is displayed using the color of the camera icons IC1 to IC9.

[0115] The camera icons IC1 to IC9 are made up of a combination of a camera-like shape and a number. The numbers displayed inside the camera-like shape correspond to the numbers of each camera C1 to C9. For example, the camera icon IC1 with the number 1 corresponds to the first camera C1, and the camera icon IC2 with the number 2 corresponds to the second camera C2.

[0116] The camera icons IC1 to IC9 are arranged in the operating status display area 210 in a position corresponding to the position of the cameras C1 to C9 in the multi-eye photography device 10 (in rear view). Therefore, they are arranged at regular angular intervals on the same circumference. In this embodiment, the camera icon IC5 of the fifth camera C5 is arranged at a 90-degree position, and the camera icons IC1 to IC9 of the cameras C1 to C9 are arranged at 30-degree intervals on the same circumference. Furthermore, the camera icons IC1 to IC9 of the cameras C1 to C9 are arranged in a direction that matches the orientation of the corresponding camera. For example, the camera icon IC5 of the fifth camera C5 is arranged facing directly upward, matching the orientation of the fifth camera C5.

[0117] In this embodiment, the camera icons IC1 to IC9 of the cameras C1 to C9 are an example of a first symbol or a first figure representing each camera. Also, information on a diagram formed by arranging the camera icons IC1 to IC9 of the cameras C1 to C9 in an arc is an example of fourth information on the relative positions of the cameras.

[0118] The operational status display area 210 also displays a diagram RS showing the general shape of the object to be photographed (hereinafter referred to as the "object outline diagram"). When the object to be photographed is a tunnel, the object outline diagram RS is composed of a diagram showing the general shape of the cross section of the tunnel. Figure 14 shows an example where the tunnel to be photographed is a horseshoe-shaped tunnel. If the tunnel to be photographed is a circular tunnel, a circular diagram is displayed; if it is a semicircular tunnel, a semicircular diagram is displayed; and if it is a rectangular tunnel, a rectangular diagram is displayed. The object outline diagram RS may be any diagram that allows the general shape of the object to be photographed to be grasped.

[0119] 14 shows an example in which the subject outline drawing RS is made up of broken lines. However, the representation of the subject outline drawing RS is not limited to this, and it can also be expressed by changing the line type, color, etc.

[0120] 14, when the object to be photographed is a tunnel, the camera icons IC1 to IC9 are arranged inside the tunnel diagram shown in the outline diagram RS of the object to be photographed, and are displayed in the operation status display area 210. This makes it easy to understand the settings (photographing direction, etc.) of each camera C1 to C9 relative to the object to be photographed.

[0121] As shown in FIG. 14, the operating status display area 210 further displays position information PP. The position information PP is information relating to the position of the carriage 2. In this embodiment, information on the distance from the starting point measured by the rangefinder 5 is displayed as the position information PP. FIG. 14 shows an example in which the position information PP is displayed inside the camera icons IC1 to IC9 arranged in an arc shape. By displaying the position information PP, the current shooting position can be easily grasped. In this embodiment, the position information PP is an example of third information relating to the position of the moving body.

[0122] As described above, in this embodiment, the operating status of each of the cameras C1 to C9 is displayed by the color of the camera icons IC1 to IC9.

[0123] FIG. 15 is a diagram showing an example of an output image when a malfunction occurs in the camera.

[0124] As shown in Fig. 15, if a malfunction occurs in any of the cameras C1 to C9 that make up the multi-eye photography device 10, the color of the camera icon for the malfunctioning camera is displayed inverted. Fig. 15 shows an example where a malfunction occurs in the second camera C2. In this case, the color of the camera icon IC2 for the second camera C2 is displayed inverted. For example, if the color of the camera icon (the color of the shape resembling a camera) is white in a normal state, it will change to black when a malfunction occurs.

[0125] The malfunctions include camera malfunctions based on the status information (insufficient battery, poor communication, etc.), as well as poor image capture. Figure 15 shows an example where the image captured by the second camera C2 is determined to be poorly captured.

[0126] In this way, by reversing the color of the camera icon of the camera where a malfunction has occurred, the operating status of the cameras C1 to C9 can be easily understood from the image, and the camera where the malfunction has occurred can be easily identified. In this embodiment, reversing the color of the camera icons IC1 to IC9 is an example of highlighting. The highlighting method is not limited to this, and other methods such as changing the color from the others or blinking can also be used.

[0127] As shown in FIG. 14, the operation status display area 210 further displays shooting mode information if1. The shooting mode information if1 is information on the currently set shooting mode. In this embodiment, "still image shooting," "interval shooting," or "video shooting" is displayed as the shooting mode information if1. FIG. 14 shows an example in which the current shooting mode is set to "interval shooting." In this embodiment, the shooting mode information if1 is located in the upper right corner of the operation status display area 210 (also in the upper right corner of the entire output image).

[0128] (5-2) Captured image display area FIG. 16 is an enlarged view of the image display area.

[0129] The captured image display area 220 displays images captured by the camera array imaging device 10. That is, images captured by each of the cameras C1 to C9 are displayed. The captured images are displayed after being reduced to a predetermined size (displayed as so-called thumbnail images). The most recent N captured images are also displayed, where N is an integer equal to or greater than 2 (N≧2). Therefore, multiple images including the most recent captured image are displayed.

[0130] As shown in Fig. 16, in this embodiment, the most recent six captured images imn (m = 1, 2, ..., 6, n = 1, 2, ..., 9) by each of the cameras C1 to C9 are arranged and displayed in a matrix. More specifically, the captured images by each of the cameras C1 to C9 are arranged at regular intervals in the row direction (n direction: horizontal direction in Fig. 16). Furthermore, the most recent six captured images by each of the cameras C1 to C9 are arranged at regular intervals in chronological order in the vertical direction (m direction: vertical direction in Fig. 16).

[0131] The arrangement of the captured images in the row direction (n direction) corresponds to the arrangement (arrangement in the circumferential direction) of the cameras C1 to C9 in the camera array 10 (the same arrangement as when the arrangement of the cameras C1 to C9 in the circumferential direction is laid out on a plane). Therefore, from left to right, the images captured by the first camera C1, the second camera C2, and so on are arranged in this order. In this embodiment, the row direction (n direction) is an example of the first direction.

[0132] The photographed images are arranged in the column direction (m direction) in the order they were photographed by each camera C1 to C9, and are arranged from top to bottom in order of oldest photographed date and time. Therefore, the photographed image arranged at the top is the most recent photographed image. On the other hand, the photographed image arranged at the bottom is the oldest photographed image currently being displayed. In this embodiment, the column direction (m direction) is an example of the second direction.

[0133] Camera icons ic1 to ic9 are placed at the top of the row of captured images imn of each camera C1 to C9 displayed in chronological order (the top position of the row). These camera icons ic1 to ic9 are configured with the same icon image as the camera icons IC1 to IC9 displayed in the operational status display area 210. In other words, they are configured with a graphic combining a figure resembling a camera and a number. Hereinafter, as necessary, the camera icons IC1 to IC9 displayed in the operational status display area 210 will be referred to as the "first camera icon," and the camera icons ic1 to ic9 displayed in the captured image display area 220 will be referred to as the "second camera icon" to distinguish between the two.

[0134] As shown in FIG. 15, like the first camera icons IC1 to IC9, the second camera icons ic1 to ic9 are displayed in inverted colors when a problem occurs in the corresponding camera C1 to C9.

[0135] In this way, by placing the camera icons ic1 to ic9 at the beginning of the columns of the images imn captured by each of the cameras C1 to C9, it becomes possible to grasp at a glance the correspondence between each of the cameras C1 to C9 and the captured images imn. In other words, it is possible to grasp at a glance which camera captured the images in each column.

[0136] Furthermore, if a problem occurs with any of the cameras C1 to C9, the color of the corresponding camera icon is displayed inverted, making it easy to grasp the occurrence of the problem and to identify the camera in which the problem occurred.

[0137] In this embodiment, the second camera icons ic1 to ic9 are an example of a second symbol or a second figure representing each camera. In addition, in this embodiment, the arrangement of the second camera icons ic1 to ic9 at the beginning of the captured image imn of each camera C1 to C9 is an example of an arrangement in which the two are associated with each other.

[0138] 16, remaining battery capacity information bi1 to bi9 of each of the cameras C1 to C9 is further displayed in the captured image display area 220. The remaining battery capacity information bi1 to bi9 is information on the remaining capacity of the battery installed in each of the cameras C1 to C9. In this embodiment, the remaining battery capacity information bi1 to bi9 is arranged between the captured image imn of each of the cameras C1 to C9 and the second camera icons ic1 to ic9. In other words, the remaining battery capacity information bi1 to bi9 of each of the cameras C1 to C9 is arranged below the second camera icons ic1 to ic9 of each of the cameras C1 to C9, followed by the captured image imn of each of the cameras C1 to C9.

[0139] Fig. 16 shows an example of displaying remaining battery capacity information bi1 to bi9 as numerical values. The numerical values ​​are displayed as a percentage, with a fully charged state being 100. Fig. 16 shows an example in which all cameras C1 to C9 have fully charged batteries (100%).

[0140] For cameras with remaining battery power below a threshold, the corresponding remaining battery power information is highlighted, for example, in a different color than in a normal state, or flashes.

[0141] In this way, by displaying the remaining battery capacity information bi1 to bi9 in association with the camera icons ic1 to ic9 and the captured images imn of the cameras C1 to C9, the remaining battery capacity status of each camera C1 to C9 can be grasped at a glance. Furthermore, by highlighting the corresponding remaining battery capacity information bi1 to bi9 when the remaining capacity is insufficient (below the threshold), the user can be alerted.

[0142] In this embodiment, the captured image display area 220 further displays information about the total number of captured images if2, information about the number of failed images if3, and a stop shooting button bt1.

[0143] The total number of shots information if2 is information on the cumulative number of shots since the start of shooting, and is a value for the entire device. Therefore, it is the total number of shots taken by the nine cameras C1 to C9. In this embodiment, as shown in Fig. 16, the total number of shots information if2 is placed in the lower left position of the captured image display area 220 (also in the lower left position of the entire output image).

[0144] The failure number information if3 is information on the number of images that failed to be captured (images that were determined to be poorly captured). As shown in Fig. 16, the failure number information if3 is arranged adjacent to the total number of images captured information if2.

[0145] The stop shooting button bt1 is a button that is displayed when interval shooting or video shooting is performed, and is a button that instructs stopping of shooting. When the stop shooting button bt1 is touched during interval shooting or video shooting, shooting stops. In this embodiment, as shown in FIG. 16, the stop shooting button bt1 is located in the lower right position of the captured image display area 220 (also in the lower right position of the entire output image).

[0146] FIG. 17 is a diagram showing an example of a display in the image display area when an imaging failure occurs.

[0147] FIG. 17 shows an example in which the latest captured image (the image captured last) by the second camera C2 is determined to be poorly captured (total determination is NG).

[0148] As shown in Fig. 17, when a shooting error occurs, an error mark Er is displayed at the display position of the captured image determined to be a shooting error. In the example shown in Fig. 17, the error mark Er is displayed at the display position of the most recent captured image by the second camera C2.

[0149] In this embodiment, the error mark Er is composed of a slot-shaped figure having a predetermined color, and position information is displayed inside the figure. The position information is information about the shooting position. More specifically, it is information about the position of the cart at the point where the photographed image determined to be poorly photographed was taken. In this embodiment, information about the distance from the starting point is displayed.

[0150] In this way, by combining information on the position where the imaging error occurred and displaying the error mark Er, it is possible to easily identify the position when re-imaging. The color of the figure that constitutes the error mark Er (the background color of the display of the position information) is preferably a color that calls attention, such as red. In this embodiment, the error mark Er is an example of second information that indicates an imaging error.

[0151] (6) Output control section The output control unit 111K outputs the output image 200 generated by the output image generation unit 111J to the display unit 114, which is the display destination.

[0152] [Effects of the shooting system] Hereinafter, a case where the inner wall surface of a tunnel is photographed using the photographing system 1 of this embodiment will be described.

[0153] As described above, photography is performed while moving within the tunnel. In this embodiment, the multi-eye photography device 10 is moved manually using a hand-pushed cart 2. The cart 2 travels within the tunnel on rails laid within the tunnel.

[0154] In this embodiment, an example will be described in which interval photography is performed while moving through a tunnel to photograph the inner wall surface of the tunnel.

[0155] (1) Preparation for filming First, the multiple-eye photographing device 10 is mounted on the dolly 2, and the multiple-eye photographing device 10 is positioned at the photographing start position (the starting point of photographing).

[0156] Next, the imaging field angle and placement position, etc. of each of the cameras C1 to C9 mounted on the camera array 10 are adjusted. That is, the imaging field angle and placement position, etc. of each of the cameras C1 to C9 are adjusted so that imaging can be performed without omission in the circumferential direction. At this time, the requirements for the overlap rate between adjacent cameras (for example, an overlap rate of 20% or more) are set.

[0157] Next, the control device 100 is connected. That is, the control device 100 is set to a state in which it can communicate with the multi-eye photographing device 10 and the rangefinder 5. This makes it possible for the control device 100 to control the multi-eye photographing device 10 and the rangefinder 5. Furthermore, the control device 100 can acquire information on images photographed by the multi-eye photographing device 10 and distances measured by the rangefinder 5.

[0158] At the shooting start position, the rangefinder 5 is reset and set to measure distance from 0.

[0159] (2) Shooting After preparation for shooting is complete, shooting begins. As described above, shooting is performed by interval shooting. In this case, the shooting interval is set to a time interval that allows for complete shooting in the direction of movement. In this embodiment, the dolly 2 is moved manually. Therefore, when a person moves at walking speed while pushing the dolly 2, the time interval is set to a time interval that allows for complete shooting.

[0160] The control device 100 sets the shooting mode and instructs the start of shooting. The user sets the shooting mode and instructs the start of interval shooting on the screen of the display unit 114. The control device 100 starts interval shooting in response to the instruction to start shooting from the user. After instructing the start of shooting, the user also starts moving. That is, the user moves through the tunnel while pushing the dolly 2.

[0161] As described above, interval photography involves repeatedly capturing still images at regular time intervals. Thus, images are captured in chronological order. Each of the cameras C1 to C9 mounted on the camera array 10 captures images in synchronization.

[0162] When shooting starts, a predetermined output image 200 is displayed on the screen of the display unit 114 (see FIG. 14). As described above, the images (captured images) captured by each of the cameras C1 to C9 are displayed in chronological order for each camera in the output image 200. The captured images of each of the cameras C1 to C9 are displayed in a captured image display area 220 within the output image 200.

[0163] 18 and 19 are diagrams showing how the captured image display area changes as the shooting progresses. In Fig. 18 and Fig. 19, the numbers in the frames showing the images captured by each camera indicate the order of shooting. For example, the number "1" indicates the first image captured, and the number "2" indicates the second image captured.

[0164] Fig. 18(A) shows the display of the captured image display area 220 when the first image is captured. As shown in Fig. 18(A), the images captured by the cameras C1 to C2 (first captured images) are displayed in a horizontal row.

[0165] Fig. 18(B) shows the display of the captured image display area 220 when the second image is captured. As shown in Fig. 18(B), when the second image is captured, the first captured image is moved down one row and displayed. Then, the second captured image (the most recently captured image) is displayed in the top position.

[0166] Fig. 18(C) shows the display of the captured image display area 220 when the third image is captured. As shown in Fig. 18(C), when the third image is captured, the first captured image and the second captured image are each moved down one row and displayed. Then, the third captured image (the most recent captured image) is displayed in the top position.

[0167] 19(A) shows the display of the captured image display area 220 when the sixth image is captured. In this embodiment, six is ​​the upper limit of the number of captured images that can be displayed for each camera.

[0168] 19(B) shows the display in the captured image display area 220 when the seventh image has been taken. When more than six images have been taken, the oldest images are deleted and the display is updated accordingly. Therefore, when the seventh image is taken, the first captured image is deleted.

[0169] In this way, the oldest images are deleted and the display is updated so that the most recently captured image is always displayed at the top of the column.

[0170] Fig. 19(C) shows the display of the captured image display area 220 when a capturing error occurs. In particular, Fig. 19(C) shows an example in which a capturing error occurs in the image captured by the second camera C2 when the eighth image is captured. In this case, an error mark Er is displayed in the display position (top position in the column) of the most recent captured image by the second camera C2. As described above, the error mark Er displays the position information of the cart 2 when the image was captured. The display of the error mark Er makes it easy to confirm that a capturing error has occurred.

[0171] Furthermore, if a problem such as poor photography occurs, the camera icon (second camera icon) of the camera where the problem occurred is highlighted. In the example shown in Fig. 19(C), the second camera icon of the second camera C2 is highlighted (inverted in this embodiment). This makes it easy to identify the camera where the problem occurred.

[0172] If a malfunction occurs, the camera icon (first camera icon) of the camera in which the malfunction occurred is also highlighted in the operational status display area 210 (see FIG. 15).

[0173] In addition, the position information PP is updated and displayed in the captured image display area 220 as the dolly 2 moves, thereby allowing the current position to be confirmed.

[0174] The above example is an example in which the captured image is determined to be poorly captured, but if the remaining battery charge in each camera C1 to C9 falls below a threshold, the camera icon and remaining battery charge information of the camera whose remaining battery charge falls below the threshold are highlighted. Note that if the battery runs out and capture is no longer possible, an error mark Er is displayed in the corresponding position, just as when capture is determined to be poorly captured.

[0175] Similarly, if communication is not possible, the camera icon for the camera that is unable to communicate will be highlighted. Note that if communication is not possible, no captured images can be obtained, so an error mark Er will be displayed in the corresponding position, just as if the image was judged to be poorly captured.

[0176] When a problem such as poor imaging occurs, the user stops imaging as soon as they recognize the problem, returns to the position where the problem occurred, and resumes imaging.

[0177] As described above, according to the photography system 1 of this embodiment, when a target is photographed using multiple cameras, the most recent N images (six images in this embodiment) taken by each camera are output to a display destination in a predetermined arrangement. If a photography defect occurs, an error mark Er is displayed instead of the photographed image. This makes it easy to grasp the occurrence of a defect. Furthermore, the photography status of each camera can be easily confirmed.

[0178] Furthermore, the images captured by each camera are displayed in an order corresponding to the arrangement of the cameras, and are also displayed in chronological order for each camera, so that if a poor image capture occurs, the target can be easily identified.

[0179] Furthermore, since the error mark Er includes information indicating the photographing position, the photographing position can be easily identified even when re-photographing. In particular, in this embodiment, information on the current position of the carriage 2 (position information PP) is displayed in the operation status display area 210, so the position for re-photographing can be easily identified from the difference. Therefore, for example, even if the carriage 2 moves forward by inertia after a photographing error occurs, it is easy to identify how far it should move back. Furthermore, since the photographed image display area 220 displays images photographed by adjacent cameras and photographed images in chronological order, these images can be referred to when adjusting the position for re-photographing. This makes it easy to adjust the position.

[0180] [Variations] [Still image capture] The basic process of still image shooting is the same as that of interval shooting, except that the user issues a shooting instruction each time. That is, each time shooting is performed, a captured image is output from each of cameras C1 to C9 to control device 100. Then, based on the obtained captured image, output image 200 is generated and output to display unit 114.

[0181] [Video recording] Video shooting is started in response to a user's instruction to start shooting, and is ended in response to a user's instruction to end shooting.

[0182] In the case of video shooting, the images constituting each frame of the video are displayed as shot images in the shot image display area 220. In this case, for example, images are extracted at a predetermined frame interval and displayed in the shot image display area 220. On the other hand, image judgment is performed for each frame. If a shooting problem occurs, for example, an error mark Er is displayed at the position of the frame that was to be extracted immediately after the problem occurred.

[0183] [Enlarged view of the captured image] It is preferable that the captured images imn of the cameras C1 to C9 displayed in the captured image display area 220 of the output image 200 can be enlarged and displayed.

[0184] FIG. 20 is a diagram showing an example of an enlarged display.

[0185] The user touches and selects the photographed image that they wish to enlarge. When the image is selected, an enlarged display window W1 is displayed superimposed on the output image 200, and the selected photographed image is enlarged and displayed in the enlarged display window W1.

[0186] It is preferable that the captured image enlarged and displayed in the enlarged display window W1 can be enlarged, reduced, moved, etc. within the enlarged display window W1. Enlargement and reduction operations are performed, for example, by pinching out, pinching in, sliding, etc. on the image.

[0187] The enlarged display window W1 is closed by touching the close button bt2 displayed in the enlarged display window W1, which ends the enlarged display.

[0188] In this way, by configuring the captured image to be displayed in an enlarged form, it becomes easier to check the capture position when, for example, re-capturing an image.

[0189] [Scrolling display of captured images] The captured images imn of each camera C1 to C9 displayed in the captured image display area 220 of the output image 200 may be configured to be scrollable in the direction of the chronological order (column direction). This allows a series of captured images to be checked. The scrolling operation is performed, for example, by a slide operation, a flick operation, or the like on the captured image display area 220.

[0190] [alarm] The control device 100 may be equipped with an audio output unit such as a speaker, and configured to issue an alarm when an imaging problem occurs, etc. This allows the user to more easily recognize the occurrence of a problem.

[0191] [Automatic stop of recording] In the case of interval photography, the photography may be automatically stopped when a photography error occurs.

[0192] FIG. 21 is a flowchart showing an example of a processing procedure of the control device when photographing is automatically stopped.

[0193] In response to a command from a user to start shooting, the control device 100 simultaneously commands all the cameras C1 to C9 to start shooting (step S1).

[0194] The control device 100 acquires images (photographed images) captured by all the cameras C1 to C9 (step S2).

[0195] The control device 100 checks the acquired images captured by all cameras C1 to C9 (step S3), that is, checks the image quality and overlap rate of the captured images.

[0196] Based on the check results, the control device 100 determines whether or not all of the cameras C1 to C9 have successfully taken images (step S4). That is, it determines whether or not the check results for the images taken by all of the cameras C1 to C9 are OK images.

[0197] If all of the cameras C1 to C9 have successfully captured images, the control device 100 updates the output image 200 (step S5). At this time, the control device 100 updates the output image 200 by treating the acquired captured image as the most recent captured image. That is, the control device 100 places the acquired captured image at the top position in the chronological order and updates the output image 200.

[0198] After updating the output image 200, the control device 100 determines whether or not the user has instructed to end the photographing (step S6). If the user has instructed to end the photographing, the control device 100 ends the interval photographing.

[0199] On the other hand, if an instruction to end shooting has not been given, shooting continues. In this case, the process returns to step S1, and at the next shooting timing, all cameras C1 to C9 are simultaneously instructed to start shooting.

[0200] In step S4, if it is determined that none of the cameras C1 to C9 are capturing images normally (if the determination in step S4 is "No"), the control device 100 suspends the interval capturing (step S7). If the control device 100 is equipped with an audio output unit such as a speaker, it is preferable to issue an alarm simultaneously with the suspension.

[0201] After the shooting is interrupted, the control device 100 acquires the position information of the dolly 2 at the point where the shooting error occurred (step S8) and updates the output image 200 (step S9). In this case, the output image 200 is updated to an image including an error display. That is, as shown in FIG. 15, the camera icon of the camera where the shooting error occurred is highlighted, and an error mark Er is displayed at the display position of the shot image. The error mark Er displays the position information of the dolly 2 at the point where the shooting error occurred.

[0202] The user determines the position for re-shooting from the output image 200 displayed on the display unit 114. Then, the user returns to the determined position and resumes shooting, that is, gives an instruction to start shooting.

[0203] In this way, imaging is automatically stopped when an imaging error occurs, which makes it possible to prevent redundant imaging.

[0204] Although the above example is for interval shooting, the automatic stop can also be configured in the same way for video shooting.

[0205] [Display of captured image] In the above embodiment, the images captured by the cameras C1 to C9 are simply arranged in a matrix and displayed, but the display mode of the images captured by the cameras C1 to C9 is not limited to this.

[0206] (1) Display with changed brightness FIG. 22 is a diagram showing another example of the display of a captured image in the captured image display area.

[0207] FIG. 22 shows an example of a case where the brightness (lightness) of images is changed in the time series direction (column direction). The older the image captured by each of cameras C1 to C9, the darker it is displayed. Therefore, in the example shown in FIG. 22, the darker the image is displayed as it moves downwards. It is preferable to display the image displayed at the top (the most recently captured image) without changing the brightness. For example, the images in each column (the images captured by each of cameras C1 to C9) are displayed with the brightness gradually decreasing at a fixed rate from top to bottom, using the image displayed at the top as the reference.

[0208] In this way, by changing the brightness of the images displayed in chronological order, it is possible to easily recognize the most recent images captured by each of the cameras C1 to C9 on the screen, and it is also possible to intuitively recognize that the images are being captured while moving.

[0209] In this example, the brightness of the image is changed, but the saturation may be changed in addition to or instead of the brightness. In other words, the color tone of the image may be changed.

[0210] Note that Fig. 22 also shows other examples of camera icons ic1 to ic9. Fig. 22 shows an example in which camera icons ic1 to ic9 for cameras C1 to C9 are configured by combining a graphic that resembles a camera with a graphic that resembles a battery. The number of the corresponding camera C1 to C9 is displayed inside the graphic that resembles a camera. The graphic that resembles a battery also serves as a battery level indicator, and the internal display changes depending on the remaining battery level of the corresponding camera C1 to C9.

[0211] (2) Resized display FIG. 23 is a diagram showing another example of the display of a captured image in the captured image display area.

[0212] Fig. 23 shows an example of a case where images are displayed with different sizes in the time series direction (column direction). The older the image captured by each of cameras C1 to C9, the smaller the image is displayed. Therefore, in the example shown in Fig. 23, the smaller the image is displayed as it moves downwards. For example, the images in each column are displayed with their size gradually reduced at a fixed ratio from top to bottom, with the image displayed at the top (the most recent captured image) being the base image.

[0213] In this way, by changing the size of the images displayed in the chronological order, it is possible to easily recognize the latest captured images from each of the cameras C1 to C9 on the screen, and it is also possible to intuitively recognize that the images are being captured while moving.

[0214] Note that Fig. 23 also shows another example of the display format of the remaining battery amount information bi1 to bi9. Fig. 23 shows an example of the case where the remaining battery amount is displayed numerically within a graphic that resembles a battery.

[0215] (3) Display with changed brightness and size FIG. 24 is a diagram showing another example of displaying a captured image in the captured image display area.

[0216] Fig. 24 shows an example of a case where the brightness and size of images are changed in the time series direction (column direction) and displayed. The older the image captured by each of cameras C1 to C9, the darker and smaller it is displayed. Therefore, in the example shown in Fig. 24, the darker and smaller the image is displayed as it moves downwards.

[0217] It is preferable that the image displayed at the top (the most recently captured image) is displayed without changing the brightness. For example, the images in each column are displayed with the brightness gradually decreased at a fixed rate from top to bottom, with the image displayed at the top as the reference. Also, the size of the images is gradually reduced at a fixed rate from top to bottom, with the image displayed at the top as the reference.

[0218] In this way, by changing the brightness and size of the images displayed in chronological order, it becomes easier to recognize the latest images captured by each camera C1 to C9 on the screen, and it also makes it easier to intuitively recognize that the images are being captured while moving.

[0219] In addition to or instead of brightness, the saturation may be changed, i.e., the color tone of the image may be changed.

[0220] Note that Fig. 24 also shows other examples of camera icons ic1 to ic9 and remaining battery capacity information bi1 to bi9. Fig. 24 shows an example in which camera icons ic1 to ic9 for cameras C1 to C9 are configured by combining a camera-like graphic with a number. For remaining battery capacity information bi1 to bi9, an example is shown in which a graphic display and a numerical display are combined.

[0221] [Display error cause] It is more preferable to have a configuration in which, when an imaging failure occurs, the cause of the failure, i.e., the cause of the determined imaging failure (cause of the error), can be notified. For example, a configuration in which the cause of the error is displayed on the screen can be adopted.

[0222] FIG. 25 is a diagram showing an example of a screen display of the cause of the error.

[0223] Fig. 25 shows an example in which a predetermined window (error cause display window) W2 is arranged within the output image 200, and the cause of the error is displayed in the error cause display window W2. In Fig. 25, in addition to the cause of the error, information on the location where the error occurred is also displayed. Note that Fig. 25 shows an example in the case of so-called out-of-focus. For example, when an error mark Er is touched, the error cause display window W2 is configured to be displayed near the touched error mark Er.

[0224] When the captured image is determined to be NG, the image determination unit 111G outputs the determination result and the reason for the NG determination to the output image generation unit 111J. The output image generation unit 111J generates an output image 200 based on the determination reason and displays an error cause display window W2.

[0225] In the above example, the reason for the determination that the imaging was poor is displayed as the cause of the error, but a more detailed error cause (such as the cause of the error) may be estimated and displayed. For example, a configuration may be adopted in which the cause of the imaging failure is estimated from the circumstances under which the imaging failure occurred and displayed.

[0226] Generally, when taking pictures while moving, poor quality images often occur due to the following reasons.

[0227] (1) When the imaging error is isolated If the shooting error is a single incident, it is highly likely that the cause of the error lies on the subject side. For example, when shooting a tunnel, the presence of an object on the inner wall of the tunnel often causes the shooting error.

[0228] (2) When a specific camera continues to fail to capture images If a particular camera continues to fail to take pictures properly, it is highly likely that the cause of the problem lies within the camera itself. For example, the camera may be dirty or have water droplets on the lens, or the camera itself may be malfunctioning.

[0229] (3) When multiple cameras periodically experience poor image quality If multiple cameras periodically experience poor image capture, it is highly likely that the cause of the poor image capture lies on the moving object. For example, if there is a problem with the wheels of a cart (deformation, shaking, etc.), multiple cameras will periodically experience poor image capture.

[0230] The control device 100 estimates the cause of the error from, for example, a history of image determination results by the image determination unit 111G, and outputs the estimation result. For the estimation, for example, a trained model that has been machine-learned to estimate the cause of the error from the history of image determination results can be used.

[0231] Furthermore, when the cause of an error is estimated and output, it is preferable to also output a countermeasure. For example, if it is estimated that the poor image quality is due to a dirty lens, the countermeasure can be to wipe the lens. Also, if it is estimated that the poor image quality is due to a moving object, the countermeasure can be to check the moving object, etc.

[0232] [subject] The above embodiment has been described taking an example of photographing a tunnel 3, but the subject is not limited to this. The cameras are arranged in a layout according to the subject, and a configuration is assumed in which a pair of cameras with overlapping photographing areas is included.

[0233] [Moving object] In the above embodiment, a hand-pushed dolly 2 is used as a moving body for moving the multiple-eye photography device 10, but the configuration of the moving body is not limited to this. It may also be configured to have a power source (including a so-called electrically assisted function).

[0234] The moving body may also be configured to be self-propelled. In the case of a self-propelled configuration, it is preferable that the moving body be configured to interrupt imaging and stop moving when a problem such as an imaging error occurs. In this case, the moving body may also be configured to automatically return to the point where the imaging error occurred. Furthermore, the moving body may also be configured to automatically re-image and resume imaging.

[0235] The moving object may also be configured to fly (for example, a drone, etc.).

[0236] [How to obtain location information of a moving object] In the above embodiment, the distance from the starting point is measured to identify the position of the cart (moving body), but the method for identifying the position of the moving body is not limited to this. For example, a configuration can be adopted in which the position of the moving body is identified using a self-position estimation technique using SLAM (Simultaneous Localization and Mapping). Since the SLAM technique itself is well known, a detailed description thereof will be omitted. As an example, SLAM using LiDAR (Light Detection and Ranging or Laser Imaging Detection and Ranging), SLAM using a camera, etc. can be adopted.

[0237] Alternatively, the position of the mobile body may be measured by a positioning system using a beacon, a global positioning system (GPS), an indoor messaging system (IMES), Wi-Fi, or an ultra-wideband (UWB), etc. Also, various sensors such as an acceleration sensor, a magnetic sensor, and an angular velocity sensor may be installed on the mobile body, and the position of the mobile body may be estimated based on values ​​obtained from each sensor.

[0238] [System Configuration] The multi-eye photographing device 10 and the control device 100 can also be configured to be communicably connected via a network such as the Internet.

[0239] The functions of the control device can also be realized by a so-called cloud computer. In this case, for example, a terminal owned by a user (such as a personal computer, smartphone, or tablet) can be used as an input device, and the display of the terminal can be used as a display destination.

[0240] [Hardware configuration of image processing device] The functions of the image processing device are realized by various processors. The various processors include a CPU and / or a GPU (Graphic Processing Unit), which are general-purpose processors that execute programs and function as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically to execute specific processing. The term "program" is synonymous with "software."

[0241] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types. For example, a single processing unit may be configured with multiple FPGAs, or a combination of a CPU and an FPGA. Alternatively, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a configuration in which a single processor is configured with a combination of one or more CPUs and software, as typified by computers used as clients or servers, and this processor functions as multiple processing units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a system on chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0242] [Note] The above modifications can be used in any suitable combination.

[0243] Furthermore, in this specification, the terms "same" and "identical" mean not only completely identical but also "almost identical," which includes tolerances allowed in design and manufacturing. Furthermore, in this specification, the term "coaxial" means not only completely coaxial but also "almost coaxial," which includes tolerances allowed in design and manufacturing. Furthermore, in this specification, the term "orthogonal" means not only completely orthogonal but also "almost orthogonal," which includes tolerances allowed in design and manufacturing. Furthermore, in this specification, the term "parallel" means not only completely parallel but also "almost parallel," which includes tolerances allowed in design and manufacturing. Furthermore, in this specification, the term "synchronized" means not only completely synchronized but also within a range that is recognized as substantially synchronized (the meaning of "almost synchronized"). Furthermore, in this specification, the term "simultaneous" means not only completely simultaneous but also within a range that is recognized as substantially simultaneous (the meaning of "almost simultaneous"). [Explanation of symbols]

[0244] 1. Shooting system 2...Cart 3. Tunnel 4...Rail 5…Distance meter 10...Multi-lens camera 11...Frame 12...Bass 13F...Front column 13R…Rear column 14F…Front panel 14R...Rear panel 20...Relay device 100...Control device 111...Processor 111A...Camera control unit 111B...Lighting control unit 111C...Range finder control unit 111E...Data acquisition section 111E1...Image acquisition unit 111E2...Status information acquisition unit 111F…Distance information acquisition unit 111G...Image judgment section 111G1...Image quality judgment section 111G2…Duplicity rate determination section 111G2a...Overlapping range detection unit 111G2b…duplication rate calculation section 111G2c…Overlap rate pass / fail judgment section 111G3…General Judging Section 111H...Status determination section 111J...Output image generation section 111K...Output control section 112...Main memory section 113…Auxiliary storage unit 114...Display section 115...Operation unit 116…Communications Department 200...Output image 210...Operation status display area 220...Captured image display area Ax...axis of the multi-lens camera B1...Bracket (first bracket) B2...Bracket (second bracket) B3...Bracket (third bracket) B4...Bracket (4th bracket) B5...Bracket (5th bracket) B6...Bracket (6th bracket) B7…Bracket (7th bracket) B8...Bracket (8th bracket) B9…Bracket (9th bracket) C1...Camera (first camera) C2...Camera (second camera) C3...Camera (third camera) C4...Camera (4th camera) C5...Camera (5th camera) C6...Camera (6th camera) C7...Camera (7th camera) C8...Camera (8th camera) C9...Camera (9th Camera) CL...Clamp Er...Error mark L1…Lighting device (1st lighting device) L2…Lighting device (second lighting device) L3…Lighting device (3rd lighting device) L4...Lighting device (fourth lighting device) L5…Lighting device (5th lighting device) L6…Lighting device (6th lighting device) L7…Lighting device (7th lighting device) L8…Lighting device (8th lighting device) L9…Lighting device (9th lighting device) PP…location information RS...Outline of the subject being photographed U1...Photography unit (first photography unit) U2...Filming unit (second filming unit) U3...Photography unit (third photography unit) U4...Photography unit (4th photography unit) U5...Photography unit (5th photography unit) U6...Photography unit (6th photography unit) U7...Photography unit (7th photography unit) U8...Photography unit (8th photography unit) U9...Photography unit (9th photography unit) W1: Enlarged display window W2: Error cause display window bi1...Battery remaining capacity information bi2…Battery remaining capacity information bi3…Battery remaining information bi4…Battery remaining information bi5…Battery remaining information bi6…Battery remaining information bi7…Battery remaining information bi8…Battery remaining information bi9…Battery remaining information bt1... Stop recording button bt2...close button IC1...Camera icon (first camera icon for the first camera) IC2...Camera icon (first camera icon for second camera) IC3...Camera icon (first camera icon for the third camera) IC4...Camera icon (first camera icon for the fourth camera) IC5...Camera icon (first camera icon of the fifth camera) IC6...Camera icon (first camera icon of the sixth camera) IC7...Camera icon (first camera icon of the seventh camera) IC8...Camera icon (first camera icon for the eighth camera) IC9...Camera icon (first camera icon of the 9th camera) ic1...Camera icon (secondary camera icon for the first camera) ic2...Camera icon (second camera icon for the second camera) ic3...Camera icon (second camera icon for the third camera) ic4...Camera icon (second camera icon for the fourth camera) ic5...Camera icon (second camera icon for the fifth camera) ic6...Camera icon (second camera icon for the sixth camera) ic7...Camera icon (second camera icon for the seventh camera) ic8...Camera icon (second camera icon for the 8th camera) ic9...Camera icon (second camera icon for the 9th camera) if1...Shooting mode information if2...Total number of shots if3...Failed number information imn(m=1,2,…,6, n=1,2,…,9)…Captured images

Claims

1. An image processing device that processes first images captured in time series by a plurality of cameras, a processor; The processor: acquiring the first image; obtaining first information relating to a capture condition of the first image; arranging the most recent N first images from the plurality of cameras in the chronological order, where N≧2, and generating a second image including second information indicating a shooting defect of the first image based on the first information; outputting the second image to a display destination; Image processing device.

2. the processor arranges the first images from the plurality of cameras in a first direction, and generates the second image by arranging the most recent N first images from the plurality of cameras in a second direction in chronological order. The image processing device according to claim 1 .

3. The plurality of cameras are arranged in a predetermined arrangement, the processor generates the second image by arranging the latest first images of the plurality of cameras in the first direction in an arrangement corresponding to the arrangement of the plurality of cameras. The image processing device according to claim 2 .

4. the plurality of cameras are mounted on a moving body and move integrally, and capture the first images as the moving body moves; The image processing device according to claim 3 .

5. The processor: acquiring third information relating to the location of the moving object; generating the second image further including the third information; The image processing device according to claim 4 .

6. The third information is information about a distance from a starting point. The image processing device according to claim 5 .

7. the second information includes information regarding the position of the moving object at the time of capturing the first image determined to be poorly captured; The image processing device according to claim 5 .

8. the processor generates the second image, which further includes fourth information regarding a positional relationship between the plurality of cameras. The image processing device according to claim 1 .

9. the fourth information is information of a diagram in which first symbols or first figures representing the individual cameras are arranged in correspondence with the arrangement of the plurality of cameras; The image processing device according to claim 8 .

10. The fourth information further includes information of a diagram showing an outline of a subject to be photographed. The image processing device according to claim 9 .

11. the processor highlights and displays the first symbol or the first graphic of the camera in which a photographing error has occurred. The image processing device according to claim 9 .

12. the processor arranges a second symbol or a second graphic representing each of the cameras, and generates the second image in which the most recent N first images of the corresponding camera are arranged in the chronological order in association with the second symbol or the second graphic. The image processing device according to claim 1 .

13. the processor highlights and displays the second symbol or the second graphic of the camera in which a photographing error has occurred. The image processing device according to claim 12.

14. The processor: obtaining fifth information regarding the status of the plurality of cameras; generating the second image further including the fifth information of the plurality of cameras; The image processing device according to claim 1 .

15. the fifth information includes information on a remaining battery capacity; The image processing device according to claim 14.

16. the processor generates the second image in which the most recent N first images of the corresponding cameras are arranged in the chronological order in association with the fifth information of the plurality of cameras. The image processing device according to claim 14.

17. the processor generates the second images by arranging the most recent N first images from the plurality of cameras in the chronological order while changing at least one of the size and color tone. The image processing device according to claim 1 .

18. the processor analyzes the first image to obtain the first information. The image processing device according to claim 1 .

19. The processor: Estimating a cause of the imaging failure based on the first information; generating the second image further including an estimation result of the cause of the imaging failure; The image processing device according to claim 1 .

20. An image processing method for processing first images captured in time series by a plurality of cameras, comprising: acquiring the first image; obtaining first information relating to a capture condition of the first image; arranging the most recent N first images from the plurality of cameras in the chronological order, where N≧2, and generating a second image including second information indicating a shooting defect of the first image based on the first information; outputting the second image to a display destination; Image processing methods.

21. An image processing program for processing first images captured in time series by a plurality of cameras, acquiring the first image; a function of acquiring first information relating to a capture state of the first image; a function of arranging the most recent N first images from the plurality of cameras in the chronological order, where N≧2, and generating a second image including second information indicating a shooting defect of the first image based on the first information; a function of outputting the second image to a display destination; An image processing program that enables a computer to achieve this.

Images