Information processing system, method, program, information processing apparatus, and photographing device

Abstract

To appropriately control an exposure condition so that a photographed image has desired brightness.SOLUTION: A photographing device includes: a photographing unit 411 that photographs an image while moving; an exposure condition control unit 412 that controls an exposure condition for the photographing unit; and an image recording unit 413 that stores a first image photographed by the photographing unit in an exposure condition according to first control in which the exposure condition control unit performs automatic exposure control, in association with an exposure condition during the photographing. A terminal device includes an operation receiving unit 424 that receives an operation to select part of the first image on a screen to be displayed including the first image stored in the image storage unit, stores an exposure condition corresponding to a portion selected according to the operation received by the operation receiving unit as a selection exposure condition, and acquires the selected exposure condition associated with the portion received by the operation receiving unit from the exposure condition storage unit. The photographing unit includes the exposure condition storage unit 422 that photographs a second image in an exposure condition according to second control based on the selected exposure condition acquired by the exposure condition control unit.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing system, a method, a program, an information processing device, and an imaging device. [Background technology]

[0002] While there is a known technology for capturing images while a vehicle equipped with a camera is moving, various conditions related to the capture may change as the vehicle moves, resulting in a degradation of image quality.

[0003] Regarding this point, Japanese Patent Laid-Open No. 2010-239479 (Patent Document 1) discloses a configuration that determines whether or not there is a light change section within a predetermined distance ahead of the vehicle where the brightness changes by more than a predetermined amount, and changes the exposure amount of the imaging unit in accordance with the change in brightness that occurs when the vehicle travels through the light change section. According to Patent Document 1, the imaging unit can be adjusted to follow the change in brightness while the vehicle is traveling.

[0004] However, Patent Document 1 does not disclose a method for photographing a subject under exposure conditions that will result in a photographed image with desired brightness.

[0005] Therefore, there is a need for a technology to control exposure conditions so that the captured image has a desired brightness. Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention has been made in consideration of the problems in the above-mentioned conventional technology, and aims to provide an information processing system, method, program, information processing device, and photographing device that appropriately control exposure conditions during photographing so that the photographed image has the desired brightness. [Means for solving the problem]

[0007] That is, according to the present invention, An imaging means for capturing images while moving; a control means for controlling the exposure conditions of the photographing means; a storage means for storing a first image captured by the photographing means under exposure conditions under first control in which automatic exposure control is performed by the control means, in association with the exposure conditions at the time of photographing; a receiving means for receiving an operation to select a part of the first image on a screen displayed including the first image stored in the storage means; Including, the storage means stores, as selected exposure conditions, exposure conditions corresponding to the location selected by the operation accepted by the acceptance means; the control means acquires from the storage means the selected exposure conditions associated with the location accepted by the accepting means; the photographing means photographs a second image under exposure conditions by second control based on the selected exposure conditions acquired by the control means; An information processing system is provided. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an information processing system, method, program, information processing device, and photographing device that appropriately control exposure conditions during photographing so that the photographed image has a desired brightness. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of an information processing system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram showing an example of a hardware configuration included in an information processing system according to an embodiment of the present invention. [Figure 3] 3A and 3B are diagrams illustrating a camera's shooting reference position and an illumination surface in the present embodiment. [Figure 4] FIG. 10 is a diagram showing another example of the hardware configuration included in the information processing system of the present embodiment. [Figure 5] FIG. 2 is a block diagram of software included in the information processing system of the present embodiment. [Figure 6]10A and 10B are diagrams showing an example of photographing the wall surface of a tunnel in the automatic exposure control mode in this embodiment. [Figure 7] 10 is a flowchart showing a process executed by an information processing system that photographs a tunnel wall surface in this embodiment. [Figure 8] 5A to 5C are diagrams illustrating examples of capturing images under various exposure conditions in this embodiment. [Figure 9] FIG. 2 is a diagram showing an example of a data structure according to the present embodiment. [Figure 10] FIG. 4 is a diagram showing an example of a screen for selecting exposure conditions in the present embodiment. [Figure 11] 5A and 5B are diagrams showing examples of images captured in a fixed exposure control mode in the present embodiment. [Figure 12] FIG. 10 is a diagram showing an example of a screen for selecting exposure conditions in another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present invention will be described below with reference to embodiments, but the present invention is not limited to the embodiments described below. In the drawings referred to below, the same reference numerals will be used for common elements, and their description will be omitted as appropriate.

[0011] Fig. 1 is a diagram showing a schematic configuration of an information processing system 1 according to this embodiment. Fig. 1(a) shows the overall hardware configuration of the information processing system 1, and Fig. 1(b) is a diagram illustrating an example of capturing an image of a tunnel wall, which is an example of an object captured by a vehicle 10 according to this embodiment.

[0012] As shown in FIG. 1(a), the information processing system 1 of this embodiment includes an image capturing device 110, a terminal device 120, and a server device 130. Each piece of hardware can be connected via a network 140 such as the Internet or a LAN. The method of connecting each piece of hardware to the network 140 may be either wired or wireless. Furthermore, the image capturing device 110 and the terminal device 120 may be connected and communicate with each other via a cable or short-range wireless communication, without the use of the network 140.

[0013] The image capturing device 110 is a device that captures images and includes a camera, various sensors, and the like. The image capturing device 110 of this embodiment is mounted on a moving body such as a vehicle 10 that travels on roads, railroads, and the like, and can capture images while the moving body is moving. For example, the image capturing device 110 can be installed so that the optical axis of the camera faces a direction rotated 90 degrees to the left with respect to the direction of travel of the vehicle 10. This allows the image capturing device 110 to capture a panoramic image that unfolds in the direction of travel of the vehicle 10 by capturing images while the vehicle 10 is traveling.

[0014] Furthermore, the image capturing device 110 can capture images and acquire sensor data related to the image capturing situation using various sensors. The sensors included in the image capturing device 110 of this embodiment can, for example, measure the speed of the traveling vehicle 10. The sensors included in the image capturing device 110 can also, for example, measure the distance from the image capturing reference position to the subject.

[0015] The terminal device 120 is, for example, an information processing device such as a personal computer. The terminal device 120 of this embodiment can view images captured by the image capturing device 110 and perform operations related to various image processing. The terminal device 120 is not limited to the form of a personal computer as shown in FIG. 1(a), and may be, for example, a tablet terminal. The terminal device 120 of this embodiment may be mounted on the vehicle 10 and connected to the image capturing device 110.

[0016] The server device 130 is, for example, an information processing device such as a server computer. The server device 130 of this embodiment can store various data, for example, images captured by the image capturing device 110.

[0017] 1(a) is an example of a system with a so-called cloud configuration in which a server device 130 provides a service for creating a record of the present embodiment, but this does not particularly limit the embodiment. Therefore, for example, it may be a so-called standalone configuration in which processing is executed on an application of a terminal device 120.

[0018] As shown in Fig. 1(b), the vehicle 10 of this embodiment captures images of the tunnel wall while traveling. In the example shown in Fig. 1(b), the imaging device 110 mounted on the vehicle 10 travels in the first lane, and is therefore able to capture images of the tunnel wall from its bottom to its top (an image of the right side of the tunnel in Fig. 1(b)). Note that the dark dashed line L in Fig. 1(b) indicates the range captured while traveling in the first lane.

[0019] Thereafter, vehicle 10 photographs the tunnel wall while traveling in the second lane in the opposite direction to the direction of travel in the first lane. By reversing the direction of travel of vehicle 10 and photographing while traveling, it is possible to photograph an image from the bottom to the top of the wall on the opposite side from when traveling in the first lane (an image of the left side of the tunnel in FIG. 1(b)). Note that the light-colored dashed line 1 in FIG. 1(b) indicates part of the range photographed while traveling in the second lane.

[0020] By stitching together the images captured in this way, an image of the entire tunnel wall can be obtained. In the example shown in FIG. 1(b), the range captured when traveling in the first lane and the range captured when traveling in the second lane do not overlap, and the images are captured so that the edges of each captured range are aligned, but this is not a limitation of the embodiment, and for example, the captured ranges may overlap. The captured images can be used, for example, to inspect the tunnel for deterioration.

[0021] Next, a description will be given of the hardware configuration of the information processing system 1. Fig. 2 is a diagram showing an example of the hardware configuration included in the information processing system 1 of this embodiment. The information processing system 1 is configured to include a camera 201, lighting 202, a TOF (Time of Flight) sensor 203, an IMU (Inertial Measurement Unit) 204, a speedometer 205, a distance meter 206, a control board 207, a storage device 208, a display 209, and an input device 210.

[0022] The camera 201 is a device that captures images and constitutes the imaging means of this embodiment. The imaging element of the camera 201 of this embodiment may be configured in the form of a line sensor, for example, and the vehicle 10 equipped with the imaging device 110 can capture an image of, for example, the wall surface of a tunnel by traveling on a road or a railroad. The imaging element of the camera 201 does not have to be a line sensor, and may be configured in the form of an area sensor, for example. The imaging device 110 of this embodiment may include one camera 201 or multiple cameras 201.

[0023] The lighting 202 is a device that emits light in the direction of the subject to be photographed by the camera 201. The lighting 202 of this embodiment constitutes the photographing device 110 and is installed on the roof of the vehicle 10. The brightness of the light emitted by the lighting 202 can be adjusted by, for example, a control board 207.

[0024] The photographing reference position in this embodiment will now be described with reference to FIG. 3. FIG. 3 is a diagram illustrating the photographing reference position and the moving reference position in this embodiment. FIG. 3(a) shows the photographing reference position and the moving reference position in this embodiment, and FIG. 3(b) is a diagram illustrating the moving distance based on the moving reference position. As shown in FIG. 3(a), the vehicle 10 is equipped with a photographing device 110 including a camera 201 and a light 202. The photographing reference position in this embodiment is the position where the imaging surface of the imaging element constituting the camera 201 is installed, as shown in FIG. 3(a). Note that, in the following embodiment, for convenience of explanation, the illumination surface of the light 202 is assumed to coincide with the photographing reference position. Also, the moving reference position in this embodiment is a position corresponding to the optical axis of the photographing lens constituting the camera 201. Note that, as shown in FIG. 3, in the described embodiment, the length direction of the tunnel wall (the moving direction of the vehicle 10) is the X direction, the height direction of the tunnel is the Y direction, and the direction from the vehicle 10 toward the tunnel wall (the direction of the optical axis of the camera 201) is the Z direction.

[0025] Returning to FIG. 2 for the explanation, the TOF sensor 203 is a device that measures the distance from the shooting reference position to the subject, and constitutes the subject distance measuring means of this embodiment. The TOF sensor 203 is composed of a plurality of elements that measure the distance to the object by the reflection of irradiated laser light. Note that a sensor other than the TOF sensor 203, such as a stereo camera or an ultrasonic sensor, may be used to measure the distance from the shooting reference position to the subject in this embodiment.

[0026] The IMU 204 is a device that measures inertia and constitutes the travel distance measurement means of this embodiment. The IMU 204 of this embodiment can measure the acceleration and angular velocity of the vehicle 10. The acceleration and angular velocity measured by the IMU 204 can be used to calculate the trajectory of the vehicle 10.

[0027] The speedometer 205 is a device that measures the speed of the vehicle 10 and constitutes the traveled distance measuring means of this embodiment. The traveled distance meter 206 is a device that measures the distance traveled by the vehicle 10 and constitutes the traveled distance measuring means of this embodiment. The speed and traveled distance measured by the speedometer 205 and the traveled distance meter 206 can be used to calculate the trajectory of the vehicle 10. The measured traveled distance can also be used to issue an instruction signal to the camera 201 to act as a shutter for capturing images at predetermined travel distances. Note that, because the traveled distance can be calculated by integrating the vehicle speed, the speedometer 205 and the traveled distance meter 206 may be configured as a single device such as a wheel encoder or a laser Doppler meter.

[0028] The control board 207 is a device that controls the operation of various hardware components, and constitutes the control means of this embodiment. The control board 207 includes a CPU that executes predetermined programs, a RAM that provides an execution space for the programs, and a ROM that stores various programs. For example, the control board 207 of this embodiment can add, as footer information, information about the situation at the time of shooting (exposure conditions, distance to the subject, distance traveled by the vehicle 10, etc.) to the data of an image captured by the camera 201, and store the data in the storage device 208.

[0029] The storage device 208 is a device that stores various data. Examples of the storage device 208 include a hard disk drive (HDD) and a solid state drive (SSD). The storage device 208 of this embodiment can store data on captured images, the situation at the time of capture, and the like.

[0030] The display 209 is a device that displays various data, image status, etc. to the user, and examples thereof include an LCD (Liquid Crystal Display). The input device 210 is a device that allows the user to perform various operations, and examples thereof include a keyboard and a mouse. The display 209 and the input device 210 may be separate devices, or may be one that has both functions, such as a touch panel display.

[0031] 2 may be included in any of the various devices that make up the information processing system 1. Therefore, for example, the hardware may be included in the form shown in Fig. 4. Fig. 4 is a diagram showing another example of the hardware configuration included in the information processing system 1 of this embodiment.

[0032] 4, the image capturing device 110 can be configured to include a camera 201, lighting 202, a TOF sensor 203, an IMU 204, a speedometer 205, and an odometer 206. The server device 130 can be configured to include a control board 207 and a storage device 208. The terminal device 120 can be configured to include a display 209 and an input device 210.

[0033] 4 is merely an example and does not limit the present embodiment. Therefore, for example, in the case of a standalone information processing system 1, the control board 207 and the storage device 208 may be included in the terminal device 120.

[0034] The hardware configuration included in the information processing system 1 of this embodiment has been described above. Next, the functional means executed by each piece of hardware in this embodiment will be described with reference to Fig. 5. Fig. 5 is a software block diagram included in the information processing system 1 of this embodiment.

[0035] 5, the photographing device 110 includes functional means of a photographing unit 411, an exposure condition control unit 412, a subject distance measurement unit 413, and a movement distance measurement unit 414. The terminal device 120 also includes functional means of a photographed image storage unit 421, an exposure condition storage unit 422, a display unit 423, and an operation reception unit 424. Each functional unit will be described in detail below.

[0036] First, the functional means of the image capturing device 110 will be described. The image capturing unit 411 is a means for capturing images, and constitutes the image capturing means in this embodiment. The image capturing unit 411 in this embodiment can capture images in accordance with the movement of the vehicle 10 on which the image capturing device 110 is mounted. This makes it possible to capture a panoramic image that spreads in the direction of movement of the vehicle 10, and to capture an image of the wall of a tunnel. The captured image data is transmitted to the terminal device 120, for example.

[0037] The exposure condition control unit 412 controls the exposure conditions when the image capturing unit 411 captures an image, and constitutes the control unit in this embodiment. Examples of the exposure conditions controlled by the exposure condition control unit 412 include shutter speed, aperture value, gain value, and illumination intensity, but these are not particularly limited to this embodiment. The exposure condition control unit 412 in this embodiment can control the exposure conditions in an automatic exposure control (AE (Automatic Exposure) control) mode, which feedback-controls the exposure conditions so that the brightness of the captured image is appropriate. The exposure condition control unit 412 can also control the exposure conditions in a fixed exposure control mode, which uses the exposure conditions of an image selected by the user during capture. In the fixed exposure control mode in this embodiment, multiple exposure conditions can be set depending on the horizontal distance to the subject (hereinafter simply referred to as the subject distance) and the travel distance. For example, the exposure condition control unit 412 can control using a first exposure condition when the subject distance is a first distance, and control using a second exposure condition when the subject distance is a second distance. When controlling the exposure conditions in fixed exposure control mode, the exposure condition control unit 412 can control the exposure conditions by reading out the exposure conditions stored in the exposure condition storage unit 422. Note that the subject distance in the described embodiment is the distance from the photographing reference position of the photographing device 110 to the subject, and is the distance in the Z direction from the photographing reference position to the subject in Figure 3(a).

[0038] The subject distance measurement unit 413 is a means for measuring the subject distance based on the data output by the TOF sensor 203, and constitutes the measurement means in this embodiment. In the embodiment being described, the subject distance measurement unit 413 can measure the horizontal distance from the shooting reference position to the wall surface of the tunnel. The value of the subject distance measured by the subject distance measurement unit 413 can be added to the image data as footer information.

[0039] The travel distance measurement unit 414 is a means for measuring the travel distance of the vehicle 10 based on data output by the IMU 204, the speedometer 205, the travel distance meter 206, etc., and constitutes the measurement means in this embodiment. The travel distance measurement unit 414 in this embodiment can measure, for example, the travel distance from the position where image capture started and the travel distance from the entrance of a tunnel. The travel distance value measured by the travel distance measurement unit 414 can be added to the image data as footer information. The entrance of a tunnel may be identified by a user operation or by image analysis. Furthermore, the travel distance in the described embodiment is identified by a travel reference position. That is, the distance from the reference position of the object to the travel reference position (corresponding to the optical axis shown in FIG. 3(a)) is the travel distance. Therefore, the travel distance from the tunnel entrance is the distance in the X direction from the tunnel entrance to the travel reference position, as shown in FIG. 3(b).

[0040] Next, the functional means of the terminal device 120 will be described. The captured image storage unit 421 is a means for storing images captured by the photographing unit 411, and constitutes the storage means in this embodiment. The captured image storage unit 421 in this embodiment can, for example, store image data and data indicating the situation when the image was captured in association with each other. The data stored in the captured image storage unit 421 is received from the photographing device 110. Here, examples of data indicating the situation when the image was captured include exposure conditions such as shutter speed, aperture value, gain value, and lighting intensity, subject distance, and travel distance of the vehicle 10, but this does not particularly limit the embodiment. Furthermore, data indicating the situation when the image was captured can be stored by adding it to the image data as footer information.

[0041] The exposure condition storage unit 422 is a means for storing exposure conditions when the exposure condition control unit 412 controls the exposure conditions in fixed exposure condition mode, and constitutes a storage means in this embodiment. The exposure condition storage unit 422 in this embodiment can store, for example, an object distance and an exposure condition in association with each other. Furthermore, the exposure condition storage unit 422 in this embodiment can store, for example, a travel distance of the vehicle 10 and an exposure condition in association with each other.

[0042] The display unit 423 is a means for controlling the display 209 to display images, processing screens, etc., and constitutes the display means in this embodiment. For example, by viewing the image displayed by the display unit 423, the user can understand the exposure conditions under which the image was taken at an appropriate brightness.

[0043] The operation reception unit 424 is a means for receiving an operation of the input device 210 by the user, and constitutes a reception means in this embodiment.

[0044] The software blocks described above correspond to functional units realized by the CPU executing the programs of the present embodiment and causing each piece of hardware to function. The functional units shown in each embodiment may be realized entirely by software, or some or all of them may be implemented as hardware that provides equivalent functions.

[0045] Furthermore, all of the above-described functional units do not necessarily have to be included in the configuration shown in Fig. 5. For example, in another preferred embodiment, the functional units may be realized by cooperation between the image capture device 110 and the terminal device 120. Also, some of the functional units shown in Fig. 5 may be included in the server device 130, and in another embodiment, for example, the captured image storage unit 421, the exposure condition storage unit 422, etc. may be included in the server device 130.

[0046] Here, a case where shooting is performed in the automatic exposure control mode will be described. FIG. 6 is a diagram showing an example of shooting a tunnel wall in the automatic exposure control mode in this embodiment. FIG. 6(a) shows an example of shooting a tunnel wall while the vehicle 10 is moving, and FIG. 6(b) shows an example of a captured image. Here, the tunnel wall may not have a uniform shape due to the presence of structures such as turnouts and pillars. In the example shown in FIG. 6, the tunnel wall has unevenness, and there are areas A and C where the horizontal distance from the shooting reference position, i.e., the subject distance, is far, and areas B and D where the horizontal distance from the shooting reference position, i.e., the subject distance, is close. Here, the horizontal distance (subject distance) from the shooting reference position of areas A and C is L1, and the horizontal distance (subject distance) from the shooting reference position of areas B and D is L2, and we consider the case where L1>L2.

[0047] As shown in FIG. 6(a), the imaging device 110 mounted on the vehicle 10 can capture images of the tunnel wall in the order of areas A, B, C, and D as the vehicle 10 moves. Here, as the vehicle 10 moves, the distance to the tunnel wall, which is the subject, changes while the image is being captured. In other words, the subject to be captured shifts from area A to area B, or from area B to area C. To simplify the explanation, we will assume that the illumination light irradiation position (the installation position of the light 202) and the illumination light intensity are the same, and the reflectance of the tunnel wall (the reflectance of the material that makes up the tunnel wall) is the same. As shown in FIG. 6, the horizontal distance (subject distance) from the imaging reference position of areas A and C is L1, and the horizontal distance (subject distance) from the imaging reference position of areas B and D is L2, where L1>L2. Because the illumination surface of the illumination light 202 is at the same position as the shooting reference position, the subject brightness in areas A and C, which are farther from the illumination surface, is darker than the subject brightness in areas B and D, which are closer to the illumination surface. In other words, the subject brightness varies between areas A and C and areas B and D. When shooting by controlling the exposure conditions in automatic exposure control mode, feedback control may not be able to keep up with the change in brightness of the subject area to be shot that accompanies this change in subject distance near the boundaries between the areas, making it impossible to shoot under appropriate exposure conditions. In such cases, images shot near the boundaries between the areas may suffer from overexposure or underexposure.

[0048] FIG. 6(b) shows an example of an image of a tunnel wall captured in automatic exposure control mode. For example, when area A is captured, the image initially becomes a relatively dark image a1, but as feedback control in automatic exposure control mode stabilizes, the image becomes an image a2 with appropriate brightness. Then, as the vehicle 10 moves forward and transitions from area A to area B, the subject distance becomes closer. At this time, the feedback control of the exposure conditions that was appropriate for capturing images at relatively long distances cannot keep up with the movement of the vehicle, and the captured image suffers from overexposure, as shown in b1 in FIG. 6(b). Then, as the vehicle 10 moves forward (over time), the image stabilizes with appropriate feedback control, and after image b2, an image with appropriate brightness, as shown in b3, is captured.

[0049] As the vehicle 10 continues to move forward, it transitions from region B to region C. In this case, the subject distance increases while the exposure conditions remain appropriate for capturing a relatively close subject, resulting in crushed blacks in the image, as shown in c1 in FIG. 6(b). As the vehicle 10 continues to move forward (over time), the image stabilizes through appropriate feedback control, and an image with appropriate brightness, such as c3, is captured after image c2.

[0050] Similarly, whiteout occurs in the image when transitioning from region C to region D (d1 in FIG. 6(b)), and then an image with appropriate brightness is captured (d2 in FIG. 6(b)).

[0051] As shown in FIG. 6(b), if the distance from the shooting reference position (the position of the illumination surface of the light 202) to the wall surface inside the tunnel is not constant, the brightness will vary depending on the shooting location, resulting in an image with a gradation. Therefore, when inspecting the tunnel wall surface using images, visibility may decrease and repair areas may be overlooked. Therefore, it is preferable to obtain images captured at a brightness suitable for inspection, that is, images captured at a brightness desired by the user.

[0052] Therefore, in this embodiment, by capturing an image using appropriate exposure conditions set for each region according to the subject distance, it is possible to obtain an image captured with uniform brightness (without gradation) and at a brightness suitable for inspection. Here, appropriate exposure conditions are described. The control parameters for exposure control in this embodiment are defined based on the so-called APEX relational expression. That is, the control parameters are the exposure time (TV), aperture value (AV), and sensitivity (SV) of the image sensor, and the exposure value (EV) during capture can be calculated as EV = TV + AV - SV. By setting the control parameters (TV, AV, SV) to an appropriate exposure value (EV) according to the subject luminance (LV), in other words, so that the subject luminance (LV) is approximately equal to the exposure value (EV), an image with appropriate exposure, i.e., an image with the desired brightness, can be captured. Note that the amount of external light inside a tunnel is very small compared to the amount of illumination light (illumination light amount), and the brightness of the tunnel wall, which is the subject, depends on the amount of illumination light. In other words, inside a tunnel, the subject luminance (LV) depends on the amount of illumination light. As described above, when the subject distance changes, that is, when the distance from the installation position of the lighting 202 (the illumination surface of the lighting 202) to the tunnel wall changes, the subject luminance (LV) also changes. Therefore, by setting the control parameters so that the subject luminance (LV) is approximately equal to the exposure (EV), it is possible to capture an image without gradation. That is, this embodiment is premised on the assumption that the subject brightness (subject luminance (LV)) will be captured in an environment where the brightness of the subject (subject luminance (LV)) depends on the amount of illumination light (illumination light amount), such as when capturing an image of the wall surface of a tunnel or other mine shaft. In other words, since the brightness of the subject depends on the amount of illumination light (illumination light amount), reproducibility is achieved.

[0053] Next, the processing executed by the information processing system 1 of this embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the processing executed by the information processing system that captures an image of a tunnel wall in this embodiment. The flowchart in Fig. 7 will be described as an example of capturing an image of the tunnel wall as shown in Fig. 6. The information processing system 1 starts the processing from step S1000.

[0054] First, in step S1001, the photographing unit 411 photographs the wall surface while the vehicle 10 is traveling through the tunnel. At this time, the exposure condition control unit 412 controls the exposure conditions in automatic exposure control mode. In the following description, photographing the tunnel wall surface in step S1001 may be referred to as "pre-photography." In this embodiment, photographing can be performed while the vehicle 10 is traveling from the tunnel entrance to the exit. Alternatively, photographing may be started just before the tunnel entrance, photographing may continue after the vehicle 10 has passed the tunnel exit, and an image of the inside of the tunnel may be extracted after photographing has ended. Here, the tunnel entrance and exit may be identified by user designation or by image analysis.

[0055] Here, the exposure conditions during pre-photography will be described with reference to Fig. 8. Fig. 8 is a diagram illustrating examples of capturing images under various exposure conditions in this embodiment. Fig. 8(a) illustrates an example of exposure conditions during pre-photography, and Fig. 8(b) illustrates an example of capturing images in fixed exposure control mode.

[0056] As shown in Figure 8(a), when shooting in automatic exposure control mode, the exposure conditions are feedback-controlled, so the conditions vary depending on the shooting position. For example, as shown in Figure 8(a), a certain part of area A is shot under condition α1, and another part of area A is shot under condition α2. Also, a certain part of area B is shot under condition β1, another part of area B is shot under condition β2, and yet another part of area B is shot under condition β3.

[0057] In pre-photography in automatic exposure control mode, as shown in FIG. 8(a), even when photographing the wall surface of the same area A, the exposure varies because the photographs are taken under different conditions, resulting in different brightnesses in images of the same area. The same thing happens for areas B to D, and the brightnesses of images of the same area also vary. Therefore, when photographing a tunnel with walls like areas A to D in FIG. 6 in the pre-photography in step S1001, an image with uneven brightness, as shown in FIG. 6(b), is captured. Therefore, by associating the image with the exposure conditions at the time of photographing as follows, it is possible to capture an image with appropriate brightness.

[0058] Footer information is added to the image captured in step S1001, and the image is stored in the captured image storage unit 421. The structure of the data stored in this embodiment will now be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of the data structure in this embodiment.

[0059] As shown in Fig. 9, the data structure in this embodiment is composed of a portion for storing image data and a portion for storing footer information associated with the image. One data structure is created by one photographing operation by the photographing device 110. Therefore, when images are photographed in accordance with the movement of the vehicle 10, a data structure is created for each photographing position in the direction of movement of the vehicle 10, as shown in Fig. 9.

[0060] 9, footer information is added to the image data. The footer information includes data such as the conditions under which the image was captured. The footer information may include, for example, shutter speed, aperture value, illumination intensity, gain value, subject distance, speed of the vehicle 10, and travel distance from a reference position (for example, the position where capture started or the entrance to a tunnel), but this does not particularly limit the embodiment.

[0061] Returning to Fig. 7 for the explanation, when an image is captured in step S1001, footer information is added to the image data, and data having the data structure shown in Fig. 9 is created. The created data is stored in the captured image storage unit 421.

[0062] When the shooting of the tunnel wall surface in the automatic exposure control mode in step S1001 is completed, the process proceeds to step S1002. In step S1002, the display unit 423 displays an exposure condition selection screen based on the data stored in the captured image storage unit 421. Thereafter, in step S1003, the operation receiving unit 424 receives an operation by the user to select an exposure condition.

[0063] Selection of exposure conditions in this embodiment will now be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of a screen for selecting exposure conditions in this embodiment.

[0064] In step S1002, a screen for selecting exposure conditions shown in FIG. 10 (hereinafter simply referred to as the "selection screen") is displayed. The selection screen includes and displays the captured image. The selection screen may also include a display of the distance from the camera position to the subject position (i.e., subject distance), a display showing the outline of the wall surface, and the like. Note that the selection screen shown in FIG. 10 may also display a numerical value indicating the average value of the 8-bit gradation of each part of the image superimposed on the captured image.

[0065] As shown in FIG. 10, a marker is superimposed on the image of the selection screen to allow the user to select a location that has been photographed with appropriate brightness. The user can move the marker to any location on the image by dragging and dropping with a mouse or swiping on a touch panel display. Therefore, the user can select the exposure conditions for a location that they determine has been photographed with appropriate brightness by moving the marker to that location. In the following description, the exposure conditions for the selected location may be referred to as the "selected exposure conditions."

[0066] In the embodiment to be described, a plurality of exposure conditions can be selected, for example, the exposure condition (e.g., α2) under which the image of a2 was captured can be selected for area A, and the exposure condition (e.g., β3) under which the image of b3 was captured can be selected for area B. Furthermore, the same exposure condition can be used for areas with the same subject distance, for example, if the exposure condition α2 is selected for area A, α2 can also be used for area C, which has the same subject distance as area A.

[0067] After the operation receiving unit 424 receives an operation to select an exposure condition in step S1003, the exposure condition storage unit 422 reads out footer information for the selected portion in step S1004. Next, in step S1005, the exposure condition storage unit 422 saves the exposure condition included in the read footer information. In a preferred embodiment, the exposure condition storage unit 422 can store the exposure condition in association with the subject distance. This allows the same exposure condition to be used when shooting in fixed exposure control mode, when shooting an area at the same subject distance. The exposure conditions stored in the exposure condition storage unit 422 may be three parameters: exposure time (TV), aperture value (AV), and sensitivity (SV) of the image sensor, or may be the exposure value (EV) at the time of shooting calculated from these parameters using the APEX relational expression.

[0068] Thereafter, in step S1006, the photographing unit 411 photographs the wall surface while the vehicle 10 is traveling through the tunnel. At this time, the exposure condition control unit 412 appropriately reads out the exposure conditions stored in the exposure condition storage unit 422 in step S1005 and controls the exposure conditions in fixed exposure control mode. In the following description, photographing the tunnel wall surface in fixed exposure control mode in step S1006 may be referred to as "main photographing." Here, consider a case where it is selected on the selection screen in FIG. 10 that the area where the subject distance is L1 uses exposure condition α2 under which the image a2 was photographed, and the area where the subject distance is L1 uses exposure condition β3 under which the image b3 was photographed.

[0069] In the described embodiment, in step S1006, the vehicle 10 moves while the subject distance measurement unit 413 measures the subject distance. At this time, when the vehicle 10 is moving from a position where area A is to be photographed, the subject distance measurement unit 413 measures L1 as the subject distance and outputs it to the exposure condition control unit 412. The exposure condition control unit 412 compares the subject distance associated with each exposure condition stored in the exposure condition storage unit 422 with the measured subject distance and reads out an exposure condition for a subject distance close to the measured subject distance. In the described embodiment, the measured subject distance is L1, and the exposure condition storage unit 422 stores an exposure condition α2 for the subject distance L1 and an exposure condition β3 for the subject distance L2. Therefore, the exposure condition control unit 412 reads out the exposure condition α2 for the subject distance L1 from the exposure condition storage unit 422. The photographing unit 411 then photographs an image of the wall surface under the exposure condition α2. In this way, while the vehicle 10 is moving to a position where it photographs the area A, the exposure condition is fixed at α2, and images can be photographed.

[0070] Thereafter, when the vehicle 10 travels and reaches a position for capturing an image of area B, the object distance measurement unit 413 measures L2 as the object distance and outputs the measured object distance to the exposure condition control unit 412. The exposure condition control unit 412 compares the measured object distance with the object distances associated with the exposure conditions stored in the exposure condition storage unit 422, and reads out the exposure condition for the object distance closest to the measured object distance. In the described embodiment, the measured object distance is L2, and the exposure condition storage unit 422 stores an exposure condition α2 for an object distance of L1 and an exposure condition β3 for an object distance of L2. Therefore, the exposure condition control unit 412 reads out the exposure condition β3 for an object distance of L2 from the exposure condition storage unit 422. The image capture unit 411 then captures an image of the wall surface under the exposure condition β3. In this way, while the vehicle 10 is moving from a position for capturing an image of area B, the exposure condition is fixed at β3, and images can be captured.

[0071] Subsequently, when the vehicle 10 travels further and reaches a position for capturing an image of area C, the object distance measurement unit 413 measures L1 as the object distance and outputs the measured object distance to the exposure condition control unit 412. The exposure condition control unit 412 compares the measured object distance with the object distances associated with the exposure conditions stored in the exposure condition storage unit 422, and reads out the exposure condition for the object distance closest to the measured object distance. In the described embodiment, the measured object distance is L1, and the exposure condition storage unit 422 stores an exposure condition α2 for the object distance L1 and an exposure condition β3 for the object distance L2. Therefore, the exposure condition control unit 412 reads out the exposure condition α2 for the object distance L1 from the exposure condition storage unit 422. The image capture unit 411 then captures an image of the wall surface under the exposure condition α2. In this manner, while the vehicle 10 is moving to the position for capturing an image of area C, the exposure condition is fixed at α2, and images can be captured.

[0072] After that, when the vehicle 10 further travels and reaches a position for capturing an image of area D, the object distance measurement unit 413 measures L2 as the object distance and outputs it to the exposure condition control unit 412. The exposure condition control unit 412 compares the object distance associated with each exposure condition stored in the exposure condition storage unit 422 with the measured object distance and reads out the exposure condition for the object distance closest to the measured object distance. In the described embodiment, the measured object distance is L2, and the exposure condition storage unit 422 stores an exposure condition α2 for an object distance of L1 and an exposure condition β3 for an object distance of L2. Therefore, the exposure condition control unit 412 reads out the exposure condition β3 for an object distance of L2 from the exposure condition storage unit 422. The image capture unit 411 then captures an image of the wall surface under the exposure condition β3. In this way, while the vehicle 10 is moving from a position for capturing an image of area D, the exposure condition is fixed at β3, and images can be captured.

[0073] In this way, the exposure condition control unit 412 controls the exposure conditions, so that areas with the same subject distance can be photographed under the same exposure conditions, as shown in Fig. 8(b). Therefore, an image can be photographed under exposure conditions that provide appropriate brightness selected by the user, and the occurrence of overexposure and underexposure can be suppressed.

[0074] The image captured in step S1006 is stored in the captured image storage unit 421. At this time, the image can be stored in the captured image storage unit 421 in a format that distinguishes it from the image captured and stored during pre-photography. Thereafter, in step S1007, the information processing system 1 ends the processing.

[0075] By performing the processing shown in FIG. 7, an image can be obtained that has been photographed to have the brightness desired by the user.

[0076] The images captured in step S1001 or S1006 and the exposure conditions saved in step S1005 may be saved on a so-called cloud, such as the server device 130, separately from the series of processes shown in FIG. 7. Storing the information on the cloud in this manner allows the saved information to be used for future inspections, enabling efficient operation. In particular, the brightness of the subject when photographing a tunnel wall depends on the amount of illumination light and is therefore reproducible. Saving the exposure conditions used during pre-photography allows them to be used for subsequent inspections. In other words, if the exposure conditions used during pre-photography are retrieved and used for actual photography during subsequent inspections, images with the desired brightness can be captured, thereby improving the efficiency of inspections.

[0077] Here, an image captured by the processing of this embodiment will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of an image captured in fixed exposure control mode in this embodiment. For comparison, Fig. 11 also includes an image captured in automatic exposure control mode (an image captured during pre-photography in step S1001). The image captured during pre-photography shown in Fig. 11 is the same as the image described in Fig. 6(b).

[0078] As shown in Fig. 11, the image captured in the fixed exposure control mode of this embodiment (the image captured in step S1006) does not have gradations within the area or blown-out highlights or crushed shadows at the area boundaries. That is, an image captured with a uniform brightness desired by the user is obtained. Therefore, unlike the image captured during pre-capture, the image is captured with a brightness suitable for inspection.

[0079] In the embodiment described above, the exposure conditions selected by the user are stored in association with the subject distance, but this is not a limitation of the embodiment. Therefore, for example, in another embodiment, the exposure conditions selected by the user may be stored in association with the travel distance of the vehicle 10. Here, an embodiment in which the exposure conditions and the travel distance are stored in association with each other will be described with reference to FIG. 12. FIG. 12 is a diagram showing an example of a screen for selecting exposure conditions in another embodiment.

[0080] The selection screen shown in Fig. 12 displays the captured image as well as the distance traveled by vehicle 10 and the general shape of the subject. For ease of explanation, it is assumed that the image capture begins at the entrance to a tunnel, and as shown in Fig. 12, it is assumed that area A extends from the capture start point (tunnel entrance) to a position 10 m away, area B extends from 10 m to 20 m away, area C extends from 20 m to 30 m away, and area D extends from 30 m to 40 m away. Note that, among the elements shown in Fig. 12, descriptions of elements common to those in Fig. 10 will be omitted as appropriate.

[0081] Here, it is assumed that the user selects a2 as a location in area A that has appropriate brightness, b3 as a location in area B that has appropriate brightness, c3 as a location in area C that has appropriate brightness, and d2 as a location in area D that has appropriate brightness (step S1003). In this case, the exposure condition storage unit 422 stores the exposure conditions of the selected locations in association with the distance in the movement direction of the area that includes the selected locations (steps S1004 and S1005).

[0082] Therefore, the exposure condition storage unit 422 stores the exposure conditions associated with the travel distances, such that while the vehicle 10 has traveled 10 m since starting image capture at the entrance of the tunnel, the exposure conditions used when a2 was captured (for example, condition α2 in FIG. 8(a)) are used; while the vehicle 10 is traveling between 10 m and 20 m from the entrance of the tunnel, the exposure conditions used when b3 was captured (for example, condition β3 in FIG. 8(a)) are used; while the vehicle 10 is traveling between 20 m and 30 m from the entrance of the tunnel, the exposure conditions used when c3 was captured (for example, condition γ3 in FIG. 8(a)) are used; and while the vehicle 10 is traveling between 30 m and 40 m from the entrance of the tunnel, the exposure conditions used when d2 was captured (for example, condition δ2 in FIG. 8(a)) are used. Thereafter, while the vehicle 10 is traveling through the tunnel, the image capture unit 411 captures images of the wall surface (step S1006). At this time, the exposure condition control unit 412 appropriately reads out the exposure conditions stored in the exposure condition storage unit 422 and controls the exposure conditions in the fixed exposure control mode.

[0083] In another embodiment to be described, during image capture in step S1006, travel distance measurement unit 414 measures the distance traveled by vehicle 10 and outputs the measured distance to exposure condition control unit 412. Exposure condition control unit 412 reads out exposure conditions corresponding to the distance traveled by vehicle 10 from the start of image capture from exposure condition storage unit 422, and controls the exposure conditions. Therefore, an image is obtained in which area A is captured under exposure condition α2 selected by the user, area B is captured under exposure condition β3 selected by the user, area C is captured under exposure condition γ3 selected by the user, and area D is captured under exposure condition δ2 selected by the user.

[0084] In this way, by capturing an image under exposure conditions according to the travel distance, an image captured with the brightness desired by the user can be obtained, as shown in FIG. 11. In particular, compared to capturing an image under exposure conditions according to the subject distance, this method is not affected by noise or delays in the TOF sensor 203, and allows for highly accurate exposure control. Furthermore, when capturing an image of a tunnel wall, if the illumination light irradiation position (the installation position of the light 202) and the illumination light intensity are the same, even if the subject distance does not change while the vehicle 10 is traveling, for example, the brightness of the subject area may change due to the material constituting the tunnel wall, which is the subject, changing depending on the location, i.e., the reflectance of the tunnel wall changing. Even in such a case, according to the other embodiment described above, it is possible to capture an image with the desired brightness, since it is possible to capture an image under exposure conditions according to the travel distance.

[0085] In the embodiment described above, if the exposure condition is directly switched in the fixed exposure control mode, whiteout or blackout may occur in a small area near the exposure condition switching position due to errors in the subject distance measurement unit 413 or the movement distance measurement unit 414. Therefore, instead of directly switching the exposure condition from α2 to β3, for example, the parameters of the exposure condition may be changed in stages or with a gradient.

[0086] For example, if the gain in region A is set to 12 dB and the gain in region B is set to 0 dB, the gain may be controlled to decrease from just before the boundary between region A and region B (in region A) until it reaches 0 dB at a predetermined distance within region B. More specifically, the gain may be set from 12 dB to 9 dB just before the boundary, and then gradually decreased to 6 dB, 3 dB, and 0 dB.

[0087] In this way, by changing the exposure conditions stepwise or with a gradient, it is possible to suppress the occurrence of blown-out highlights and crushed shadows in minute areas near the boundaries.

[0088] According to the embodiments of the present invention described above, it is possible to provide an information processing system, method, program, information processing device, and photographing device that appropriately control exposure conditions during photographing so that the photographed image has the desired brightness.

[0089] Each function of the above-described embodiments of the present invention can be realized by a device-executable program written in C, C++, C#, Java (registered trademark), etc., and the program of this embodiment can be stored and distributed on a device-readable recording medium such as a hard disk drive, CD-ROM, MO, DVD, flexible disk, EEPROM (registered trademark), EPROM, etc., and can also be transmitted over a network in a format that can be used by other devices.

[0090] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by electronic circuits, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a conventional circuit module designed to perform each of the above-described functions.

[0091] Although the present invention has been described above with reference to embodiments, the present invention is not limited to the above-described embodiments, and any embodiment that can be conceived by a person skilled in the art is included in the scope of the present invention as long as it exhibits the functions and effects of the present invention. [Explanation of symbols]

[0092] 1...information processing system, 10...vehicle, 110...imaging device, 120...terminal device, 130...server device, 140...network, 201...camera, 202...lighting, 203...TOF sensor, 204...IMU, 205...speedometer, 206...travel distance meter, 207...control board, 208...storage device, 209...display, 210...input device, 411...imaging unit, 412...exposure condition control unit, 413...subject distance measurement unit, 414...travel distance measurement unit, 421...captured image storage unit, 422...exposure condition storage unit, 423...display unit, 424...operation reception unit [Prior art documents] [Patent documents]

[0093] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-239479

Claims

1. An imaging means for capturing images while moving; a control means for controlling the exposure conditions of the photographing means; a storage means for storing a first image captured by the image capturing means under exposure conditions under first control in which automatic exposure control is performed by the control means, in association with the exposure conditions at the time of image capture; a receiving means for receiving an operation to select a part of the first image on a screen displayed including the first image stored in the storage means; Including, the storage means stores, as selected exposure conditions, exposure conditions corresponding to the location selected by the operation accepted by the acceptance means; the control means acquires from the storage means the selected exposure conditions associated with the location accepted by the accepting means; the photographing means photographs a second image under exposure conditions by second control based on the selected exposure conditions acquired by the control means; Information processing system.

2. a first measuring means for measuring a distance from the image capturing means to a subject while capturing an image under the first control or the second control; the storage means stores the selected exposure conditions and the distances measured by the first measurement means in association with each other; the photographing means photographs the second image under the selected exposure condition corresponding to the distance measured while the first measurement means photographs under the exposure condition according to the second control; The information processing system according to claim 1 .

3. the value of the distance to the subject measured by the first measuring means is displayed on the screen in association with the first image; The information processing system according to claim 2 .

4. further comprising a second measuring means for measuring a distance traveled by the photographing means when photographing an image under the first control or the second control, the storage means stores the selected exposure conditions and the distances measured by the second measurement means in association with each other; the photographing means photographs the second image under the selected exposure condition corresponding to the distance measured while the second measurement means photographs under the exposure condition according to the second control; The information processing system according to claim 1 .

5. The value of the distance traveled measured by the second measuring means is displayed on the screen in association with the first image. The information processing system according to claim 4 .

6. the accepting means accepts a part of the first image as a first region on a screen displayed including the first image, and accepts another part different from the part of the first image as a second region; the control means changes the exposure conditions from the first selected exposure conditions associated with the first region accepted by the accepting means to the second selected exposure conditions associated with the second region accepted by the accepting means in a stepwise or gradual manner when photographing the vicinity of a boundary where the exposure conditions change from the first selected exposure conditions to the second selected exposure conditions. The information processing system according to claim 1 .

7. A method executed by an information processing system including an imaging means, a first photographing step of controlling the photographing means under exposure conditions according to first control that performs automatic exposure control while moving, and photographing a first image; a first storage step of storing the first image captured in the first photographing step in a storage means in association with an exposure condition at the time of photographing; a receiving step of receiving an operation to select a part of the first image on a screen displayed including the first image stored in the first storing step; a second storage step of storing, in the storage means, exposure conditions corresponding to the location selected by the operation accepted in the accepting step as selected exposure conditions; an acquisition step of acquiring the selected exposure conditions associated with the location accepted in the acceptance step from the storage means; a second photographing step of photographing a second image under exposure conditions by second control based on the selected exposure conditions acquired in the acquiring step; A method comprising:

8. A program executed by an information processing system including a photographing means, the information processing system a first photographing step of controlling the photographing means under exposure conditions according to first control that performs automatic exposure control while moving, and photographing a first image; a first storage step of storing the first image captured in the first photographing step in a storage means in association with an exposure condition at the time of photographing; a receiving step of receiving an operation to select a part of the first image on a screen displayed including the first image stored in the first storing step; a second storage step of storing, in the storage means, exposure conditions corresponding to the location selected by the operation accepted in the accepting step as selected exposure conditions; an acquisition step of acquiring the selected exposure conditions associated with the location accepted in the acceptance step from the storage means; a second photographing step of photographing an image under exposure conditions by second control based on the selected exposure conditions acquired in the acquisition step; A program that executes.

9. An information processing device, a display means for displaying a screen including an image; a receiving means for receiving an operation to select a part of the image included on the screen; Including, The image is an image taken while moving, reading out, from a storage means that stores the image and the exposure conditions for the image, exposure conditions corresponding to the portion selected by the operation accepted by the accepting means, and storing the selected exposure conditions in the storage means; Information processing device.

10. An imaging means for capturing images while moving; a control means for controlling the exposure conditions of the photographing means; Including, The control means acquires the selected exposure conditions from an information processing device according to claim 9 and controls the exposure conditions of the photographing means. Filming equipment.

Images