Imaging device, program and control method

Abstract

To determine the priority of imaging conditions and generate appropriate images.SOLUTION: An imaging device includes an imaging unit for capturing an image under a set imaging condition, a recognition result acquisition unit for acquiring a recognition result obtained by performing object recognition processing on the captured image using an image recognition device, and an imaging condition change unit by which when it is determined that the captured image has a saturated area, a priority between a first imaging condition for the saturated area and a second imaging condition for a non-saturated area other than the saturated area is determined based on the state of a recognized object, imaging parameters are determined such that the proportion of images captured for performing recognition processing under an imaging condition having a higher priority is increased, and the imaging condition of the imaging unit is changed so as to achieve the proportion of the determined imaging parameters.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to an imaging device, a program, and a control method.

Background Art

[0002] Patent Document 1 discloses an imaging device capable of selectively executing an imaging operation in a normal imaging mode that outputs one exposure image signal per unit period and an imaging operation in a composite imaging mode that outputs a long-time exposure image signal with a relatively long exposure time and a short-time exposure image signal with a relatively short exposure time per unit period. Optimal operating points for each of the normal shooting mode and the composite shooting mode are stored in advance, and by automatically switching the operating point according to each operating mode, the operating point is expanded to the saturation region in the normal shooting mode to improve the S / N, and in the composite shooting mode, the operating region is set lower than the saturation region to prevent saturation unevenness.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Patent Document 1, in the composite imaging mode, long-time exposure and short-time exposure are performed in one field period, and the long-time exposure image signal and the short-time exposure image signal are output in a time-division manner. Then, by synthesizing these image signals, imaging image data for one field is generated. Further, Patent Document 1 also describes a process of performing long-time exposure in a certain field period and short-time exposure in the next field period and synthesizing each exposure image signal.

[0005] In Patent Document 1, two shooting conditions of long-time exposure and short-time exposure are alternately executed to generate shooting image data from two exposure image signals, and no consideration is given to which of the two shooting conditions should be preferentially executed.

[0006] In view of the above problems, an object of the present disclosure is to provide an imaging device, a program, and a control method capable of determining the priority of shooting conditions and generating an appropriate image.

Means for Solving the Problems

[0007] The imaging device according to the present disclosure includes an imaging unit that captures a captured image under set shooting conditions, a recognition result acquisition unit that acquires a recognition result obtained by performing recognition processing of an object on the captured image by an image recognition device, a determination unit that determines the presence or absence of a saturated region in the captured image, and when it is determined that there is a saturated region in the captured image, based on the state of the recognized object, the priority of a first shooting condition for the saturated region and a second shooting condition for a non-saturated region other than the saturated region is determined, and shooting parameters are determined so that the ratio of shooting a captured image for performing recognition processing under the shooting condition with a higher priority increases, and a shooting condition change unit that changes the shooting conditions of the imaging unit so as to be the ratio of the determined shooting parameters.

[0008] The program according to the present disclosure causes a computer to execute a process of capturing a captured image under set shooting conditions, a process of acquiring a recognition result obtained by performing recognition processing of an object on the captured image by an image recognition device, a process of determining the presence or absence of a saturated region in the captured image, and when it is determined that there is a saturated region in the captured image, based on the state of the recognized object, the priority of a first shooting condition for the saturated region and a second shooting condition for a non-saturated region other than the saturated region is determined, shooting parameters are determined so that the ratio of shooting a captured image for performing recognition processing under the shooting condition with a higher priority increases, and a process of changing the shooting conditions of the imaging unit so as to be the ratio of the determined shooting parameters.

[0009] The control method according to the present disclosure includes a process in which a computer captures a captured image under set imaging conditions, a process of obtaining a recognition result obtained by performing recognition processing of an object on the captured image by an image recognition device, a process of determining the presence or absence of a saturated region in the captured image, and when it is determined that the saturated region exists in the captured image, based on the state of the recognized object, the priority of a first imaging condition for the saturated region and a second imaging condition for a non-saturated region other than the saturated region is determined, the imaging parameters are determined so that the ratio of capturing a captured image for performing recognition processing under the imaging condition with the higher priority increases, and the imaging conditions of the imaging unit are changed to the ratio of the determined imaging parameters.

Advantages of the Invention

[0010] According to the present disclosure, it is possible to determine the priority of imaging conditions and generate an appropriate image.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10A

Figure 10B

Figure 11

Figure 12A

Figure 12B

Figure 13

Figure 14

Figure 15

BEST MODE FOR CARRYING OUT THE INVENTION

[0012] Hereinafter, specific embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same reference numerals are assigned to the same elements, and redundant descriptions are omitted as necessary for clarity of explanation.

[0013] An embodiment relates to an imaging device that captures a recognition image to be input to a recognition device that performs recognition processing on an object in an image recognition system, as an example. The imaging device according to the embodiment is, for example, an in-vehicle camera mounted on a vehicle and performing recognition processing on other vehicles, pedestrians, etc. in front of the vehicle.

[0014] In an image recognition system, the image quality of the image input to the recognition device that performs recognition processing greatly affects the discrimination judgment. In an image in which an area with an appropriate luminance level and an area with a low luminance level or a high luminance level are mixed, the recognition accuracy by the image recognition device may decrease. The imaging device according to the embodiment determines the priority of the imaging conditions of the imaging device in consideration of the authentication accuracy by the recognition device, and generates an image with high recognition accuracy.

[0015] FIG. 1 is a block diagram showing the overall configuration of an image recognition system 100 including an imaging device 10 according to an embodiment. As shown in FIG. 1, the image recognition system 100 includes an imaging device 10, an image recognition device 20, and a display device 30. The image recognition system 100 has a function of displaying an image captured by the imaging device 10 for user visual recognition. Further, the provided functions of the image recognition system 100 also include a function of recognizing an object included in the captured image and displaying it in a visually recognizable manner to the user. Note that when the image recognition system 100 does not require the function of displaying the image captured by the imaging device 10 for user visual recognition, the display device 30 may not be included. <Imaging device 10> The imaging device 10 generates display image data and recognition image data from a captured signal obtained by capturing a landscape including a plurality of objects under set imaging conditions. FIG. 2 is a block diagram showing a configuration example of the imaging device 10 in FIG. 1. As shown in FIG. 2, the imaging device 10 includes a camera unit 11, a signal processing unit 12, an output unit 13, a processing unit 14, a control unit 15, and a storage unit 1.

[0016] The camera unit 11 includes, as an imaging device, for example, a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) sensor, and the like. Note that the camera unit 11 includes a lens group (imaging optical system) including a zoom lens and a focus lens, an iris diaphragm, a mechanical shutter, and the like. The camera unit 11 adjusts the level of a shooting signal based on an image formed by the imaging optical system according to a predetermined amplification gain, performs A / D conversion, and sequentially outputs the shooting signal to the signal processing unit 12 as a shooting signal. The camera unit 11 may be referred to as an imaging unit.

[0017] The camera unit 11 adjusts the brightness of a captured image by adjusting the amount of light incident on the imaging device using the mechanical shutter and the iris diaphragm of the imaging device 10. In the following description, it is assumed that the imaging device 10 controls the mechanical shutter according to the determined shooting conditions to adjust the exposure time per frame (charge accumulation time of the imaging device). Here, the shooting conditions are not limited to those that adjust the exposure time per frame, and may be, for example, those that adjust the opening amount of an aperture blade (not shown) provided in the imaging device 10.

[0018] When the captured image includes a saturation region, the imaging device 10 can perform shooting by switching a plurality of shooting conditions according to the state of an object recognized in the captured image. The shooting conditions include saturation improvement exposure (corresponding to the first shooting condition in the claims) and normal exposure (corresponding to the second shooting condition in the claims).

[0019] For saturation improvement exposure, depending on the luminance of each pixel in the captured image, it includes black crush improvement exposure where the exposure time is made longer than the normal exposure (the shutter speed is made slower), and white bloom improvement exposure where the exposure time is made shorter than the normal exposure (the shutter speed is made faster). The black crush improvement exposure can brighten the black crush areas in the captured image by long-time exposure and make the image have an appropriate brightness for the recognition process by the image recognition device 20. The white bloom improvement exposure can darken the white bloom areas in the captured image by short-time exposure and make it have an appropriate brightness for the recognition process by the image recognition device 20.

[0020] In the embodiment, the imaging device 10 generates a normal exposure shooting signal which is shooting data captured with the normal exposure time, a long-time exposure shooting signal which is shooting data captured under the long-time exposure condition among the saturation improvement exposures, and a short-time exposure shooting signal which is shooting data captured under the short-time exposure condition among the saturation improvement exposures, and outputs them to the signal processing unit 12. In FIG. 2, these three shooting signals are collectively referred to as the "shooting signal". Note that the amplification gain of the camera unit 11 may be fixed or may be changed according to the luminance of each pixel in the captured image.

[0021] The signal processing unit 12 acquires the shooting signals continuously captured by the camera unit 11 and performs various image processes on the image data in frame units (hereinafter referred to as image frames). The signal processing unit 12 can, for example, process the shooting signals output from the camera unit 11 in a time-division manner in the order in which they are input. The signal processing unit 12 can generate an image frame conforming to a predetermined display signal format. For example, when generating image data in the full high-definition format, the signal processing unit 12 performs white balance processing, gamma processing, etc. on each image frame, and then executes enlargement / reduction processing to an image size of 1920x1080 pixels. The signal processing unit 12 delivers the image data subjected to each process to the output unit 13.

[0022] The output unit 13 can obtain the recognition result by the image recognition device 20 described later. The output unit 13 obtains the recognition result via, for example, the processing unit 14. The output unit 13 can perform a processing operation on the image data using the recognition result and generate display image data. The output unit 13 outputs the generated display image data to the display device 30. The display device 30 is a liquid crystal display device or the like having an image display function. The display device 30 can display a captured image with high visibility and a display image including the recognition result based on the display image data. As will be described later, when the captured image includes a white-out area or a blacked-out area, the output unit 13 can also output image data for simultaneously displaying an area captured under normal exposure conditions and an area captured under short-time exposure or long-time exposure. When the image recognition system 100 does not include the display device 30, the imaging device 10 may not include the output unit 13.

[0023] Note that the output unit 13 may generate information to be displayed superimposed on the captured image using the recognition result. The display device 30 can, for example, superimpose and display a frame surrounding the area of the recognized object, character information corresponding to the type of the recognition target, recognition determination information such as the recognition rate, etc. on the captured image. Further, the output unit 13 may create auxiliary display information such as menu setting display used when the user makes various inputs using an input device (not shown). The output unit 13 can superimpose the menu setting display on the captured image. The image data processed by the signal processing unit 12 is supplied to the processing unit 14 in addition to being used for display on the display device 30. The image data processed by the signal processing unit 12 is input to the processing unit 14. The processing unit 14 generates recognition image data suitable for the recognition processing of the object from the input image data. The recognition image data is supplied to the image recognition device 20 and used for the recognition of the object. Note that the shooting signal generated by the camera unit 11 may be directly supplied to the processing unit 14 without passing through the signal processing unit 12. Also, a signal being processed by the signal processing unit 12 may be supplied to the processing unit 14.

[0024] The processing unit 14 can obtain the recognition results by the image recognition device 20 described later. When there is a saturation area in the captured image, the processing unit 14 determines the priority of the shooting conditions based on the state of the recognized object, and determines the shooting parameter conditions so that the ratio of shooting the captured image for performing the recognition process with the shooting conditions having a higher priority increases. The processing unit 14 will be described in detail later. The control unit 15 changes the shooting conditions in the camera unit 11 based on the change instruction signal instructed to be at the ratio determined by the processing unit 14. Specifically, the control unit 15 controls the speed of the mechanical shutter of the camera unit 11 to switch between normal exposure and saturation improvement exposure.

[0025] In addition to changing the ratio of the shooting conditions described above, the processing unit 14 may perform image quality adjustment processing based on the image quality adjustment parameters on the recognition image data. For example, an image recognition support device (not shown) can generate image quality adjustment parameters that affect the recognition rate of the object and are used when the processing unit 14 performs image quality adjustment processing. Such image quality adjustment parameters include a luminance gain for controlling the luminance (brightness) of the captured image, tone mapping characteristics, an aperture gain for edge enhancement, and the like.

[0026] The image quality adjustment support device can generate image quality adjustment parameters used for adjusting the image quality to improve the recognition accuracy of the recognition target area in the captured image in subsequent recognition according to the recognition result. Note that the generation of such image quality adjustment parameters may be performed multiple times. For example, within a certain time period, the image quality adjustment may be repeatedly executed until the "recognition frequency", which is the number of successful image recognitions in the total number of image recognitions, is equal to or greater than a predetermined value. <Image Recognition Device 20> The image recognition device 20 performs image recognition processing on the recognition image supplied from the processing unit 14 and feeds back the recognition result to the imaging device 10. The recognition images are continuously input to the image recognition device 20, and the recognition processing is continuously executed as needed.

[0027] The recognition result includes the presence or absence of the object, the type of the object, the area or position of the object, and the recognition rate. The presence or absence of the object is information indicating whether the object has been recognized, i.e., identified, through image recognition processing on the recognition image. The type of the object is information indicating the type of the recognized object. The area of the object is a set of coordinates defining the range of the area including the recognized object within the recognition image. The area of the object is, for example, a range specified by pixel values in the XY coordinate system. Note that the position of the object is, for example, a representative point such as the center coordinates of the object recognized within the recognition image.

[0028] The recognition rate is an example of the degree of certainty of recognition by image recognition. That is, the recognition rate is numerical information indicating the recognition accuracy of the presence or absence, type, and area of the object recognized through image recognition processing on the recognition image. The recognition rate may be indicated, for example, from 0 to 100%. Also, for calculating the recognition rate, for example, a threshold value indicating the degree of similarity to the object, the number of passing stages of the discriminator, etc. may be used. When a plurality of recognized objects are recognized, the recognition result may generate a set of type, area, and recognition rate for each object.

[0029] The image recognition device 20 is hardware or software capable of executing known image recognition processing, or a combination thereof. For example, the image recognition device 20 is realized by executing a known image recognition processing program on a computer. Note that the image recognition device 20 may be redundant among a plurality of computers, and each functional block may be realized by a plurality of computers. Also, the image recognition device 20 may be realized in a form in which each of a client-server system, a cloud computing system, etc. is connected via a communication network. Also, the function of the image recognition device 20 may be provided in the form of SaaS (Software as a Service). Alternatively, the image recognition device 20 may be incorporated into a part of the imaging device 10 and realized on the same computer.

[0030] As the image recognition processing by the image recognition device 20, a method may be used in which a plurality of images are stored for each object and the object is recognized using pattern matching. At this time, deep learning (deep neural network learning) may be performed using a plurality of images captured from various angles to create a model for recognizing the object. It is generally known that such image recognition processing has a recognition rate that varies depending on the characteristics of the image, such as the brightness of the recognition image and the ease of distinguishing the object to be recognized from the background in the recognition image.

[0031] In addition, the image recognition device 20 has a function of tracking each recognized object by a known technique such as motion compensation between image frames. The image recognition device 20 can associate identification information such as an identification number for identifying the same object with the position information and provide it to the imaging device 10. The above-described recognition result may also include the identification information.

[0032] Here, with reference to FIG. 3, the processing unit 14 of the imaging device 10 will be described. FIG. 3 is a block diagram showing a detailed configuration of the processing unit 14 and the storage unit 1 of FIG. 2. As shown in FIG. 3, the processing unit 14 includes a recognition result acquisition unit 16, a determination unit 17, a motion vector calculation unit 18, and a shooting condition change unit 19.

[0033] The processing unit 14 is, for example, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field-Programmable Gate Array), or a quantum processor (quantum computer control chip). The processing unit 14 causes the memory to read and execute a shooting control program stored in the storage unit 1 provided in the imaging device 10. Thereby, the processing unit 14 realizes the functions of the recognition result acquisition unit 16, the determination unit 17, the motion vector calculation unit 18, and the shooting condition change unit 19, and executes the above-described shooting condition change process. A part or all of each configuration of the processing unit 14 may be realized by a general-purpose or dedicated circuit realized by, for example, a semiconductor device.

[0034] Note that the storage unit 1 may include a non-volatile storage device such as a hard disk or a flash memory, and a memory such as a RAM (Random Access Memory), that is, a volatile storage device. The storage unit 1 stores, for example, a shooting control program and recognition results. The shooting control program is a computer program in which the processing of the control method of the shooting device according to the embodiment is implemented. The recognition results include the recognition rate, position information, and identification information for identifying the same object for each object.

[0035] The recognition result acquisition unit 16 acquires the recognition result obtained by performing the recognition process of the object on the captured image by the image recognition device 20. The recognition result acquisition unit 16 also stores the recognition result in the storage unit 1. The determination unit 17 determines the presence or absence of a saturation area in the captured image. Here, the saturation area refers to a region where the dynamic range of the imaging element of the camera unit 11 is small (insufficient) with respect to the dynamic range of the scene to be photographed, and in the captured image, the signal saturates beyond the minimum or maximum signal intensity that can be digitally represented. The saturation area includes a whiteout area and a blacked-out area where detailed information of the image is lost. The whiteout area refers to an area where the bright part of the image becomes white as a result of overexposure when the camera unit 11 captures a digital image. The blacked-out area refers to an area where the dark part of the image becomes black as a result of underexposure when the camera unit 11 captures a digital image.

[0036] Specifically, the determination unit 17 can acquire the luminance value of each pixel of the captured image, and determine a pixel whose luminance value exceeds the upper threshold value as a whiteout area, and a region where the threshold value is below the lower threshold value as a blacked-out area. Note that the determination unit 17 may divide the captured image into a plurality of blocks, and determine whether each block is a saturation area based on the average luminance value of each block.

[0037] The motion vector calculation unit 18 calculates the motion vector of each object from the transition of the positions (coordinates) of a plurality of objects between consecutive image frames. For example, the motion vector calculation unit 18 can obtain the motion vector by calculating the moving direction and the moving amount of each object using the identification information of each object input from the image recognition device 20 and the position information associated therewith. Note that the motion vector can be calculated using known techniques such as motion compensation processing between image frames.

[0038] In the image recognition system 100, the image quality of the recognition image input to the image recognition device 20 that performs the recognition process greatly affects the identification determination of the object. In a recognition image in which there are mixed black crushed regions with low luminance levels and white crushed regions with high luminance levels, the recognition accuracy of the image recognition device 20 may decrease.

[0039] Therefore, when it is determined that there is a saturation region in the captured image, the shooting condition change unit 19 determines the priority of the shooting conditions based on the state of the recognized object. Then, the shooting condition change unit 19 determines the shooting parameters so that the ratio of shooting the shooting image for performing the recognition process with the shooting conditions having a high priority increases, and outputs a change instruction signal to the control unit 15 in order to change the shooting conditions to the ratio of the determined shooting parameters of the camera unit 11. Specifically, the change instruction signal is, for example, information on the exposure time.

[0040] In this way, when a saturation region is included, the processing unit 14 changes the shooting conditions of the camera unit 11 according to the recognition result from the image recognition device 20. The camera unit 11 further performs shooting under the changed shooting conditions and supplies the recognition image to the image recognition device 20. Then, when the processing unit 14 obtains the recognition result for the recognition image shot under the changed shooting conditions, the processing unit 14 further changes the shooting conditions of the camera unit 11. In this way, the imaging device 10 performs a process of repeatedly changing the shooting conditions of the camera unit 11 according to the recognition result for the recognition image. Thereby, it becomes possible to generate a shooting image capable of improving the recognition accuracy of the image recognition device 20.

[0041] Here, with reference to FIGS. 4A and 4B, a shooting control method according to an embodiment will be described. FIGS. 4A and 4B are flowcharts for explaining the control method according to the embodiment. In the present embodiment, (1) first, shooting is performed with normal exposure, recognition processing of the non-saturated region in the captured image is performed, and then the shooting conditions are changed to perform recognition processing of the saturated region, and recognition processing of the entire image is performed. Then, (2) based on the state of the object recognized in the entire image, the priority of the shooting conditions is determined, and the process of determining the ratio of the shooting conditions is performed. Then, (3) based on the determined ratio, the shooting conditions are sequentially changed, and a loop process of repeating shooting and recognition processing is performed.

[0042] (1) Recognition processing of the entire image Referring to FIG. 4A, the imaging device 10 first performs imaging processing under the set shooting conditions of normal exposure (step S10). Then, the imaging device 10 obtains the average luminance of the image (step S11). The imaging device 10 can obtain the average luminance, for example, by acquiring the luminance values of each pixel of the image. Note that the imaging device 10 can also obtain the luminance histogram of the image and obtain the average luminance. Then, the luminance of each pixel of the image is adjusted so that the average luminance of the image becomes the target value (step S12). For example, the imaging device 10 may adjust the luminance of each pixel by changing the shooting conditions including the setting of the exposure time of normal exposure and performing imaging processing again, or may adjust the luminance of each pixel by various image processes such as gamma processing by the signal processing unit 12.

[0043] Thereafter, recognition processing of the object is executed on the image whose luminance has been adjusted (step S13). Note that continuous shooting may be performed without changing the shooting conditions until the recognition result satisfies a predetermined standard, and the object recognition processing may be repeatedly performed (step S14). However, step S14 may be skipped. At this time, image quality adjustment for recognition processing may be performed as preprocessing of the frame image input to the image recognition device 20, the recognition processing result may be fed back, and readjustment may be repeated.

[0044] At this time, based on the recognition rate of the recognition result by image recognition for the target image after image quality adjustment, narrowing down the recognition target area of the captured image, selecting the image quality type to be adjusted, and finely adjusting the adjustment value for each image quality type are performed, and the image quality adjustment parameter value can be optimized. That is, it is possible to adjust to an image quality advantageous for image recognition processing, that is, an image quality with an improved recognition rate.

[0045] For example, apart from the standard image quality adjustment suitable for human visual recognition for an image captured under low illumination conditions, such as the processing performed in step S12, image quality adjustment more suitable for recognition processing, for example, increasing the luminance, is performed. Therefore, although the image becomes one in which noise becomes conspicuous as the luminance increases, image data adjusted to an image quality advantageous for image recognition processing, for example, an image in which the image of a person who is the recognition target object becomes clearer, can be input to the image recognition device 20.

[0046] Note that "the recognition result satisfies a predetermined standard" means, for example, "the recognition rate in the next image recognition becomes a predetermined value or more" or "the recognition frequency is a stable number of times or more". The "recognition frequency" is the number of successful image recognitions in the total number of image recognitions within a certain time period. Specifically, the number of times of successfully recognizing a predetermined target object in the image can be used as the recognition number, and the recognition frequency can be calculated as the recognition number per total recognition number.

[0047] When the object recognition process is completed, the recognition result acquisition unit 16 stores the recognition result in the storage unit 1 (step S15). Here, the recognition result may include image data for one frame of the area where the object recognition process was performed, the type, size, position, identification information, etc. of the object. Further, the recognition result may include image data for each frame surrounding the area of the object. When there is no saturated area in the image, the image data for the entire image is stored in the storage unit 1. On the other hand, when the image contains a saturated area, only the image data for the non-saturated area obtained by removing the saturated area from the entire image may be stored in the storage unit 1. When the image contains a saturated area, the recognition result acquisition unit 16 adds information for identifying that the object exists in the non-saturated area to the recognition result and stores it in the storage unit 1. The recognition result acquisition unit 16 may compress the recognition result and store it in the storage unit 1. Specifically, the recognition result acquisition unit 16 reduces the image data in the recognition result at an arbitrary reduction rate using a known image processing technique and stores it in the storage unit 1. Here, the arbitrary reduction rate can be determined, for example, on the condition that a person can visually recognize the object in the reduced image data.

[0048] As described above, an image may contain a saturated area where detailed information of the image is lost. In the saturated area, it is not possible to perform the object recognition process. Therefore, in step S16, the determination unit 17 determines whether there is no saturated area in the image. When there is no saturated area in the frame image (step S16, YES), the determination unit 17 determines that the entire image has an optimal exposure. FIG. 5 is a diagram showing an example of an image for recognition when there is no saturated area. In the example shown in FIG. 5, since the dynamic range of the imaging element of the camera unit 11 is sufficient for the dynamic range of the scene, the low-luminance area and the high-luminance area can be depicted without breakdown. When there is no saturated area, the vehicle T1 can be recognized from the area A1 outside the tunnel in the image, and the person T2 can be detected from the area A2 inside the tunnel. In this case, the display device 30 can display the recognition result by superimposing it on the image (step S17). Then, the process returns to step S10.

[0049] On the other hand, if there is a saturated area in the image (step S16, NO), the process proceeds to step S20. In step S20, the shooting condition changing unit 19 obtains the exposure time to be changed based on the luminance information of the saturated area. As described above, the saturated area includes a white overflow area and a black overflow area.

[0050] FIG. 6 is a diagram showing an example of a recognition image when the saturated area is a blacked-out area. In the recognition image shown in FIG. 6, the area A1 outside the tunnel has an appropriate luminance when photographed with normal exposure. On the other hand, the area A2 inside the tunnel has insufficient exposure and becomes a blacked-out area. In the blacked-out area, the recognition process of the object cannot be performed. Therefore, in the recognition process of step S13, the recognition process of the object is performed only on the area of the image excluding the blacked-out area. In the example of FIG. 6, the recognition process of the object is performed only on the area A1 outside the tunnel where the road outside the tunnel is photographed, and a vehicle T1 or the like is detected as the object. In this case, in step S20, the exposure time is made longer than the normal exposure (long-time exposure) so that the luminance of the blacked-out area corresponding to the area A2 inside the tunnel in FIG. 6 becomes brighter.

[0051] FIG. 7 is a diagram showing an example of a recognition image when the saturated area is a white overflow area. In the recognition image shown in FIG. 7, the area A2 inside the tunnel has an appropriate luminance when photographed with normal exposure. On the other hand, the area A1 outside the tunnel has excessive exposure and becomes a white overflow area. In the white overflow area, the recognition process of the object cannot be performed. Therefore, in the recognition process of step S13, the recognition process of the object is performed only on the area of the image excluding the white overflow area. In the example of FIG. 7, the recognition process of the object is performed only on the area A2 inside the tunnel, and a person T2 or the like is detected as the object. In this case, in step S20, the exposure time is made shorter than the normal exposure (short-time exposure) so that the luminance of the white overflow area corresponding to the area A1 outside the tunnel in FIG. 7 becomes darker.

[0052] Then, the shooting condition changing unit 19 outputs the determined exposure time to the control unit 15 as a change instruction signal. The control unit 15 changes the exposure time by changing the shutter speed of the camera unit 11 (step S21). Thereby, the saturation area can be eliminated. In this way, the shooting conditions related to the exposure for improving the saturation area are called saturation improvement exposure. Also, an area that was a saturation area in the image and has become a non-saturation area by changing the shooting conditions is called a saturation improvement area.

[0053] After that, object recognition processing is executed on the saturation improvement area (step S22). Thereby, object recognition processing can be performed over the entire image including the saturation area. Similar to step S14, for the saturation improvement area, continuous shooting may be performed without changing the shooting conditions until the recognition result meets a predetermined standard, and the object recognition processing may be repeatedly executed (step S23). However, step S23 may be skipped. At this time, image quality adjustment for recognition processing may be performed as preprocessing of the image input to the image recognition device 20, the recognition processing result may be fed back, and readjustment may be repeated.

[0054] When the object recognition processing for the saturation improvement area is completed, the recognition result acquisition unit 16 saves the recognition result in the storage unit 1 (step S24). In step S24, the recognition result acquisition unit 16 saves only the recognition result of the entire image or the recognition result of the saturation improvement area excluding the non-saturation area from the entire image in the storage unit 1. Also, the recognition result acquisition unit 16 adds information for identifying that it is an object in the saturation improvement area to the recognition result and saves it in the storage unit 1. For other processes, the same processes as in step S15 are performed. The display device 30 can display the recognition result by superimposing it on an image that simultaneously displays the non-saturation area and the saturation improvement area (step S30). As described above, when the shooting conditions are changed from normal exposure to saturation improvement exposure, the display device 30 can display, as a still image, the image captured under the previous shooting conditions (normal exposure) saved in the storage unit 1 with respect to the non-saturation area. Also, the display device 30 can display, in real time, the moving image captured under saturation improvement exposure with respect to the saturation improvement area.

[0055] Note that the display device 30 can also display a composite image based on composite image data obtained by combining a normal exposure image and a saturation improvement exposure image.

[0056] FIG. 8 is a diagram showing an example of a display image in which an unsaturated region (normal exposure region) A1 and a blackout improvement region (long-time exposure region) A2 are simultaneously displayed when the saturated region is a blackout region. In FIG. 8, the region A1 in which the vehicle T1 is detected is a normal exposure image of a still image. FIG. 9 is a diagram showing an example of a display image in which an unsaturated region (normal exposure region) A2 and a whiteout improvement region (short-time exposure region) A1 are simultaneously displayed when the saturated region is a whiteout region. In FIG. 9, the region in which the person T2 is detected is a normal exposure image of a still image.

[0057] (2) Shooting condition ratio determination process Referring to FIG. 4B, subsequently, in steps S31 and later, based on the state of the object recognized in the entire image, the priority of the shooting conditions is determined, and a shooting condition ratio determination process is performed. For example, when the speed of the recognized object is relatively high and the distance to the imaging device 10 is relatively short, the recognized object is dangerous for the driver of the vehicle on which the imaging device 10 is installed. Therefore, it is desirable to continue monitoring the operation of the object. In this case, if the shooting conditions are changed from saturation improvement exposure to normal exposure, the object recognized in the saturation improvement region cannot be detected. Therefore, in the embodiment, the ratio of the shooting conditions is determined based on at least one of the speed of the object recognized in the saturation improvement region and the distance to the imaging device 10.

[0058] The shooting condition changing unit 19 calculates the moving speed of each object recognized in the image (step S31). Here, each object that is the processing target of the shooting condition changing unit 19 refers to both each object located in the non-saturated region recognized in step S13 and each object located in the saturation improvement region recognized in step S22. In step S31, the moving speed of each object can be the magnitude of the motion vector obtained by the motion vector calculation unit 18. Here, the moving speed of each object can be the relative speed with respect to the imaging device 10.

[0059] Next, the shooting condition changing unit 19 calculates the distance from the imaging device 10 to each object (step S32). The distance from the imaging device 10 to each object can be measured, for example, using a sensor capable of measuring the distance between the imaging device 10 and the object. Note that it is also possible to calculate the distance using the angle-of-view information and installation conditions of the imaging device 10.

[0060] Then, the shooting condition changing unit 19 determines the priority of the shooting conditions based on the moving speed of each object and the distance from the imaging device 10 to each object, and determines the shooting parameters so that the ratio of taking the shooting images for performing the recognition process with the shooting conditions having a higher priority increases (step S33). For example, if the moving speed of the object recognized in the non-saturated region is relatively slower compared to the object recognized in the saturation improvement region, and the distance from the object recognized in the non-saturated region to the imaging device 10 is relatively farther compared to the object recognized in the saturation improvement region, the object recognized in the non-saturated region is not dangerous for the driver of the vehicle on which the imaging device 10 is installed, so the monitoring of the operation of the object can be paused. For example, in FIG. 8, if the relative moving speed of the person T2 is faster than the relative moving speed of the vehicle T1, and the person T2 is closer to the imaging device 10 than the vehicle T1, the shooting conditions of the saturation improvement region A2 where the person T2 is located have a higher priority. In this case, even if the ratio of the imaging conditions is set to be higher for the saturation improvement exposure, since the risk level of the recognized object is low, it often does not pose a major problem for the driver.

[0061] Further, the shooting condition changing unit 19 may determine the ratio of shooting conditions based on either the moving speed of the object or the distance from the imaging device 10 to the object. Further, not limited to the moving speed of the object and the distance from the imaging device 10 to the object, the ratio of the shooting conditions may be determined based on other conditions such as the type of the object. That is, the shooting condition changing unit 19 may increase the ratio of the shooting conditions for the area where a highly dangerous recognized object exists.

[0062] Here, the ratio of the shooting conditions can be, for example, 2:1. For example, when the priority of the shooting conditions for saturation improvement exposure is high, the shooting condition changing unit 19 sets the ratio of the shooting conditions for saturation improvement exposure to 2 and the ratio of the shooting conditions for normal exposure to 1. In this case, the shooting condition changing unit 19 sets the order of the shooting conditions to repeat as (1) saturation improvement exposure (2) saturation improvement exposure (3) normal exposure. The ratio of the shooting conditions is not limited to this, and may be, for example, 3:2. In this case, the shooting condition changing unit 19 may repeat the order of the shooting conditions as (1) saturation improvement exposure (2) normal exposure (3) saturation improvement exposure (4) normal exposure (5) saturation improvement exposure, or may repeat as (1) saturation improvement exposure (2) saturation improvement exposure (3) saturation improvement exposure (4) normal exposure (5) normal exposure. That is, the shooting condition changing unit 19 can determine the ratio and order of the shooting conditions. Here, when no object is recognized in either the non-saturated area or the saturation improvement area, the ratio may be 1:1.

[0063] (3) Sequentially change the shooting conditions based on the determined ratio, and repeat the shooting and recognition processes loop process Subsequently, after step S40, based on the ratio of the shooting conditions determined in step S33, the shooting conditions are sequentially changed, and a loop process of repeating the shooting and recognition processes is performed. First, it is determined whether the next shooting condition is normal exposure or saturation improvement exposure (step S40). If the next shooting condition is normal exposure (step S40, YES), the process proceeds to step S50. Since step S50 is the same process as steps S10 to 15 described above, the description is omitted. If the next shooting condition is saturation improvement exposure (step S40, NO), the process proceeds to step S60. Since step S60 is the same process as steps S20 to 24 described above, the description is omitted.

[0064] Then, the process of S70 is performed. In S70, similar to the processes of S17 or S30 described above, the process of displaying the recognition result on the display device 30 is executed. Subsequently, it is determined whether to end the loop process based on the determined ratio (step S71). The conditions for ending the loop process are considered as follows, but are not limited to this. · When no object is recognized in either the non-saturated area or the saturation improvement area in the processes of step S50 and step S60. · When a new object that was not recognized at the time of the process of step S33 is recognized in the processes of step S50 and step S60. · When there is no saturated area in the process of step S50. · When a preset period (for example, 30 seconds) has elapsed since the process of step S33 was performed. Case.

[0065] If any of the conditions for ending the above loop process is met (step S71, YES), the process returns to the process of step S10. If none of the conditions for ending the above loop process is met (step S71, NO), the process returns to the process of step S40.

[0066] The above process is repeatedly executed to perform recognition processing. As a result, among a plurality of shootings for which recognition processing is executed, it is possible to determine the ratio of the first shooting condition for the saturation region and the second shooting condition for the non-saturation region other than the saturation region. Also, the order of normal exposure and short-time exposure is determined according to the state of the recognized object.

[0067] In this way, by increasing the ratio of the shooting conditions with high priority among the plurality of shooting conditions, it is possible to obtain a recognition image capable of tracking a dangerous object, that is, to generate an appropriate image.

[0068] Note that the long-time exposure has a longer exposure time than the normal exposure. In normal exposure, when accumulating light for one frame (for example, for 1 / 30 second), in long-time exposure, light may be accumulated over two or more frames.

[0069] Note that usually, at the boundary between the actual shooting scene that becomes the overexposed region and the non-saturation region in the recognition image, there is little sudden lack of illuminance. However, when the object is located at the boundary between the overexposed region and the non-saturation region, if the exposure time of the overexposed region is shortened, there is a risk of insufficient illuminance for the object. In this case, for example, noise reduction processing for performing interpolation processing using surrounding values or gain processing may be performed to ensure the illuminance around the boundary.

[0070] Also, for example, when driving at night, the headlights of an oncoming vehicle may enter the field of view of the imaging device 10 mounted on the own vehicle, and it may be difficult to recognize an object in the peripheral region of the headlights. When there is a person around the headlights, the risk level is particularly high. Therefore, a headlight detection unit can be provided in the processing unit 14.

[0071] When a headlight is detected in the recognition image, it is also possible to set the order of the shooting conditions so that the ratio of the shooting conditions for improving the recognition accuracy of the peripheral region of the headlight increases. As a result, the image around the headlight can be sharpened, and the recognition accuracy can be improved.

[0072] Note that for the detection of the headlight, for example, in the input image, known techniques such as determining whether there is a set of regions with brightness assumed when the headlight is on can be used.

[0073] As described above, according to the embodiment, even for an imaging device with a relatively narrow dynamic range, it is possible to improve the recognition accuracy by the image recognition device at a lower cost. Although the description has been made on the premise of the interlace scanning method so far, shooting may be performed by the interlace scanning method. [First Modification Example] With reference to FIGS. 10A and 10B, the flow of the reproduction process according to the first modification example of the embodiment will be described. FIGS. 10A and 10B are flowcharts for explaining the control method according to the first modification example. The configuration of the imaging device 10 according to the first modification example of the embodiment is the same as the block diagram shown in FIG. 2. However, the recognition result acquisition unit 16 according to the first modification example determines the reduction rate of each of the recognition result for the image captured under the first shooting condition and the recognition result for the image captured under the second shooting condition based on the ratio of the shooting conditions, and reduces them at the respective reduction rates and stores them in the storage unit 1, which is different from the recognition result acquisition unit 16 described above.

[0074] The processes from step S210 to step S240 are the same as the processes from step S10 to step S40 shown in FIGS. 4A and 4B respectively, and the process of step S271 is the same as the process of step S71, so the description is omitted.

[0075] Step S250 is the same process as steps S210 to S214. That is, since it is the same process as steps S10 to 14 described above, the description is omitted. After the processing unit 14 executes the recognition process on the image captured by normal exposure in step S213, or until the recognition result satisfies a predetermined standard in step S214, continuous shooting is performed without changing the shooting conditions, and the object recognition process is repeatedly performed, and then the process proceeds to step S251.

[0076] The recognition result acquisition unit 16 stores the recognition result in the storage unit 1 based on the ratio of the imaging conditions determined by the imaging condition change unit 19 in step S233 (step S251). Specifically, the recognition result acquisition unit 16 compresses the recognition result for the image captured under normal exposure and stores it in the storage unit 1 based on the ratio of the imaging conditions of normal exposure. Here, for example, when the ratio of saturated improvement exposure is larger than the ratio of normal exposure, the recognition result acquisition unit 16 reduces the image data in the recognition result at an arbitrary reduction rate. Conversely, when the ratio of saturated improvement exposure is smaller than the ratio of normal exposure, the recognition result acquisition unit 16 stores the image data in the recognition result in the storage unit 1 without reduction.

[0077] When the recognition result includes image data for each frame surrounding the object region in the recognition result, the recognition result acquisition unit 16 may reduce each piece of image data at a different reduction rate and store it in the storage unit 1. For example, the reduction rate for each is determined based on the moving speed of each object acquired in step S231. Specifically, the recognition result acquisition unit 16 decreases the reduction rate in the order of relatively faster moving speeds of the objects, and determines each reduction rate so that the resolution of the reduced image data becomes larger as the moving speed of the object is faster. For example, the reduction rate for each is determined based on the distance from the imaging device 10 to each object acquired in step S232. Specifically, the recognition result acquisition unit 16 decreases the reduction rate in the order of relatively shorter distances from the object to the imaging device 10, and determines each reduction rate so that the resolution of the reduced image data becomes larger as the distance from the object to the imaging device 10 is shorter. Also, the recognition result acquisition unit 16 may set a lower limit value for the reduction rate to such an extent that the visibility can be maintained even after reducing the object at a far distance from the imaging device 10.

[0078] Step S260 is the same process as steps S220 to S223. That is, since it is the same process as steps S20 to S23 described above, the description is omitted. After performing recognition processing on the saturation improvement region in the image captured by saturation improvement exposure in step S222, or until the recognition result satisfies a predetermined standard in step S223, continuous shooting is performed without changing the shooting conditions, and after repeatedly performing object recognition processing, the process proceeds to step S261.

[0079] The recognition result acquisition unit 16 stores the recognition result in the storage unit 1 based on the ratio of the shooting conditions determined by the shooting condition change unit 19 in step S233 (step S261). Specifically, the recognition result acquisition unit 16 compresses the recognition result for the image captured by saturation improvement exposure based on the ratio of the shooting conditions of saturation improvement exposure and stores it in the storage unit 1.

[0080] The display device 30 displays the recognition result by superimposing it on an image in which the non-saturated region and the saturation improvement region are simultaneously displayed (step S270). Specifically, the output unit 13 generates the display content based on the recognition result of the non-saturated region saved in step S251 and the recognition result of the saturation improvement region saved in step S261. Then, the display device 30 displays the display content. For example, the output unit 13 acquires the saved recognition result via the recognition result acquisition unit 16 or the like, or directly from the storage unit 1. The output unit 13 synthesizes the image data of the non-saturated region and the image data of the saturation improvement region included in the recognition result, and draws a frame surrounding the recognized object region or the like to generate the display content. When the reduction ratios are different between the recognition result of the non-saturated region and the recognition result of the saturation improvement region, the resolutions of the image data of the non-saturated region and the image data of the saturation improvement region may be different. In this case, the output unit 13 performs enlargement processing on the image data with a lower resolution to match the image data with a higher resolution and then performs synthesis.

[0081] In the first modification of the embodiment, the reduction rate of the recognition result is determined based on the ratio of the shooting conditions. Therefore, it is possible to appropriately reduce the capacity of the storage unit 1 because it is possible to save the capacity of the storage unit 1 while clearly storing the images of the shooting conditions including more important objects. [Second Modification] With reference to FIGS. 11, 12A, and 12B, the flow of the reproduction process according to the second modification of the embodiment will be described. FIG. 11 is a block diagram showing the configuration of the processing unit according to the second modification of the embodiment. FIGS. 12A and 12B are flowcharts for explaining the control method according to the second modification. The configuration of the imaging device 10 according to the second modification of the embodiment is the same as the block diagram shown in FIG. 2. However, as shown in FIG. 11, the processing unit 114 according to the second modification further includes a recognition synthesis unit 21 that synthesizes the captured images of both the first shooting condition and the second shooting condition to generate a recognition synthesis image, and the shooting condition change unit 19 determines the priority of each of the first shooting condition, the second shooting condition, and the third shooting condition for acquiring the recognition synthesis image, which is different from the processing unit 14 in FIG. 3.

[0082] The processing from step S310 to step S330 is the same as the processing from step S10 to step S33 shown in FIGS. 4A and 4B, respectively, and thus the description thereof is omitted. However, the steps corresponding to step S14 and step S23 are deleted in FIGS. 12A and 12B. After the display device 30 displays the recognition result in step S330, the processing unit 114 proceeds to step S331.

[0083] The recognition synthesis unit 21 generates a recognition synthesis image for performing the image recognition process (step S331). The recognition synthesis unit 21 generates a recognition synthesis image by synthesizing the image data captured under the normal exposure shooting conditions stored in the storage unit 1 in step S315 and the image data captured under the saturation improvement exposure shooting conditions stored in the storage unit 1 in step S324. Specifically, the recognition synthesis unit 21 generates a recognition synthesis image by adding the pixel values of the corresponding pixels of the entire image data captured under the normal exposure shooting conditions and the entire image data captured under the saturation improvement exposure shooting conditions.

[0084] FIG. 13 is a diagram showing an example of an image taken by normal exposure when an object exists across a saturation region and a non-saturation region. FIG. 14 is a diagram showing an example of an image taken by saturation-improved exposure when an object exists across a saturation region and a non-saturation region. FIG. 15 is a diagram showing an example of a recognition image when an object exists across a saturation region and a non-saturation region. For example, assume that the image shown in FIG. 13 is taken in step S315 and the image shown in FIG. 14 is taken in step S324. Here, in FIGS. 13 and 14, a vehicle T3 as an object exists across an out-of-tunnel region A1 and an in-tunnel region A2. Therefore, in the image of FIG. 13 taken by normal exposure, only the right half of the vehicle T3 can be visually recognized, and in the image of FIG. 14 taken by saturation-improved exposure, only the left half of the vehicle T3 can be visually recognized. In a known image recognition technique including the image recognition apparatus 20, when only a part of the object can be visually recognized in this way, the vehicle T3 cannot be recognized, or the reliability of the recognition of the vehicle T3 decreases. On the other hand, in the recognition composite image generated by the recognition composite unit 21, the entire vehicle T3 can be visually recognized as shown in FIG. 15, so that the image recognition apparatus 20 can recognize the vehicle T3.

[0085] The recognition composite unit 21 may generate a recognition composite image by adding with different weights for each region where the pixel is located. For example, for the pixels located in the out-of-tunnel region A1, since the image taken by the saturation-improved exposure shown in FIG. 14 is an image that is easier to recognize, a large weight is given to the image taken by the saturation-improved exposure and the pixel values are added. Conversely, for the pixels located in the in-tunnel region A2, since the image taken by the normal exposure shown in FIG. 13 is an image that is easier to recognize, a large weight is given to the image taken by the normal exposure and the pixel values are added.

[0086] The process in which the recognition composite unit 21 generates a recognition composite image is not limited to this, and the recognition composite unit 21 may generate a recognition composite image by superimposing the in-tunnel region A2 shown in FIG. 13 and the out-of-tunnel region 13 shown in FIG. 14.

[0087] The recognition synthesis unit 21 supplies the recognition synthesis image to the image recognition device 20, and the image recognition device 20 performs image recognition processing on the recognition synthesis image (step S332). Subsequently, the recognition result acquisition unit 16 stores the recognition result obtained from the image recognition device 20 in the storage unit 1 (step S333).

[0088] For example, when performing image recognition processing on the recognition synthesis image shown in FIG. 15, the person T2 in the in-tunnel area A2 and the vehicle T3 existing across the out-of-tunnel area A1 and the in-tunnel area A2 are detected as objects and included in the recognition result output by the image recognition device 20. At this time, the recognition result acquisition unit 16 adds information for identifying that the vehicle T3 is an object existing across the out-of-tunnel area A1 and the in-tunnel area A2 to the recognition result and stores it in the storage unit 1. That is, the recognition result acquisition unit 16 adds information for identifying that it is an object existing across the saturation area and the non-saturation area to the recognition result and stores it in the storage unit 1.

[0089] Here, since the person T2 among the objects included in the recognition result is only included in the in-tunnel area A2, that is, the saturation improvement area, it is also included in the recognition result obtained by the recognition result acquisition unit 16 in step S324 and is already stored in the storage unit 1. Therefore, the recognition result acquisition unit 16 in S333 may store only the recognition result of the vehicle T3 in the storage unit 1. That is, the recognition result acquisition unit 16 may store only the recognition results of the objects existing across the saturation area and the non-saturation area in the storage unit 1.

[0090] The shooting condition change unit 19 calculates the moving speed of each object recognized in the image (step S334). Here, each object that is the processing target of the shooting condition change unit 19 refers to all of the objects located in the non-saturation area recognized in step S313, the objects located in the saturation improvement area recognized in step S322, and the objects existing across the saturation improvement area and the non-saturation area recognized in step S332. Next, the shooting condition change unit 19 calculates the distance from the imaging device 10 to each object (step S335).

[0091] Then, based on the moving speed of each object and the distance from the imaging device 10 to each object, the imaging condition changing unit 19 determines the priority of the imaging conditions, and determines the imaging parameters so that the ratio of taking the imaging images for performing the recognition process under the imaging conditions with higher priority increases (step S336). Here, since the process of step S336 is similar to the process of step S33 described above, only the parts different from the process of step S33 will be described.

[0092] Based on the moving speed of each object recognized in the non-saturated region, the object recognized in the saturation improvement region, and the object existing across the saturated region and the non-saturated region, and the distance from the imaging device 10 to each object, the imaging condition changing unit 19 determines the ratio of each imaging condition of the composite imaging that takes and synthesizes both the saturation improvement exposure, the normal exposure, and both the saturation improvement exposure and the normal exposure. The detailed process of the composite imaging will be described later. Here, when there is no object existing across the saturated region and the non-saturated region, the ratio of the composite imaging among the ratios of the imaging conditions may be set to zero.

[0093] Here, the imaging conditions of the saturation improvement exposure, the normal exposure, and the ratio of each of the composite imaging can be, for example, 2:1:1. For example, when the priority of the imaging conditions of the saturation improvement exposure is the highest and the other conditions have the same priority, the imaging condition changing unit 19 sets the ratio of the imaging conditions of the saturation improvement exposure to 2, the ratio of the imaging conditions of the normal exposure to 1, and the ratio of the composite imaging to 1. In this case, the imaging condition changing unit 19 sets the order of the imaging conditions to repeat as (1) saturation improvement exposure (2) saturation improvement exposure (3) normal exposure (4) composite imaging.

[0094] Subsequently, after step S340, based on the ratio of the shooting conditions determined in step S336, a loop process is performed in which the shooting conditions are sequentially changed and shooting and recognition processing are repeated. First, the next shooting condition is determined (step S340). If the next shooting condition is normal exposure (step S340, normal exposure), the process proceeds to step S350. Since step S350 is the same process as steps S310 to 315 described above, the description is omitted. If the next shooting condition is saturation improvement exposure (step S340, saturation improvement exposure), the process proceeds to step S360. Since step S360 is the same process as steps S320 to 324 described above, the description is omitted.

[0095] If the next shooting condition is composite shooting (step S340, composite shooting), the process proceeds to step S370. The imaging device 10 performs shooting under normal exposure (step S370). Specifically, the shooting condition changing unit 19 controls the camera unit 11 by outputting the setting of the exposure time for normal exposure to the control unit 15, and the camera unit 11 performs shooting processing under the shooting conditions of normal exposure. When the recognition composite unit 21 acquires the image data of normal exposure from the signal processing unit 12, the recognition composite unit 21 stores the image data in the storage unit 1. For example, when the recognition composite unit 21 acquires the image data shown in FIG. 13, the recognition composite unit 21 may store all of the image data in the storage unit 1, or may perform trimming processing only on the tunnel inner region A2, which is the non-saturated region portion of the image data, and store it in the storage unit 1. Then, the process proceeds to step S390.

[0096] Subsequently, the imaging device 10 performs shooting with saturation improvement exposure (step S371). Specifically, the shooting condition changing unit 19 controls the camera unit 11 by outputting the setting of the exposure time of the saturation improvement exposure to the control unit 15, and the camera unit 11 performs shooting processing under the shooting conditions of the saturation improvement exposure. When the control unit 14 acquires the image data of the saturation improvement exposure from the signal processing unit 12, it stores the image data in the storage unit 1. For example, when acquiring the image data shown in FIG. 14, the recognition synthesis unit 21 may store all of the image data in the storage unit 1, or may only trim and store in the storage unit 1 only the tunnel outer region A1 which is the portion corresponding to the saturation improvement region among the image data. Then, the process proceeds to step S390.

[0097] Then, the processing unit 14 proceeds to step S372. Since step S372 is the same processing as steps S331 to S333 described above, the description is omitted. Then, the process proceeds to step S390.

[0098] Then, the display device 30 displays the recognition result (step S390). Subsequently, it is determined whether to end the loop processing based on the determined ratio (step S391). The conditions for ending the loop processing are as follows, but are not limited to this. · When no object is recognized in any of the processes of step S350, step S360, and step S372. · When a new object that was not recognized at the time of the process of step S336 is recognized in any of the processes of step S350, step S360, and step S372. · When there is no saturation region in the process of step S350. · When a preset period (for example, 30 seconds) has elapsed since the time of the process of step S336 case.

[0099] In the second modification of the embodiment, as shooting conditions, it further has composite shooting in which both saturation improvement exposure and normal exposure are shot and composite is performed. Therefore, an object that exists straddling the saturated region and the non-saturated region, that is, an object located at the boundary between the saturated region and the non-saturated region and that may not be recognizable in the image before composite can be surely recognized. Also, in the second modification of the embodiment, the respective moving speeds of the objects in the non-saturated region, the objects in the saturation improvement region, and the objects that exist straddling the saturated region and the non-saturated region, and the distances from the imaging device 10 to each object are used to determine the ratio of the shooting conditions. Thereby, since the images shot under important shooting conditions can be continuously subjected to recognition processing, more appropriate images can be generated.

[0100] Each functional block that performs various processes described in the drawings can be configured, in terms of hardware, by a processor, a memory, and other circuits. Also, it is possible to realize the above-described processes by causing the processor to execute a program. Therefore, these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and are not limited to any one of them.

[0101] The above-described program can be stored using various types of non-transitory computer readable media and supplied to a computer. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include semiconductor memories (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM). Also, the program may be supplied to a computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. A transitory computer readable media can supply the program to a computer via a wired communication path such as electric wires and optical fibers, or a wireless communication path.

[0102] The content of the present disclosure can be used in various fields that utilize image recognition.

[0103] Some or all of the above embodiments can be described as follows in the appended claims, but are not limited thereto. (Appended Claim A1) An imaging unit that captures a captured image under set imaging conditions; A recognition result acquisition unit that acquires a recognition result obtained by performing recognition processing of an object on the captured image by an image recognition device; A determination unit that determines the presence or absence of a saturation region in the captured image; When it is determined that there is a saturation region in the captured image, based on the state of the recognized object, determine the priority of the imaging conditions, determine imaging parameters so that the ratio of capturing a captured image for performing recognition processing under imaging conditions with a high priority increases, and change the imaging conditions of the imaging unit to the ratio of the determined imaging parameters. An imaging condition changing unit; Including, An imaging device. (Appended Claim A2) The shooting condition changing unit determines the ratio between a first shooting condition for the saturation region and a second shooting condition for a non-saturation region other than the saturation region in a plurality of shootings executed within a predetermined period during which the recognition process is executed. The imaging device according to Addendum A1. (Addendum A3) The shooting condition changing unit determines the priority based on at least one of the movement of the recognized object and the distance between the recognized object and the imaging unit. The imaging device according to Addendum A1 or A2. (Addendum A4) The first shooting condition includes a long exposure condition when the saturation region is a blackout region and a short exposure condition when the saturation region is a whiteout region. The imaging device according to Addendum A3. (Addendum A5) The determination unit determines the presence or absence of the saturation region based on the luminance of each pixel of the captured image. The imaging device according to any one of Addenda A1 to A4. (Addendum A6) It further includes a storage unit for storing the recognition result. The recognition result acquisition unit determines a reduction rate for each of a first recognition result for an image captured under a first shooting condition and a second recognition result for an image captured under a second shooting condition based on the ratio, and reduces the first recognition result and the second recognition result at their respective reduction rates and stores them in the storage unit. The imaging device according to any one of Addenda A1 to A5. (Addendum A7) It further includes a recognition synthesis unit that synthesizes captured images under both the first shooting condition and the second shooting condition to generate a synthesized image for recognition. The shooting condition changing unit in the previous shooting determines the priority for each of the first shooting condition, the second shooting condition, and a third shooting condition for acquiring the synthesized image for recognition. The imaging device according to any one of Addenda A1 to A6 (Addendum B1) A computer A process of capturing a captured image under set shooting conditions, A process of obtaining a recognition result obtained by performing an object recognition process on the captured image by an image recognition device; A process of determining the presence or absence of a saturation region in the captured image; When it is determined that the saturation region exists in the captured image, the priority of the shooting conditions is determined based on the state of the recognized object, and the shooting parameters are determined so that the ratio of shooting the captured image for performing the recognition process under the shooting conditions with high priority increases, and the shooting conditions of the imaging unit are changed to the ratio of the determined shooting parameters. A control method for executing the above. (Appendix C1) A process of capturing a captured image under the set shooting conditions; A process of obtaining a recognition result obtained by performing an object recognition process on the captured image by an image recognition device; A process of determining the presence or absence of a saturation region in the captured image; When it is determined that the saturation region exists in the captured image, the priority of the shooting conditions is determined based on the state of the recognized object, and the shooting parameters are determined so that the ratio of shooting the captured image for performing the recognition process under the shooting conditions with high priority increases, and the shooting conditions of the imaging unit are changed to the ratio of the determined shooting parameters. A program for causing a computer to execute the above.

[0104] Some or all of the elements described in Appendices A2 to A5 subordinate to Appendix A1 (imaging device) may also be subordinate to Appendix B1 (shooting control method) and Appendix C1 (program) in the same subordinate relationship.

Explanation of symbols

[0105] 100 Image recognition system 1 Storage unit 10 Imaging device 11 Camera unit 12 Signal processing unit 13 Output unit 14 Processing unit 15 Control unit 16 Recognition result acquisition unit 17 Determination unit 18 Motion vector calculation unit 19 Shooting condition change unit 20 Image recognition device 30 Display device

Claims

1. An imaging unit that captures a captured image under set imaging conditions; A recognition result acquisition unit that acquires a recognition result obtained by performing recognition processing of an object on the captured image by an image recognition device; A determination unit that determines the presence or absence of a saturation region in the captured image; When it is determined that there is a saturation region in the captured image, based on the state of the recognized object, the priority of the first imaging condition for the saturation region and the second imaging condition for the non-saturation region other than the saturation region is determined, and the imaging parameters are determined so that the ratio of capturing a captured image for performing recognition processing under the imaging condition with the higher priority increases, and the imaging conditions of the imaging unit are changed to the ratio of the determined imaging parameters. An imaging condition changing unit; Including An imaging device.

2. The imaging condition changing unit determines the priority based on at least one of the moving speed of the recognized object and the distance between the recognized object and the imaging unit. The imaging device according to Claim 1.

3. The first imaging condition includes a long exposure condition when the saturation region is a blackout region and a short exposure condition when the saturation region is a whiteout region. The imaging device according to Claim 1.

4. Further comprising a storage unit for storing the recognition result, The recognition result acquisition unit determines a reduction rate of each of a first recognition result for an image captured under the first imaging condition and a second recognition result for an image captured under the second imaging condition based on the ratio, and reduces the first recognition result and the second recognition result at their respective reduction rates and stores them in the storage unit. The imaging device according to Claim 1.

5. Further comprising a recognition synthesis unit that synthesizes captured images under both the first imaging condition and the second imaging condition to generate a synthesized image for recognition, The imaging condition changing unit in the previous stage determines the priority of each of the first imaging condition, the second imaging condition, and a third imaging condition for acquiring the synthesized image for recognition. The imaging device according to Claim 1.

6. A computer, A process of capturing a captured image under set imaging conditions; A process of acquiring a recognition result obtained by performing recognition processing of an object on the captured image by an image recognition device; A process of determining the presence or absence of a saturation region in the captured image; When it is determined that the saturation region exists in the captured image, based on the recognized state of the object, determine the priority between the first shooting condition for the saturation region and the second shooting condition for the non-saturation region outside the saturation region, and determine the shooting parameters so that the ratio of shooting the captured image for performing the recognition process with the shooting condition having the higher priority increases, and a process of changing the shooting condition to the ratio of the determined shooting parameters; A program for causing a computer to execute. **Claim 7** The computer: A process of capturing a captured image under the set shooting conditions; A process of obtaining a recognition result obtained by performing an object recognition process on the captured image by an image recognition device; A process of determining the presence or absence of a saturation region in the captured image; When it is determined that the saturation region exists in the captured image, based on the recognized state of the object, determine the priority between the first shooting condition for the saturation region and the second shooting condition for the non-saturation region outside the saturation region, and determine the shooting parameters so that the ratio of shooting the captured image for performing the recognition process with the shooting condition having the higher priority increases, and a process of changing the shooting condition to the ratio of the determined shooting parameters; A control method for executing.

Images