Imaging device and image processing device

Abstract

To suppress such a situation that an image of a region where a flicker does not occur becomes unnatural by executing processing of suppressing the flicker to a region where the flicker occurs.SOLUTION: An imaging apparatus comprises: an image pick-up device part which has a plurality of photoelectric conversion elements arranged in a matrix shape and is driven for each of a plurality of lines respectively including a prescribed number of photoelectric conversion elements arrayed in one direction; a flicker detection part which detects the occurrence of a flicker on the basis of image data generated by the image pick-up device part; and an exposure adjustment part which makes the exposure frequency in one frame period of the photoelectric conversion element included in the line where the flicker is detected by the flicker detection part greater than the exposure frequency in one frame period of the photoelectric conversion element included in the line where the flicker is not detected.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an imaging device and an image processing device.

Background Art

[0002] When a captured image by an imaging device includes a light source such as a traffic signal, flicker may occur. In order to suppress flicker, for example, the exposure time of the imaging device is adjusted, or image data acquired from the imaging device in multiple times is synthesized.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Patent Document 6

Patent Document 7

Summary of the Invention

Problems to be Solved by the Invention

[0004] When performing a process of suppressing flicker over the entire image based on the image data acquired by the imaging device, blurring or the like may occur in the image of the area where no flicker occurs, and the image may become unnatural.

[0005] The present invention has been made in view of the above points, and aims to prevent the image from becoming unnatural in areas where flicker does not occur by performing a flicker suppression process on areas where flicker occurs. [Means for solving the problem]

[0006] In one aspect of the present invention, the imaging device is The image sensor unit has a plurality of photoelectric conversion elements arranged in a matrix, a flicker detection unit detects a flicker region where flicker occurs using image data acquired by the image sensor unit, and a synthesis processing unit that, when the flicker detection unit detects the flicker region, synthesizes a plurality of image data with different acquisition timings in the flicker region. The image sensor unit has a plurality of lines, each containing a predetermined number of the photoelectric conversion elements arranged in one direction, which are driven multiple times in one frame period, and the flicker detection unit determines a pixel region to be a flicker region if the difference in pixel values ​​for each pixel generated in two drives is greater than or equal to a first threshold, and the sum of the pixel values ​​for each pixel generated in the two drives is greater than or equal to a second threshold. [Effects of the Invention]

[0007] According to the disclosed technology, by applying a flicker suppression process to the areas where flicker occurs, it is possible to prevent the image from appearing unnatural in areas where flicker does not occur. [Brief explanation of the drawing]

[0008] [Figure 1] This is an illustrative diagram showing an example of an image processing system including an image processing device according to the first embodiment. [Figure 2] This block diagram shows an overview of the configuration of various devices mounted on the mobile body shown in Figure 1. [Figure 3] Figure 1 is a block diagram showing an example of the configuration of an imaging device. [Figure 4] Figure 2 is a block diagram showing an example of the configuration of an image processing device. [Figure 5] Figure 1 is an explanatory diagram showing an example of an image acquired by the imaging device 19. [Figure 6] Figure 3 is a timing diagram showing an example of the operation of the imaging device when the flicker detection unit does not detect flicker. [Figure 7] Figure 3 is a timing diagram showing an example of the operation of the imaging device when the flicker detection unit detects flicker. [Figure 8] Figure 3 is a flowchart showing an example of the operation of the imaging device. [Figure 9] This block diagram shows an example of the configuration of an imaging device included in the image processing system of the second embodiment. [Figure 10] It is a timing chart showing an example of the operation of the imaging device in FIG. 9. [Figure 11] It is a flowchart showing an example of the operation of the imaging device in FIG. 9. [Figure 12] It is a block diagram showing an example of the configuration of the imaging device included in the image processing system of the third embodiment. [Figure 13] It is a block diagram showing an example of the image synthesizing unit in FIG. 12. [Figure 14] It is an explanatory diagram showing an example of the operation of the image synthesizing unit in FIG. 13. [Figure 15] It is a block diagram showing an example of the configuration of the imaging device included in the image processing system of the fourth embodiment. [Figure 16] It is a block diagram showing an example of the image synthesizing unit in FIG. 15. [Figure 17] It is an explanatory diagram showing the relationship between the composite gain K calculated by the composite gain calculation unit in FIG. 16, the brightness of the scene, and the frame rate. [Figure 18] It is a flowchart showing an example of the operation of the image processing device in FIG. 16. [Figure 19] It is an explanatory diagram showing an example of the operation of the image synthesizing unit in FIG. 16. [Figure 20] It is a block diagram showing an example of the image synthesizing unit included in the image processing system of the fifth embodiment. [Figure 21] It is an explanatory diagram showing an example of the image data of odd frames and even frames output by the imaging device when the frame rate is set to 120 fps. [Figure 22] It is an explanatory diagram showing an example of the image data of the frames Fm and Fi generated by the frame reconstruction unit in FIG. 20 and the image data of the frames Fu[n] and Fu[n+1] respectively generated by the vertical enlargement unit in FIG. 20. [Figure 23] It is an explanatory diagram showing an example of the operation of the image synthesizing unit in FIG. 20. [Figure 24] It is an explanatory diagram showing an example of outputting image data from the imaging device at a low speed when the frame rate is set to 120 fps. [Modes for carrying out the invention]

[0009] The embodiments will be described below with reference to the drawings. In the following description, image data may be simply referred to as images.

[0010] (First Embodiment) Figure 1 shows an example of an image processing system including an image processing device in the first embodiment. The image processing system 100 shown in Figure 1 is mounted on a mobile body 200, such as an automobile. Cameras or other imaging devices 19a, 19b, 19c, 19d, and 19e are installed in the front, rear, left, and right sides of the mobile body 200 in the direction of travel, and in the front of the interior of the mobile body 200. In the following description, when imaging devices 19a, 19b, 19c, 19d, and 19e are described without distinction, they will also be referred to as imaging device 19.

[0011] The number and placement of the imaging devices 19 installed on the mobile body 200 are not limited to those shown in Figure 1. For example, one imaging device 19 may be installed only in front of the mobile body 200, or two imaging devices 19 may be installed only in the front and rear. Alternatively, the imaging devices 19 may be installed on the ceiling of the mobile body 200. Furthermore, the mobile body 200 on which the image processing system 100 is mounted is not limited to an automobile, but may be, for example, a transport robot or drone operating in a factory. In addition, the image processing system 100 may be a system that processes images acquired from imaging devices other than those installed on the mobile body 200, such as a surveillance camera, a digital still camera, or a digital camcorder.

[0012] The image processing system 100 includes an image processing device 10, an information processing device 11, and a display device 12. In Figure 1, for the sake of clarity, the image processing system 100 is superimposed on an image of the mobile body 200 viewed from above. However, in reality, the image processing device 10 and the information processing device 11 are mounted on a control board or the like mounted on the mobile body 200, and the display device 12 is installed in a position visible to a person inside the mobile body 200. The image processing device 10 may also be mounted on a control board or the like as part of the information processing device 11. The image processing device 10 is connected to each imaging device 19 via signal lines or wirelessly.

[0013] The display device 12 is, for example, a side mirror monitor, rearview mirror monitor, or navigation device display installed on the mobile unit 200. Alternatively, the display device 12 may be a display mounted on the dashboard, or a head-up display that projects images onto a projection board or windshield.

[0014] The information processing device 11 includes a computer such as a processor that performs recognition processing based on image data received via the image processing device 10. For example, the information processing device 11 mounted on the mobile body 200 performs recognition processing on the image data to detect other mobile bodies, signals, signs, road markings, and people, and determines the surrounding conditions of the mobile body 200 based on the detection results. The information processing device 11 may also include an automatic driving control device that controls the movement, stopping, right turns, left turns, etc., of the mobile body 200.

[0015] In this embodiment, the image processing to reduce flicker, which will be described later, is performed by the imaging device 19. For this reason, at least one of the multiple imaging devices 19, either the imaging device 19e or the imaging device 19a installed in front of the moving body 200, includes an image processing device for reducing flicker.

[0016] Figure 2 shows an overview of the configuration of various devices mounted on the mobile unit 200 shown in Figure 1. The mobile unit 200 has an image processing device 10, an information processing device 11, a display device 12, at least one ECU (Electronic Control Unit) 13, and a wireless communication device 14, all interconnected via an internal network. The mobile unit 200 also has a sensor 15, a drive device 16, a lamp device 17, a navigation device 18, and an imaging device 19. For example, the internal network is an in-vehicle network such as CAN (Controller Area Network) or Ethernet (registered trademark).

[0017] The image processing device 10 corrects the image data (frame data) acquired by the imaging device 19 and generates corrected image data. The image processing device 10 may record the generated corrected image data on an external or internal recording device.

[0018] The information processing device 11 may function as a computer that controls various parts of the mobile body 200. The information processing device 11 controls the entire mobile body 200 by controlling the ECU 13. The information processing device 11 may recognize objects outside the mobile body 200 based on images generated by the image processing device 10, and may track the recognized objects.

[0019] The display device 12 displays images and corrected images generated by the image processing device 10. When the mobile body 200 is moving backward, the display device 12 may display an image of the mobile body 200 in the direction of backward movement in real time. The display device 12 may also display images output from the navigation device 18.

[0020] Each ECU 13 is provided to correspond to a mechanical part such as an engine or transmission. Each ECU 13 controls the corresponding mechanical part based on instructions from the information processing device 11. The wireless communication device 14 communicates with devices outside the mobile body 200. The sensor 15 is a sensor that detects various types of information. The sensor 15 may include, for example, a position sensor that acquires the current position information of the mobile body 200. The sensor 15 may also include a speed sensor that detects the speed of the mobile body 200.

[0021] The drive unit 16 consists of various devices for moving the mobile body 200. The drive unit 16 may include, for example, an engine, a steering system, and brakes. The lamp device 17 consists of various lighting fixtures mounted on the mobile body 200. The lamp device 17 may include, for example, headlights, turn signals, taillights, and brake lights. The navigation device 18 is a device that provides directions to a destination via voice and display.

[0022] Figure 3 shows an example of the configuration of the imaging device 19 shown in Figure 1. The imaging device 19 includes a pixel cell array 190, a drive unit 191, a readout unit 192, a bus 193, and an operation control unit 196. The operation control unit 196 includes a flicker detection unit 194 and an exposure adjustment unit 195. The pixel cell array 190 is an example of an image sensor unit.

[0023] Of the imaging devices 19a-19e shown in Figure 1, at least one imaging device 19e or imaging device 19a may have the configuration shown in Figure 3. The remaining imaging device 19 may have a configuration in which the flicker detection unit 194 and exposure adjustment unit 195 are omitted from the configuration shown in Figure 3. Alternatively, the functions of the flicker detection unit 194 and exposure adjustment unit 195 may be masked in the remaining imaging device 19. The configuration in which the flicker detection unit 194 and exposure adjustment unit 195 are omitted from the configuration shown in Figure 3 is shown, for example, as imaging device 19C in Figure 15.

[0024] The pixel cell array 190 has multiple pixel cells PX arranged in a matrix. Although not shown in the figure, each pixel cell PX has, for example, a photodiode, a reset transistor, and a source follower transistor. The pixel cell array 190 is an example of an image sensor, and the pixel cell PX is an example of a photoelectric conversion element. For simplicity, Figure 3 shows an example in which the pixel cell array 190 contains 64 pixel cells PX (8 x 8). In an actual imaging device 19, the pixel cell array 190 has, for example, several hundred thousand pixels to several million pixels.

[0025] The photodiode converts received light into electric charge. The reset transistor is turned on during the high-level period of the reset signal RST (Figure 6), setting the charge storage node between the photodiode and the gate of the source follower transistor to the reset voltage. A transfer transistor may be provided between the photodiode and the storage node to control the charge storage time from the photodiode to the storage node. The source follower transistor outputs an output voltage to the readout unit 192 according to the voltage level received at the gate.

[0026] The drive unit 191 outputs a reset signal RST for each row of pixel cells PX arranged in the X direction in Figure 3. The drive unit 191 also outputs a selection signal SEL to the readout unit 192, which is used to read pixel values ​​from the pixel cells PX that receive the reset signal RST. If a transfer transistor is provided between the photodiode and the storage node in the pixel cell PX, the drive unit 191 outputs a transfer control signal to drive the transfer transistor for each pixel row (X) of pixel cells PX arranged in a predetermined number of rows in the unidirectional X direction. Hereafter, the pixel row (X) will also be referred to as a line.

[0027] The readout unit 192 has a readout circuit (not shown) for each pixel row (Y) of pixel cells PX arranged in the Y direction in Figure 3, including a selection transistor and an amplifier. Each selection transistor, while receiving a high-level selection signal SEL at its gate, outputs an output voltage, which is image information output from the corresponding source follower transistor, to the amplifier. The amplifier amplifies the output voltage received from the selection transistor and outputs the amplified output voltage as image data via the bus 193. The image data is acquired by the acquisition unit 51 of the image processing device 10.

[0028] The flicker detection unit 194 performs flicker detection processing by monitoring the pixel value of each pixel cell PX included in the image data. When the flicker detection unit 194 detects the occurrence of flicker in a pixel cell PX whose pixel value changes periodically, it notifies the exposure adjustment unit 195 of the location of the pixel cell PX where the flicker occurred. For example, flicker occurs depending on the emission period of a light source such as a traffic light, lamp, or car headlight.

[0029] When the exposure adjustment unit 195 receives notification of flicker occurrence from the flicker detection unit 194, it performs exposure adjustment processing to the drive unit 191, which increases the drive frequency (i.e., the number of exposures) of the pixel sequence (X) containing the pixel cell PX in which the flicker occurred. The drive unit 191 sets the drive frequency of the reset signal RST and the selection signal SEL corresponding to the pixel sequence (X) containing the pixel cell PX in which the flicker occurred to a higher than normal level. As a result, as explained in Figure 6, the exposure time of the pixel sequence (X) containing the pixel cell PX in which the flicker was detected is set to be longer than the exposure time of the other pixel sequences (X).

[0030] The flicker detection unit 194 and the exposure adjustment unit 195 may be mounted on the image processing device 10 instead of the motion control unit 196. In this case, the flicker detection unit 194 of the image processing device 10, which has acquired image data via the acquisition unit 51, will detect the occurrence of flicker. When the exposure adjustment unit 195 of the image processing device 10 receives notification of the occurrence of flicker from the flicker detection unit 194, it outputs control information to the motion control unit 196. The motion control unit 196 then controls the drive unit 191 to increase the drive frequency of the pixel sequence (X) containing the pixel cell PX where the flicker occurred, to a higher than normal level.

[0031] Figure 4 shows an example of the configuration of the image processing device 10 in Figure 2. The configuration of the information processing device 11 in Figure 2 is the same as in Figure 4. The image processing device 10 has a CPU (Central Processing Unit) 20, an interface device 21, a drive device 22, an auxiliary storage device 23, and a memory device 24 that are interconnected by a bus.

[0032] The CPU 20 executes various image processing operations, described later, by running the image processing program stored in the memory device 24. The interface device 21 is used to connect to a network (not shown). The auxiliary storage device 23 is, for example, an HDD (Hard Disk Drive) or SSD (Solid State Drive), and holds the image processing program, image data, and various parameters used for image processing.

[0033] The memory device 24 is, for example, a DRAM (Dynamic Random Access Memory) and holds image processing programs, etc., transferred from the auxiliary storage device 23. The drive device 22 has an interface for connecting the recording medium 30 and, for example, transfers the image processing program stored in the recording medium 30 to the auxiliary storage device 23 based on instructions from the CPU 20. The drive device 22 may also transfer image data, etc., stored in the auxiliary storage device 23 to the recording medium 30.

[0034] Figure 5 shows an example of an image IMG acquired by the imaging device 19 in Figure 1. For example, Figure 5 is acquired by the imaging device 19e installed in front of the moving body 200. Note that the symbols L1-L11 indicate the line numbers of the pixel array (X) added for reference and are not included in the image IMG. For the sake of simplicity, the pixel array (X) is shown enlarged vertically in the image IMG.

[0035] The image in Figure 5 includes two vehicles traveling on a road, a traffic light, and trees. The traffic light has blue, yellow, and red lights arranged from left to right in the image. In the example in Figure 5, the darker blue and yellow lights indicate that the lights are off, while the lighter red light indicates that the lights are on. For example, the blue, yellow, and red lights are illuminated using LED (Light Emitting Diode) light sources. LED light sources blink repeatedly at a predetermined cycle, which can cause flicker.

[0036] The flicker detection unit 194 in Figure 3 detects the flicker of the illuminated red light of the traffic signal and notifies the exposure adjustment unit 195 of the line numbers L3 and L4, which include the red light. Based on the notification from the flicker detection unit 194, the exposure adjustment unit 195 controls the drive unit 191 to increase the exposure time of lines (pixel rows (X)) L3 and L4 compared to the exposure time of other lines. For example, the increase in exposure time is achieved by increasing the number of exposures in one frame period.

[0037] Figure 6 shows an example of the operation of the imaging device 19 when the flicker detection unit 194 in Figure 3 does not detect flicker. In Figure 6, the imaging device 19 is also assumed to have 64 pixels. The imaging device 19 acquires an image using a rolling shutter method, sequentially exposing all pixel cells PX of lines L1-L8 during one frame period.

[0038] If no flicker occurs, the drive unit 191 in Figure 3 sequentially outputs high-level pulse reset signals RST (RST1-RST8). The reset signals RST are generated so as not to overlap between lines L1-L8. After receiving the reset signal RST, the storage node of the pixel cell PX is reset and then stores the charge from the photodiode.

[0039] Furthermore, the drive unit 191 sets the selection signals SEL (SEL1-SEL8) to a high level for a predetermined period after the exposure time EXPT has elapsed from the output of the reset signal RST. The selection signals SEL are generated so as not to overlap between lines L1-L8. The readout unit 192 in Figure 3 amplifies the output voltage output from the corresponding pixel cell PX in response to the high-level selection signal SEL and generates readout data RD.

[0040] The imaging device 19 then outputs the sequentially generated readout data RD as image data to the image processing device 10 in Figure 2. For example, the image data may be output from the imaging device 19 to the image processing device 10 for each frame, which is the image displayed on the screen of the display device 12 in Figure 2. The image processing device 10 performs image processing on the image data received from the imaging device 19.

[0041] Figure 7 shows an example of the operation of the imaging device 19 when the flicker detection unit 194 in Figure 3 detects flicker. Detailed explanations of elements similar to those in Figure 6 are omitted. Figure 7 shows an example of the operation when flicker is detected in the images of lines L3 and L4 in the image IMG of Figure 5.

[0042] Based on the control by the exposure adjustment unit 195, the drive unit 191 in Figure 3 increases the number of times it generates the reset signal RST and the selection signal SEL corresponding to the lines L2 and L3 where flicker is detected, compared to the number of times shown in Figure 6. The reset signal RST is generated so as not to overlap with each other, and the selection signal SEL is generated so as not to overlap with each other.

[0043] As a result, readout data RD from lines L2 and L3 is added, as shown by the shaded rectangles in Figure 7. The readout unit 192 synthesizes multiple readout data RD from each of lines L2 and L3 each time the pixel cell array 190 is driven by the drive unit 191 and outputs it as image data to the image processing device 10. In other words, within one frame period, the exposure time of lines L2 and L3 is increased compared to the exposure time of other lines by increasing the number of exposures. By increasing the exposure time, flicker occurring in lines L2 and L3 can be locally suppressed.

[0044] Furthermore, the brightness of the images of lines L2 and L3, where the exposure time has been increased, will be higher than the brightness of the images of the other lines. For this reason, the imaging device 19 may perform a gain adjustment process that reduces the pixel values ​​of the image data of lines L2 and L3 in proportion to the increase in exposure time of lines L2 and L3.

[0045] Figure 8 shows an example of the operation of the imaging device 19 shown in Figure 3. That is, Figure 8 shows an example of the image processing method of the imaging device 19. For example, the operation shown in Figure 8 may be implemented by a control program executed by a CPU mounted on the imaging device 19, or by hardware mounted on the imaging device 19. The operation shown in Figure 8 starts, for example, when the imaging device 19 is powered on.

[0046] First, in step S100, the operation control unit 196 of the imaging device 19 controls the drive unit 191 and the readout unit 192 to acquire image data frame by frame. Next, in step S102, the flicker detection unit 194 detects the presence or absence of flicker by comparing the acquired image data with, for example, image data from at least one frame prior.

[0047] If the flicker detection unit 194 detects flicker, the motion control unit 196 performs steps S104, S106, and S108 in parallel with steps S110 and S112. If the flicker detection unit 194 does not detect flicker, the motion control unit 196 performs step S116. In addition, in the processing of step S102 for the first predetermined frame after the imaging device 19 is started up, the motion control unit 196 may determine that no flicker has been detected and perform step S116.

[0048] In step S104, the operation control unit 196 selects additional lines to be exposed. In the examples shown in Figures 5 and 7, the additional lines are L3 and L4. Next, in step S106, the operation control unit 196 performs additional exposure on the additional lines. Then, in step S108, the operation control unit 196 performs additional readout on the additional lines and then executes step S114.

[0049] In step S110, the operation control unit 196 performs normal exposure. Next, in step S112, the operation control unit 196 performs normal readout and then executes step S114. As shown in Figure 7, the operations in steps S110 and S112 are performed using a rolling shutter method, and the operations in steps S106 and S108 are performed in parallel with the operations in steps S110 and S112.

[0050] In step S114, the motion control unit 196 synthesizes the image data obtained from the additional readout and the normal readout to generate image data for one frame. For example, the motion control unit 196 generates image data by adding the pixel values ​​of the image data obtained from the additional readout and the pixel values ​​of the image data obtained from the normal readout.

[0051] The motion control unit 196 may perform processing such as averaging of pixel values ​​instead of adding pixel values. Furthermore, the motion control unit 196 may perform pixel value addition or averaging only in the vicinity of the area where flicker is detected.

[0052] On the other hand, if no flicker is detected, in step S116, the motion control unit 196 performs normal exposure using a rolling shutter method, as shown in Figure 6. Next, in step S118, the motion control unit 196 performs normal readout of pixel information from the pixel cell array 190 to generate a single frame image, and then executes step S120.

[0053] In step S120, the operation control unit 196 outputs the image data of one frame generated in step S114 or step S118 to the image processing device 10 or the like. Next, in step S122, the operation control unit 196 determines whether or not to terminate the image acquisition process. If the operation control unit 196 terminates the image acquisition process, it terminates the operation shown in Figure 8. If the operation control unit 196 continues the image acquisition process, it performs the operation in step S100.

[0054] Furthermore, the image data synthesis process in step S114, which is performed when flicker is detected, may be performed by the image processing device 10. In this case, when flicker is detected, the operation control unit 196 outputs the image data obtained from the additional readout, along with the line number L, to the image processing device 10 in step S120, in addition to the image data obtained from the normal readout. Furthermore, the process shown in Figure 8 may also be performed by the image processing device 10.

[0055] In this embodiment, based on the detection of flicker, image processing to suppress flicker can be performed locally on the line where flicker occurs. This suppresses side effects such as blurring of images of moving subjects, compared to, for example, suppressing flicker by increasing the exposure time across the entire pixel cell array 190. For example, in the image IMG shown in Figure 5, the flicker suppression process can suppress the occurrence of blurring of a vehicle traveling towards a traffic light.

[0056] Furthermore, by locally performing image processing to suppress flicker, the image processing load on the motion control unit 196 can be reduced compared to performing flicker suppression image processing on the entire image data of one frame. In addition, since the imaging device 19 performs image processing to reduce flicker, the increase in the image processing load on the image processing device 10 can be suppressed. Moreover, the image processing system 100 can be constructed using an existing image processing device 10.

[0057] (Second embodiment) Figure 9 shows an example of the configuration of an imaging device included in the image processing system of the second embodiment. Elements similar to those in the embodiments described above are denoted by the same reference numerals, and detailed descriptions are omitted. In the image processing system 100, the configuration is the same as in Figures 1, 2, and 4, except for the imaging device 19A. For example, of the imaging devices 19a-19e in Figure 1, at least imaging device 19e is replaced by imaging device 19A shown in Figure 9.

[0058] The imaging device 19A has an operation control unit 196A instead of the operation control unit 196 in Figure 3, and does not have an exposure adjustment unit 195. The imaging device 19A has a readout unit 192A instead of the readout unit 192 in Figure 3. The other configurations of the imaging device 19A are the same as those of the imaging device 19 in Figure 3.

[0059] The imaging device 19A has the function of acquiring images using a rolling shutter method. However, the imaging device 19A has the function of setting the exposure process with an exposure time EXPT to be executed multiple times, for example twice, within one frame, compared to the normal rolling shutter method shown in Figure 6. The operation control unit 196A causes the readout unit 192A to select pixel information to be acquired with one exposure time EXTP in pixel sequences (X) where the flicker detection unit 194A does not detect the occurrence of flicker. The operation control unit 196A causes the readout unit 192A to select pixel information to be acquired with two exposure times EXTP in pixel sequences (X) where the flicker detection unit 194A has detected the occurrence of flicker. Note that the exposure times may be different in the exposure processes that are executed multiple times within one frame.

[0060] Figure 10 shows an example of the operation of the imaging device 19A in Figure 9. Detailed explanations of operations similar to those in Figure 7 are omitted. The operation control unit 196A of the imaging device 19A performs control to generate two pairs of reset signals RST and selection signals SEL for each line L1-L8. Since each line L1-L8 is driven multiple times (twice) in one frame period, the readout unit 192 receives pixel information from each line L1-L8 multiple times in one frame period. For this reason, the pixel value of each pixel cell PX included in the readout data RD acquired in one frame period is twice the pixel value acquired in one frame period using the normal rolling shutter method shown in Figure 6.

[0061] Furthermore, the generation cycles of the reset signal RST and the selection signal SEL in each line L1-L8 are set to half the duration of one frame. This allows the exposure interval of the pixel cell PX in each line L1-L8 to be set to an equal and maximum possible interval. Therefore, the generation of flicker can be suppressed by combining the readout data (pixel values) as explained in Figure 11. In the readout data RD shown in Figure 10, the open rectangles represent readout data RD with normal exposure, and the shaded rectangles represent readout data RD with additional exposure.

[0062] Figure 11 shows an example of the operation of the imaging device 19A in Figure 9. That is, Figure 11 shows an example of the image processing method of the imaging device 19A. For operations similar to those in Figure 8, the same step numbers are used, and detailed explanations are omitted. For example, the operation shown in Figure 11 may be implemented by a control program executed by a CPU mounted on the imaging device 19, or by hardware mounted on the imaging device 19. The operation shown in Figure 8 starts, for example, when the imaging device 19 is started up by powering it on.

[0063] In this embodiment, first, in steps S106 and S110, the operation control unit 196A performs additional exposure and normal exposure in parallel (alternating). Next, in steps S108 and S112, the operation control unit 196A performs additional reading and normal reading in parallel (alternating). That is, the operation control unit 196 performs the operation shown in Figure 10 in steps S106, S108, S110, and S112.

[0064] Next, in step S102A, the flicker detection unit 194 detects whether or not flicker occurs for each line by comparing the image data acquired in steps S108 and S112 with, for example, the image data from at least one frame prior. The motion control unit 196A performs step S104A for the lines in which the flicker detection unit 194 has detected flicker. The motion control unit 196A performs step S120 for the lines in which the flicker detection unit 194 has not detected flicker.

[0065] In other words, the motion control unit 196A does not perform image data synthesis in step S114A for lines in which no flicker is detected. The motion control unit 196A then outputs image data obtained by normal readout, which is part of the pixel information that the readout unit 192A receives multiple times from the pixel cell array 190 for each line. In the processing of step S102A in the first predetermined frame after the imaging device 19 is started up, the motion control unit 196 may determine that no flicker is detected in any of the pixel sequences (X) and perform step S120.

[0066] In step S104A, the motion control unit 196A selects the line of the pixel sequence (X) where flicker was detected. For example, in the example shown in Figure 5, lines L3 and L4 are selected. The motion control unit 196A may perform a gain adjustment process in the selected line to reduce the pixel value in proportion to the increase in additional exposure time. Alternatively, the motion control unit 196A may perform a process such as averaging the pixel values ​​instead of adding the pixel values ​​in the selected line. Furthermore, the motion control unit 196 may perform pixel value addition or averaging only in the vicinity of the area where flicker was detected.

[0067] Next, in step S114A, the operation control unit 196A causes the readout unit 192A to combine the image data acquired during normal exposure and additional exposure for the line L selected in step S104A, and generates image data for two exposure times EXTP. Doubling the exposure time EXTP can suppress the occurrence of flicker. After step S114A, the operation control unit 196A performs step S120.

[0068] In step S120, the motion control unit 196A causes the image processing device 10 or the like to output the image data for one frame generated by the readout unit 192A. The motion control unit 196A also causes the readout unit 192A to generate image data (image data for one exposure time EXTP) for pixel sequences (X) in which no flicker is detected.

[0069] Next, in step S122, the operation control unit 196A determines whether or not to terminate the image acquisition process. If the operation control unit 196A terminates the image acquisition process, it terminates the operation shown in Figure 11. If the operation control unit 196A continues the image acquisition process, it performs the operations in steps S106 and S110.

[0070] Note that steps S102A, S104A, and S114A may be performed by the image processing device 10. In this case, the motion control unit 196A does not perform flicker detection. The motion control unit 196A and the readout unit 192A are then instructed to combine the image data acquired for all pixel sequences (X) with normal exposure and additional exposure, respectively, to generate image data for two exposure times EXTP, and then perform the process in step S120.

[0071] As described above, the same effects as those of the embodiments described can be obtained in this embodiment as well. For example, based on the detection of flicker, image processing to suppress flicker can be performed locally on the line where flicker occurs. This makes it possible to suppress side effects such as blurring of images of moving subjects compared to, for example, suppressing flicker by increasing the exposure time for the entire pixel cell array 190.

[0072] Furthermore, in this embodiment, the motion control unit 196A always performs the operation shown in Figure 10, regardless of whether or not the flicker detection unit 194A detects flicker. In other words, the motion control unit 196A does not need to switch between the operations shown in Figures 6 and 7 depending on whether or not the flicker detection unit 194A detects flicker. This makes the configuration of the motion control unit 196A simpler than that of the motion control unit 196 in Figure 3. For example, the exposure adjustment unit 195 in Figure 3 can be made unnecessary.

[0073] (Third embodiment) Figure 12 shows an example of the configuration of an imaging device included in the image processing system of the third embodiment. Elements similar to those in the embodiments described above are denoted by the same reference numerals, and detailed descriptions are omitted. In the image processing system 100, the configuration is the same as in Figures 1, 2, and 4, except for the imaging device 19B. For example, of the imaging devices 19a-19e in Figure 1, at least imaging device 19e is replaced by imaging device 19B shown in Figure 12.

[0074] The imaging device 19B has an operation control unit 196B instead of the operation control unit 196 in Figure 3, and does not have a flicker detection unit 194 and an exposure adjustment unit 195. The imaging device 19B has a readout unit 192B instead of the readout unit 192 in Figure 3. The imaging device 19B also has an image synthesis unit 197B between the readout unit 192B and the bus 193. The other configurations of the imaging device 19B are the same as those of the imaging device 19 in Figure 3.

[0075] Similar to the imaging device 19A in Figure 9, the imaging device 19B has a function to set the exposure process with exposure time EXPT to be executed multiple times, for example twice, within one frame, compared to the normal rolling shutter method shown in Figure 6. The operation control unit 196B controls the overall operation of the imaging device 19B, including the drive unit 191 and the image synthesis unit 197B, and operates the imaging device 19B at the same timing as in Figure 10. Note that the exposure times for each of the multiple exposure processes executed within one frame may be different.

[0076] As shown in Figure 10, the readout unit 192B alternately acquires readout data RD from normal exposure and readout data from additional exposure, and outputs the acquired readout data RD to the image synthesis unit 197B. The image synthesis unit 197B uses the two types of readout data RD from the readout unit 192B to perform pixel value synthesis processing for each pixel. The image synthesis unit 197B outputs the image data generated by synthesizing the pixel values ​​to the image processing device 10 via a bus. The pixel value synthesis processing is explained in Figure 13.

[0077] Figure 13 shows an example of the image synthesis unit 197B shown in Figure 12. The image synthesis unit 197B includes a difference calculation unit 41, a sum value calculation unit 42, a synthesis gain calculation unit 43, a blend ratio calculation unit 44, and a blend processing unit 45. For example, the image synthesis unit 197B has multiple configurations as shown in Figure 13, and performs the synthesis processing of pixel values ​​of multiple pixels in parallel. The difference calculation unit 41, the sum value calculation unit 42, and the blend ratio calculation unit 44 are examples of flicker detection units that detect flicker regions where flicker occurs based on multiple image data with different acquisition timings. The blend processing unit 45 is an example of a synthesis processing unit that synthesizes multiple image data with different acquisition timings in the flicker region.

[0078] In Figure 13, the symbol F[n] represents the pixel value for each pixel under normal exposure, and the symbol F[n+1] represents the pixel value for each pixel under additional exposure. Normal exposure and additional exposure are the same as those described in Figure 10. Hereafter, the symbol F[n] will be referred to as the pixel value F[n] or normal pixel value F[n], and the symbol F[n+1] will be referred to as the pixel value F[n+1] or additional pixel value F[n+1].

[0079] The difference calculation unit 41 calculates the difference DIF as the absolute value (abs) of the difference between the pixel value F[n] and the value obtained by multiplying the pixel value F[n+1] by the exposure ratio D. Note that pixels with a larger difference DIF are more likely to be images of moving objects.

[0080] The exposure ratio D is the ratio NEXPT / AEXPT between the normal exposure time NEXPT and the additional exposure time AEXPT. In Figure 10, the exposure ratio D(NEXPT / AEXPT) is "1". By using the exposure ratio D, the differential DIF can be calculated by giving the additional exposure time AEXPT the same weight as the normal exposure time NEXPT.

[0081] For example, if the additional exposure time AEXPT per frame is equal to the normal exposure time NEXPT per frame, the weight of the additional exposure time AEXPT will be the same as that of the normal exposure time NEXPT because the exposure ratio D = 1. If the additional exposure time AEXPT per frame is twice that of the normal exposure time NEXPT per frame, the weight of the additional exposure time AEXPT will be set to half that of the normal exposure time NEXPT because the exposure ratio D = 0.5.

[0082] The total value calculation unit 42 calculates the sum of pixel values ​​F[n] and F[n+1] as the total value M. The composite gain calculation unit 43 calculates the composite gain K as the ratio of the normal exposure time NEXPT to the sum of the normal exposure time NEXPT and the additional exposure time AEXPT, NEXPT / (NEXPT+AEXPT). ​​In Figure 10, the composite gain K is "0.5".

[0083] For example, if the additional exposure time AEXPT per frame is equal to the normal exposure time NEXPT per frame, the pixel value (luminance) doubles due to the addition of pixel values ​​F[n] and F[n+1]. In this case, the composite gain K is set to "0.5", and the pixel value F[n2] that increases due to the addition is converted to the pixel value as it would be without the addition.

[0084] If the additional exposure time AEXPT per frame is three times the normal exposure time NEXPT per frame, the pixel value (luminance) becomes four times greater due to the addition of pixel values ​​F[n] and F[n+1]. In this case, the composite gain K is set to "0.25", and the pixel value F[n2] that increases due to the addition is converted to the pixel value as it would be without the addition.

[0085] The blend ratio calculation unit 44 uses equation (1) to calculate the blend ratio α of pixel values ​​F[n] and F[n+1] based on the difference DIF and the sum value M. α = 2 - DIFr * Mr …(1)

[0086] In equation (1), the ratio DIFr is "0" when the difference DIF is less than or equal to the threshold VT1, and "1" when the difference DIF is greater than or equal to the threshold VT2. The ratio DIFr gradually increases from "0" to "1" when the difference DIF is between the threshold VT1 and the threshold VT2. In equation (1), the ratio Mr is "0" when the sum M is less than or equal to the threshold VT3, and "1" when the sum M is greater than or equal to the threshold VT4. The ratio Mr gradually increases from "0" to "1" when the sum M is between the threshold VT3 and the threshold VT4.

[0087] As a result, the blend ratio α is "2" when the difference DIF is less than or equal to the threshold VT1 or the sum M is less than or equal to the threshold VT3. The blend ratio α is "1" when the difference DIF is greater than or equal to the threshold VT2 and the sum M is greater than or equal to the threshold VT4. Otherwise, the blend ratio α is greater than "1" and less than "2".

[0088] The blending processing unit 45 uses equation (2) to blend, for example, the normal pixel value F[n] and the additional pixel value F[n+1] pixel by pixel to generate the pixel value F[n2]. In equation (2), the sign * indicates the multiplication sign. F[n2]=F[n]*α*K+F[n+1]*(2-α)*K …(2)

[0089] From equation (2), the larger the blending ratio α, the larger the ratio of the normal pixel value F[n] to the blended pixel value F[n2] becomes. The smaller the blending ratio α, the smaller the ratio of the normal pixel value F[n] to the blended pixel value F[n2] becomes.

[0090] Figure 14 shows an example of the operation of the image synthesis unit 197B shown in Figure 13. In the operation shown in Figure 14, the image data IMG shown in Figure 5 is used. The image shown F[n] is the image generated when the normal pixel value F[n] is used. The image shown F[n] is also called the normal image.

[0091] The image shown for F[n+1] represents the image generated when the additional pixel value F[n+1] is used. The image shown for F[n+1] is also called the additional image. The image shown for F[n]+F[n+1] represents the image generated when the normal pixel value F[n] and the additional pixel value F[n+1] are added together. The image shown for F[n]+F[n+1] is also called the composite image.

[0092] In traffic light images, flicker can cause the red light to be off in either the normal image or the supplementary image. In the composite image, the red light remains lit. Therefore, for images taken under conditions where flicker is likely to occur, it is preferable to use the composite image indicated by the asterisk. Furthermore, in the traffic light image, both the pixel values ​​of the sum M and the pixel values ​​of the difference DIF are larger in the area of ​​the red light and smaller in areas other than the red light.

[0093] Therefore, it can be seen that flicker is more likely to occur when both the pixel value of the total value M and the pixel value of the difference value DIF are large. Accordingly, the blend ratio calculation unit 44 in Figure 3 determines that the pixel region where the difference DIF is greater than or equal to the threshold VT2 and the total value M is greater than or equal to the threshold VT4 is a flicker region where flicker has occurred. From the above, the ideal blend ratio α is "1.0".

[0094] Images of moving objects (vehicles) are normal in both the normal and additional images, but blurring (afterimages) due to vehicle movement may occur in the composite image. For this reason, for example, it is preferable to use the normal image indicated by the star for images of moving objects. Also, in images of moving objects, the pixel values ​​of the total value M are small in all areas. The pixel values ​​of the difference value DIF are large in the vehicle area and small in areas other than the vehicle. Therefore, it can be seen that image blurring due to moving objects is more likely to occur when the pixel values ​​of the total value M are small and the pixel values ​​of the difference value DIF are large. Accordingly, the blend ratio calculation unit 44 determines that the pixel area in which the difference DIF is greater than or equal to the threshold VT2 and the total value M is less than or equal to the threshold VT3 is the moving object area. From the above, the ideal blend ratio α is "2.0".

[0095] Images of stationary objects are normal in all types of images: normal images, supplemental images, and composite images. Furthermore, in images of stationary objects, the pixel values ​​of the sum M are small in all regions, and the pixel values ​​of the difference DIF are small in all regions. For this reason, any of the normal images, supplemental images, or composite images may be used for stationary objects. Here, it is preferred that the normal image, indicated by the asterisk, be used for images of stationary objects. In addition, the ideal blend ratio α for images of stationary objects is set to 2.0, but any blend ratio α is acceptable. Therefore, the blend ratio calculation unit 44 determines that pixel regions where the difference DIF is less than or equal to the threshold VT1 and the sum M is less than or equal to the threshold VT3 are stationary regions containing stationary objects.

[0096] The image synthesis unit 197B shown in Figure 13 generates a pixel value F[n2] by combining the normal pixel value F[n] and the additional pixel value F[n+1], thereby generating image data in which flicker and motion blur are suppressed. In this case, image data in which flicker and motion blur are suppressed can be generated without using a flicker detection unit or a motion detection unit. Since the generation of the pixel value F[n2] is performed on a pixel-by-pixel basis, even when the image data acquired on a single line includes traffic lights, moving objects, and stationary objects, as shown in Figure 5, synthesis processing appropriate for each image can be performed.

[0097] For example, the image synthesis unit 197B sets the blend ratio α to "1.0" and blends the normal pixel value F[n] with the additional pixel value F[n+1] for pixels with a large difference DIF and a large total value M, as these pixels are likely to be moving objects. The image synthesis unit 197B sets the blend ratio α to "2.0" and outputs the normal pixel value F[n] as the blended pixel value F[n2] for pixels with a large difference DIF and a total value M below normal, as these pixels are likely to be moving objects. The image synthesis unit 197B also considers pixels with a small difference DIF and a total value M below normal to be likely to be stationary objects (normal subjects). In this case, the blend ratio α may be set to either "1.0" or "2.0". However, to reduce the computational load of the blend processing unit 45 and minimize calculation errors, it is preferable to set the blend ratio α to "2.0".

[0098] Furthermore, since the image synthesis unit 197B performs synthesis processing on a pixel-by-pixel basis, it can locally process images with flicker and images with blur. In other words, it can suppress the degradation of image quality in areas of the image without flicker and areas without blur due to the effects of the synthesis processing.

[0099] Although Figures 12 and 13 illustrate an example in which the image synthesis unit 197B is provided on the imaging device 19B, the image synthesis unit 197B may also be provided on the image processing device 10. In this case, the imaging device 19B outputs the normal pixel value F[n] and the additional pixel value F[n+1] as image data to the image processing device 10.

[0100] As described above, the same effects as those of the embodiments described can be obtained in this embodiment as well. Furthermore, in this embodiment, the normal pixel value F[n] and the additional pixel value F[n+1] can be combined on a pixel-by-pixel basis to generate the pixel value F[n2]. As a result, as shown in Figure 5, even when the image data acquired in a single line includes traffic lights, moving objects, and stationary objects, a synthesis process suitable for each image can be performed.

[0101] Furthermore, the image synthesis unit 197B shown in Figure 13 synthesizes the normal pixel value F[n] and the additional pixel value F[n+1] to generate the pixel value F[n2], thereby generating image data in which flicker and motion blur are suppressed. In this case, image data in which flicker and motion blur are suppressed can be generated without using the flicker detection unit and the motion detection unit.

[0102] (Fourth embodiment) Figure 15 shows an example of the configuration of an imaging device included in the image processing system of the fourth embodiment. Elements similar to those in the embodiments described above are denoted by the same reference numerals, and detailed descriptions are omitted. In the image processing system 100, the configuration is the same as in Figures 1, 2, and 4, except that an image processing device 10C is provided instead of the image processing device 10. For example, of the imaging devices 19a-19e in Figure 1, at least imaging device 19e is replaced with imaging device 19C shown in Figure 15.

[0103] The imaging device 19C has an operation control unit 196C instead of the operation control unit 196B in Figure 12. Furthermore, the imaging device 19C does not have the image synthesis unit 197B in Figure 12. The other configurations of the imaging device 19A are the same as those of the imaging device 19 in Figure 3 and the imaging device 19B in Figure 12. That is, the imaging device 19C has the function of acquiring images using a rolling shutter method.

[0104] The image processing device 10C includes an acquisition unit 51, an image synthesis unit 52C, and a switching control unit 53C. The acquisition unit 51 acquires image data output from the imaging device 19C frame by frame. The image synthesis unit 52C performs image data synthesis processing using the image data acquired by the acquisition unit 51. The switching control unit 53C outputs a switching signal to the imaging device 19C to switch the frame rate of the imaging device 19C.

[0105] The motion control unit 196C controls the drive unit 191 and the readout unit 192 to acquire image data frame by frame. The motion control unit 196C also switches the frame rate of the imaging device 19C according to the switching signal from the switching control unit 53C.

[0106] Figure 16 shows an example of the image synthesis unit 52C in Figure 15. Elements similar to those in Figure 13 are given the same reference numerals, and detailed explanations are omitted.

[0107] The image synthesis unit 52C includes a difference calculation unit 41C, a sum value calculation unit 42, a synthesis gain calculation unit 43C, a blend ratio calculation unit 44C, and a blend processing unit 45. The configuration and function of the sum value calculation unit 42 and the blend processing unit 45 are the same as those of the sum value calculation unit 42 and the blend processing unit 45 in Figure 16. The image synthesis unit 52C has multiple configurations as shown in Figure 16 and performs the synthesis processing of pixel values ​​of multiple pixels in parallel. The difference calculation unit 41C, the sum value calculation unit 42, and the blend ratio calculation unit 44C are examples of flicker detection units that detect flicker regions where flicker occurs based on image data of multiple frames.

[0108] The difference calculation unit 41C calculates the absolute value (abs) of the difference between pixel value F[n] and pixel value F[n+1] as the difference DIF. The composite gain calculation unit 43C calculates the composite gain K according to the brightness of the scene in the subject area captured by the imaging device 19C. The method for calculating the composite gain K is explained in Figure 17. For example, the brightness of the scene may be measured by an illuminance meter mounted on the mobile body 200 or the imaging device 19C.

[0109] The blend ratio calculation unit 44C uses equation (3) to calculate the blend ratio α of the pixel values ​​of frames F[n] and F[n+1] based on the difference DIF and the sum M. The ratio DIFr and ratio Mr are determined in the same way as in Figure 13. The sign MIN indicates that the smaller of "2-DIFr*Mr" and "1 / K" is selected. Equation (3) adds a condition to equation (1) that clips the blend ratio α so that it does not exceed "1 / K". That is, the minimum value of the blend ratio α is clipped at "1 / K". α = MIN(2 - DIFr*Mr, 1 / K) …(3)

[0110] The blending processing unit 45 uses the blending ratio α calculated by the blending ratio calculation unit 44C and the above-described equation (2) to blend, for example, the pixel values ​​of adjacent odd-numbered frames F[n] and even-numbered frames F[n+1] pixel by pixel. The blending processing unit 45 then generates the pixel value F[n2].

[0111] Figure 17 shows the relationship between the composite gain K calculated by the composite gain calculation unit 43C in Figure 16, the scene brightness, and the frame rate. When the scene brightness measured by an illuminometer or the like is brighter than a predetermined threshold, the image processing device 10C outputs a switching signal to the imaging device 19C to switch the frame rate from 60fps (frames per second) to 120fps. The imaging device 19C switches the frame rate from 60fps to 120fps based on the switching signal.

[0112] For example, when the frame rate is 60fps (approximately 16.7ms cycle), the imaging device 19C sets the exposure time to 11ms or more depending on the brightness of the scene. For example, the imaging device 19C uses its built-in AE (Auto Exposure) function to determine the exposure time. Also, when the frame rate is 120fps (approximately 8.3ms cycle), the imaging device 19C fixes the exposure time to 5.5ms.

[0113] Note that the minimum exposure time at a frame rate of 60fps and the minimum exposure time at a frame rate of 120fps are not limited to 11ms and 5.5ms. The minimum exposure time at a frame rate of 60fps is sufficient if it is twice the exposure time at a frame rate of 120fps and is effective in suppressing flicker.

[0114] Images captured with an exposure time of 11ms can suppress flicker from LED light sources, but there is a risk of motion blur (afterimages). On the other hand, images captured with an exposure time of 5.5ms will produce flicker from LED light sources, but motion blur (afterimages) can be reduced.

[0115] The composite gain calculation unit 43C does not calculate the composite gain K during periods when the frame rate is 60fps, and outputs "1" to indicate, for example, that no gain processing is performed. In this case, the other elements of the image composite unit 52C in Figure 16 also stop operating. Then, the respective pixel values ​​of frames F[n] and F[n+1] of the image captured with an exposure time of 11ms or more are output as the pixel value F[n2].

[0116] The composite gain calculation unit 43C suppresses pixel saturation by gradually lowering the composite gain K from "1.0" to "0.5" as the scene brightens during the period when the frame rate is 120fps. Furthermore, by setting the composite gain K to "1.0" when the frame rate is switched, it is possible to suppress the change in brightness due to the exposure time switch when the blend processing unit 45 composites images captured with an exposure time of 5.5ms.

[0117] For example, if the brightness of the scene slightly exceeds the threshold and the frame rate switches from 60fps to 120fps, the composite gain K is approximately "1" as shown in Figure 17. At this time, assume that the ratio DIFr is large and the ratio Mr is small in the moving area of ​​the image. In this case, the blend ratio calculation unit 44 in Figure 13 sets the blend ratio α to "2", but the blend ratio calculation unit 44C in this embodiment sets the blend ratio α to "1" because "1 / K" is clipped to "1".

[0118] The blending processing unit 45 outputs a pixel value F[n2], which is the sum of the pixel values ​​for each pixel in the moving area of ​​frames F[n] and F[n+1], based on the blending ratio α="1". The total exposure time for frames F[n] and F[n+1] when the frame rate is 120fps is 11ms, the same as when the frame rate is 60fps. Therefore, the afterimage of the moving area before and after the frame rate switch is about the same, and the motion control unit 196C can output an image (F[n2]) with less noticeable distortion.

[0119] In contrast, under the same conditions, if there is no clipping by "1 / K", the blend ratio α becomes approximately "2", so the blend processing unit 45 doubles the pixel value of each pixel in the moving area of ​​frame F[n] and outputs it as the pixel value F[n2]. The exposure time of frame F[n] is 5.5ms, which is half the exposure time of frame F[n] before the frame rate switch (11ms). Therefore, after the frame rate switch, the afterimage of the moving area of ​​the image suddenly decreases, and there is a risk that an unnatural image (F[n2]) will be output.

[0120] Furthermore, if the scene brightness is sufficiently greater than the threshold, "1 / K" becomes approximately "2". Therefore, the blend ratio α calculated by the blend ratio calculation unit 44C changes appropriately between "1.0" and "2.0" depending on the image characteristics, based on "2-DIFr*Mr" in equation (2). As the scene brightens, the composite gain K changes gradually, so any unnaturalness caused by a sudden switch in the composite gain K is suppressed.

[0121] Figure 18 shows an example of the operation of the image processing device 10C shown in Figure 16. That is, Figure 18 shows an example of the image processing method of the image processing device 10C. For example, the operation shown in Figure 18 may be realized by an image processing program executed by the CPU 20 (Figure 4) mounted on the image processing device 10C, or by hardware mounted on the image processing device 10C. The operation shown in Figure 18 starts, for example, when the image processing device 10C is started up by powering on.

[0122] First, in step S200, the image processing device 10C determines whether the brightness of the scene in the subject area captured by the imaging device 19C is less than a threshold. If the brightness is less than the threshold, the image processing device 10C performs step S202; if the brightness is greater than the threshold, it performs step S206.

[0123] In step S202, the image processing device 10C outputs a switching signal to the imaging device 19C to set the frame rate to 60fps. Based on the switching signal, the imaging device 19C sets the exposure time for one frame to 11ms or more according to the brightness of the scene. Next, in step S204, the image processing device 10C acquires one frame of image data from the imaging device 19C. Next, in step S206, the image processing device 10C outputs the one frame of image data generated in step S204 to the display device 12 or the like, and then performs step S214.

[0124] Meanwhile, in step S208, the image processing device 10C outputs a switching signal to the imaging device 19C to set the frame rate to 120fps. Based on the switching signal, the imaging device 19C sets the exposure time for one frame to 5.5ms regardless of the brightness of the scene. Next, in step S210, the image processing device 10C acquires two consecutive frames F[n] and F[n+1] image data from the imaging device 19C. Next, in step S212, the image processing device 10C combines the image data of the two consecutive frames F[n] and F[n+1] acquired from the imaging device 19C and outputs it to the display device 12 or the like at 60fps. After step S212, step S214 is performed.

[0125] In step S214, the image processing device 10C determines whether to terminate the image data acquisition process and the image data output process. If the operation control unit 196 terminates the image data acquisition process and the image data output process, it terminates the operation shown in Figure 18. If the operation control unit 196 continues the image data acquisition process and the image data output process, it performs the operation in step S200.

[0126] Figure 19 shows an example of the operation of the image synthesis unit 52C in Figure 16. Detailed explanations of elements similar to those in Figure 14 are omitted. In the operation shown in Figure 19, the image data shown in Figure 5 is used. In Figure 19, the image shown F[n] is also called the previous image. The image shown F[n+1] is also called the next image. The image shown F[n]+F[n+1] is also called the composite image.

[0127] The states of the previous image F[n], the subsequent image F[n+1], and the composite image F[n]+F[n+1] are the same as those of the normal image F[n], additional image F[n+1], and composite image F[n]+F[n+1] shown in Figure 14, except that the frames are different images. Therefore, the characteristics of the total value M, the characteristics of the difference DIF, and the preferred image to use, indicated by the asterisk, are the same as in Figure 14.

[0128] However, in this embodiment, in image regions (moving objects) where the difference DIF is large and the total value M is small between the previous image F[n] and the subsequent image F[n+1], it is preferable to set the blend ratio α to "1 / K". Also, in image regions (stationary objects) where both the difference DIF and the total value M are small between the previous image F[n] and the subsequent image F[n+1], it is preferable to set the blend ratio α to "1 / K".

[0129] The image synthesis unit 52C shown in Figure 16 may also be provided in the imaging device 19C. In this case, the imaging device 19C performs the synthesis of the images shown in Figures 17 and 18, and the synthesized image is output from the imaging device 19C to the image processing device 10C or the like.

[0130] As described above, the same effects as those of the embodiments described can be obtained in this embodiment as well. Furthermore, in this embodiment, the image synthesis unit 52C provided in the image processing device 10C is used to perform image synthesis processing within the frame. As a result, as shown in Figure 5, even when the image data acquired in a single line includes traffic lights, moving objects, and stationary objects, synthesis processing suitable for each image can be performed.

[0131] Furthermore, during periods when the frame rate is 120fps, by gradually lowering the composite gain K from 1.0 to 0.5 as the scene becomes brighter, it is possible to suppress pixel saturation even when combining two frames at a frame rate of 120fps. When the scene is not bright, setting the frame rate to 60fps can suppress the generation of noise. In addition, since the imaging device 19C does not have an image synthesis unit, it is possible to suppress flicker for each pixel of the image and suppress motion blur by using a normal imaging device 19C.

[0132] (Fifth embodiment) Figure 20 shows an example of an image synthesis unit 52D included in the image processing system of the fifth embodiment. Elements similar to those in the embodiments described above are denoted by the same reference numerals, and detailed descriptions are omitted. The image synthesis unit 52D is included in the image processing device 10D mounted in the image processing system 100 instead of the image processing device 10 in Figure 2. The image processing device 10D has the same configuration and functions as the image processing device 10C in Figure 15, except that it has an image synthesis unit 52D instead of the image synthesis unit 52C in the image processing device 10C.

[0133] The image processing system 100 is the same as in Figures 1, 2, and 4, except that an image processing device 10D is provided instead of the image processing device 10. In addition, of the imaging devices 19a-19e mounted on the mobile body 200 shown in Figure 1, at least imaging device 19e is replaced with imaging device 19D, which is not shown.

[0134] When the frame rate is set to 120 fps, the imaging device 19D outputs image data for odd lines in odd frames F[n] and image data for even lines in even frames F[n+1]. The other configurations and functions of the imaging device 19D are the same as those of the imaging device 19C shown in Figure 15.

[0135] When the frame rate is set to 120fps, the imaging device 19D may, for example, output image data of even lines in odd-numbered frames F[n] and image data of odd lines in even-numbered frames F[n+1]. An example of the image data for odd-numbered frames F[n] and even-numbered frames F[n+1] output by the imaging device 19D when the frame rate is set to 120fps is shown in Figure 21.

[0136] The image synthesis unit 52D includes a difference calculation unit 41C, a total value calculation unit 42, a synthesis gain calculation unit 43C, a blend ratio calculation unit 44D, a blend processing unit 45D, a frame reconstruction unit 46D, and vertical scaling units 47D and 48D. In other words, the image synthesis unit 52D has a blend ratio calculation unit 44D and a blend processing unit 45D instead of the blend ratio calculation unit 44C and the blend processing unit 45 of the image synthesis unit 52C in Figure 16.

[0137] Furthermore, the image synthesis unit 52D adds a frame reconstruction unit 46D and vertical magnification units 47D and 48D to the image synthesis unit 52C in Figure 16. The image synthesis unit 52D synthesizes image data of frames F[n] and F[n+1] that are downsampled and output from the imaging device 19D when the frame rate is 120fps. The difference calculation unit 41C, the total value calculation unit 42, and the blend ratio calculation unit 44D are examples of flicker detection units. The blend processing unit 45D is an example of a synthesis processing unit. The vertical magnification units 47D and 48D are examples of interpolated image generation units.

[0138] The frame reconstruction unit 46D uses frame F[n] containing image data for odd-numbered lines and frame F[n+1] containing image data for even-numbered lines to generate frames Fm and Fi, respectively, containing image data for all lines. Examples of the image data for frames Fm and Fi generated by the frame reconstruction unit 46D are shown in Figure 22. The image data for frame Fm is an example of the first reconstructed image data, and the image data for frame Fi is an example of the second reconstructed image data.

[0139] The vertical scaling unit 47D increases the number of lines in frame F[n], which contains image data for odd-numbered lines, to generate frame Fu[n], which contains image data for all lines. The vertical scaling unit 48D increases the number of lines in frame F[n+1], which contains image data for even-numbered lines, to generate frame Fu[n+1], which contains image data for all lines. Examples of the image data for frames Fu[n] and Fu[n+1] generated by vertical scaling units 47D and 48D, respectively, are shown in Figure 22. The image data for frame Fu[n] is an example of first interpolated image data, and the image data for frame Fu[n+1] is an example of second interpolated image data.

[0140] The blend ratio calculation unit 44D outputs the ratio DIFr and ratio Mr, which are used to calculate the blend ratio α in Figures 13 and 16, to the blend processing unit 45D instead of the blend ratio α. The blend processing unit 45D uses equations (4), (5), and (6) in sequence to calculate the pixel values ​​of frame F[n2] from the pixel values ​​of frame Fm and Fi.

[0141] Tm=(Fm+Fi)*Mr+2*Fm*(1-Mr) …(4) Td=2*Fu[n]*DIFr+Tm*(1-DIFr) …(5) F[2n]=K*Td …(6)

[0142] By sequentially using equations (4), (5), and (6), in pixel regions where ratios Mr and DIFr are close to 1, the sum of the pixel values ​​of frame Fm and Fi (Fm + Fi) is obtained as the pixel value of frame F[n2], thus suppressing the occurrence of flicker from LED light sources, etc., in the image. In pixel regions where ratio Mr is close to 0 and ratio DIFr is close to 1, twice the pixel value of frame F[n] is obtained as the pixel value of frame F[n2], thus suppressing blur (afterimage) of moving objects, etc., in the image. In pixel regions where both ratios Mr and DIFr are close to 0, twice the pixel value of frame Fm is obtained as the pixel value of frame F[n2], thus improving the resolution of stationary objects, etc., in the image. An example of the operation of the image synthesis unit 52D is explained in Figure 23.

[0143] Figure 21 shows an example of image data for odd-numbered frames F[n] and even-numbered frames F[n+1] output by the imaging device 19D when the frame rate is set to 120fps. For simplicity, it is assumed below that the imaging device 19D generates 8 lines (L1-L8) of image data for each frame.

[0144] The imaging device 19D outputs image data of odd lines L1, L3, ..., L7, indicated by diagonal lines, in odd-numbered frames F[n]. The imaging device 19D outputs image data of even lines L2, L4, ..., L8, indicated by diagonal lines, in even-numbered frames F[n+1]. Figure 21 shows an example where the pixel arrangement of the imaging device 19D is a Bayer array, with red pixels R, green pixels G, and blue pixels B arranged in a 1:2:1 ratio in a 2x2 grid. However, the pixel arrangement of the imaging device 19D is not limited to a Bayer array.

[0145] Figure 22 shows an example of the image data of frames Fm and Fi generated by the frame reconstruction unit 46D in Figure 20, and the image data of frames Fu[n] and Fu[n+1] generated by the vertical expansion units 47D and 48D, respectively.

[0146] The image data for frame Fm is generated by arranging the lines L1, L3, L5, L7 of the downsampled frame F[n] output from the imaging device 19D and the lines L2, L4, L6, L8 of frame F[n+1] alternately in this order. The image data for frame Fi is generated by arranging the lines L2, L4, L6, L8 of the downsampled frame F[n+1] output from the imaging device 19D and the lines L1, L3, L5, L7 of frame F[n] alternately in this order.

[0147] Furthermore, in frame Fm+Fi, which is obtained by adding the pixel values ​​of each pixel in each line of frame Fm and Fi, the image data of adjacent odd-numbered lines and even-numbered lines (for example, L1 and L2) will be the same.

[0148] The image data of frame Fu[n], obtained by vertically scaling the downsampled frame F[n], is interpolated using the pixel values ​​of adjacent odd lines to generate the pixel values ​​of even lines between odd lines. Additionally, the image data of the final line L8 of frame Fu[n] is generated by copying the pixel values ​​of line L7.

[0149] The image data of frame Fu[n+1], which is obtained by vertically scaling the downsampled frame F[n+1], is interpolated using the pixel values ​​of adjacent even lines to generate the pixel values ​​of odd lines between even lines. Additionally, the image data of the first line L1 of frame Fu[n+1] is generated by copying the image data of line L2.

[0150] Figure 23 shows an example of the operation of the image synthesis unit 52D in Figure 20. Detailed explanations of elements similar to those in Figures 14 and 19 are omitted. In the operation shown in Figure 23, the image data shown in Figure 5 is used. In Figure 23, the images shown as F[n] and F[n+1] are also called downsampled images. The characteristics of the sum M and the difference DIF are the same as in Figures 14 and 19.

[0151] The asterisks in Figure 23 indicate preferred methods for combining image data in images of traffic lights, vehicles (moving objects), and stationary objects, respectively. As explained in Figure 20, in images of traffic lights where both ratio Mr and DIFr are large, it is preferable to use the pixel values ​​of frame F[n2] as the pixel values ​​of frame F[n2], which are obtained by adding the image data of frame Fm and Fi pixel by pixel.

[0152] In images of vehicles (moving objects) with a small ratio Mr and a large ratio DIFr, it is preferable to set the pixel values ​​of frame F[n] to the pixel values ​​of frame F[n2] in order to suppress image blur (afterimage). In images of stationary objects with both ratio Mr and DIFr being small, it is preferable to set the pixel values ​​of frame Fm to the pixel values ​​of frame F[n2] in order to improve image resolution.

[0153] Figure 24 shows an example of low-speed output of image data from the imaging device 19D when the frame rate is set to 120fps. In Figure 24, the lines shown in solid frames represent image data output from the imaging device 19D. The lines shown in dashed frames represent no image data output from the imaging device 19D.

[0154] The section in brackets in Figure 24 shows an example of outputting image data from the imaging device 19D at normal speed. At normal speed, the image data of the odd lines of the odd frame F[n] is stored in memory such as the frame buffer, and then the image data of the even lines of the even frame F[n+1] is stored in memory such as the frame buffer. In other words, when outputting image data at normal speed, the image data of the odd frame F[n] stored in memory is used to combine the image data of the odd frame F[n] and the even frame F[n+1].

[0155] In contrast, when image data is output at a low speed, it becomes possible to output image data for even frames F[n+1] from the imaging device 19D while the image data for odd frames F[n] is being sequentially stored in memory. Therefore, it is possible to combine the image data of odd frames F[n] and even frames F[n+1] without storing all of the image data for odd frames F[n] in memory. As a result, memory usage can be reduced.

[0156] As described above, the same effects as those of the embodiments described can be obtained in this embodiment as well. Furthermore, in this embodiment, line-dropped image data is output from the imaging device 19D to the image processing device 10D. This makes it possible to generate image data with suppressed flicker and reduced motion blur while reducing the amount of image data read out in the imaging device 19D and the amount of image data transferred to the image processing device 10D. In addition, since the amount of image data read out in the imaging device 19D can be reduced, the power consumption of the imaging device 19D can be reduced. In other words, the power consumption of the imaging device 19D when the frame rate is set to 120fps can be made equivalent to the power consumption of the imaging device 19D when the frame rate is set to 60fps.

[0157] Although the present invention has been described above based on various embodiments, the present invention is not limited to the requirements shown in the above embodiments. These points can be modified as long as they do not impair the spirit of the present invention, and can be appropriately determined according to their application. [Explanation of Symbols]

[0158] 10, 10C, 10D Image Processing Device 11 Information Processing Devices 12 Display device 13 ECU 14 Wireless communication equipment 15 sensors 16 Drive unit 17 Lamp device 18 Navigation System 19 (19a, 19b, 19c, 19d, 19e) Imaging device 19B, 19C, 19D Imaging devices 20 CPU 21 Interface device 22 Drive unit 23 Auxiliary storage device 24 memory devices 30 Recording media 41, 41C Difference calculation part 42 Total Calculation Unit 43, 43C Combined Gain Calculation Unit 44, 44C, 44D Blend Ratio Calculation Unit 45, 45D Blending Process 46D Frame Reconstruction Unit 47D, 48D Vertical Enlargement Section 51 Acquisition Department 52C, 52D Image synthesis unit 53C Switching Control Unit 100 Image Processing Systems 190-pixel cell array 191 Drive unit 192, 192A, 192B Readout section 193 Bus 194, 194A Flicker detection unit 195 Exposure adjustment section 196, 196A, 196B, 196C Operation Control Unit 197B Image Synthesis Unit 200 Mobile Units BUS DIF (Difference) DIFr ratio EXPT Exposure time Fi Frame Fm Frame F[n], F[n+1] frames F[n2] Pixel value Fu[n], Fu[n+1] IMG image K synthesis gain M Total value Mr ratio PX pixel cell RST reset signal SEL selection signal RD read data α blending ratio

Claims

1. An image sensor unit having multiple photoelectric conversion elements arranged in a matrix, A flicker detection unit detects a flicker region where flicker occurs using image data acquired by the image sensor unit. When the flicker detection unit detects the flicker region, a synthesis processing unit synthesizes multiple image data with different acquisition timings in the flicker region. It has, The image sensor unit has a plurality of lines, each containing a predetermined number of photoelectric conversion elements arranged in one direction, and each of these lines is driven multiple times during one frame period. The flicker detection unit determines a pixel region to be a flicker region if the difference between the pixel values ​​generated for each pixel in two separate drives is greater than or equal to a first threshold, and the sum of the pixel values ​​generated for each pixel in the two separate drives is greater than or equal to a second threshold. Imaging device.

2. The flicker detection unit determines a pixel region containing a moving object as a region where the difference between the pixel values ​​generated in each of the two drives is greater than or equal to the first threshold, and the sum of the pixel values ​​generated in each of the two drives is less than or equal to the third threshold, which is less than or equal to the second threshold. The synthesis processing unit outputs one of the pixel values ​​generated by the two drives in the moving object region as image data. The imaging apparatus according to claim 1.

3. An image processing device for processing image data acquired by an imaging device including an image sensor unit having a plurality of photoelectric conversion elements arranged in a matrix, A flicker detection unit that uses the aforementioned image data to detect flicker regions where flicker occurs, When the flicker detection unit detects the flicker region, a synthesis processing unit synthesizes multiple image data with different acquisition timings in the flicker region. It has, The image sensor unit has a plurality of lines, each containing a predetermined number of photoelectric conversion elements arranged in one direction, and each of these lines is driven multiple times during one frame period. The flicker detection unit determines a pixel region to be a flicker region if the difference between the pixel values ​​generated for each pixel in two separate drives is greater than or equal to a first threshold, and the sum of the pixel values ​​generated for each pixel in the two separate drives is greater than or equal to a second threshold. Image processing device.

4. The flicker detection unit detects the flicker region using image data from multiple frames, When the flicker detection unit detects the flicker region, the synthesis processing unit synthesizes the image data of the flicker region from the plurality of frames. The image processing apparatus according to claim 3.

5. The flicker detection unit determines a pixel region to be a flicker region if the difference in pixel values ​​between two frames is greater than or equal to a first threshold, and the sum of the pixel values ​​between the two frames is greater than or equal to a second threshold. The image processing apparatus according to claim 4.

6. The flicker detection unit determines a pixel region containing a moving object as a region where the difference in pixel values ​​between the two frames is greater than or equal to the first threshold, and the sum of the pixel values ​​between the two frames is less than or equal to the third threshold (which is less than the second threshold). The synthesis processing unit outputs image data from one of the two frames in the motion region. The image processing apparatus according to claim 5.

7. The flicker detection unit calculates the blending ratio of the pixel values ​​of the two frames according to the difference in the pixel values ​​and the sum of the pixel values. The synthesis processing unit blends the pixel values ​​of the two frames according to the blend ratio calculated by the flicker detection unit. The image processing apparatus according to claim 5 or claim 6.

8. The imaging device has a switching control unit that switches the frame rate of the imaging device from a first frame rate to a second frame rate higher than the first frame rate when the brightness of the image shown in the image data acquired by the imaging device exceeds a fourth threshold. The synthesis processing unit outputs the image data of the two frames sequentially without combining them while setting the first frame rate, and outputs the image data of the two frames combined according to the blend ratio while setting the second frame rate. The image processing apparatus according to claim 7.

9. During the setting of the second frame rate, the system includes a composite gain calculation unit that calculates a gain to be multiplied by the composite image data according to the brightness of the image. The image processing apparatus according to claim 8.

Images