Information processing device, imaging device, and information processing method

Abstract

An information processing device according to an embodiment comprises: a blinking detection unit for detecting the presence or absence of blinking and blinking coordinates from an image based on exposure data; a blinking correction processing unit for performing blinking correction processing for correcting pixel values of the image so as to suppress blinking of the image; a non-blinking correction processing unit for performing non-blinking correction processing for correcting pixel values of the image so as to adjust the brightness of the image; and a synthesis processing unit for synthesizing the image subjected to the blinking correction processing and the image subjected to the non-blinking correction processing on the basis of the presence or absence of the blinking and the blinking coordinates.

Description

Information processing device, imaging device, and information processing method

[0001] The present disclosure relates to an information processing device, an imaging device, and an information processing method.

[0002] Traffic signals, such as traffic lights, have, for example, light-emitting diodes (LEDs). Traffic signals with LEDs are powered by commercial power and flash according to a cycle of an AC power frequency (e.g., 50 or 60 Hz). On the other hand, an imaging device mounted on a moving object such as an automobile captures images at a fixed cycle. Depending on the imaging conditions, an image of an unlit traffic light may be captured. For this reason, a technique for capturing an image of a lit traffic light has been proposed (see, for example, Patent Document 1).

[0003] JP 2013-038600 A

[0004] When an imaging device captures an image of a flashing traffic light, for example, periodic luminance changes (changes in brightness) in accordance with the flashing period of the traffic light, i.e., flickering, may appear in the captured image. For this reason, there is a demand for suppressing flickering in the captured image.

[0005] Therefore, the present disclosure provides an information processing device, an imaging device, and an information processing method that are capable of suppressing flicker in a captured image.

[0006] The information processing device according to the embodiment comprises a blinking detection unit that detects the presence or absence of blinking and the coordinates from an image based on exposure data; a blinking correction processing unit that performs blinking correction processing to correct the pixel values ​​of the image so as to suppress blinking of the image; a non-blinking correction processing unit that performs non-blinking correction processing to correct the pixel values ​​of the image so as to adjust the brightness of the image; and a synthesis processing unit that synthesizes the image on which the blinking correction processing has been performed and the image on which the non-blinking correction processing has been performed based on the presence or absence of blinking and the coordinates.

[0007] The imaging device according to the embodiment comprises a sensor that generates exposure data, a blinking detection unit that detects the presence or absence of blinking and the coordinates from an image based on the exposure data, a blinking correction processing unit that performs blinking correction processing to correct the pixel values ​​of the image so as to suppress blinking of the image, a non-blinking correction processing unit that performs non-blinking correction processing to correct the pixel values ​​of the image so as to adjust the brightness of the image, and a synthesis processing unit that synthesizes the image on which the blinking correction processing has been performed and the image on which the non-blinking correction processing has been performed based on the presence or absence of blinking and the coordinates.

[0008] The information processing method according to the embodiment includes an information processing device detecting the presence or absence of blinking and coordinates from an image based on the exposure data, performing a blinking correction process to correct pixel values ​​of the image so as to suppress blinking of the image, performing a non-blinking correction process to correct pixel values ​​of the image so as to adjust the brightness of the image, and synthesizing the image on which the blinking correction process has been performed and the image on which the non-blinking correction process has been performed based on the presence or absence of blinking and coordinates.

[0009] 1 is a diagram illustrating an example of the configuration of an imaging device according to an embodiment; FIG. 2 is a diagram illustrating an example of the configuration of a sensor according to an embodiment; FIG. 3 is a diagram illustrating a pixel block of a pixel array unit according to an embodiment; FIG. 4 is a diagram illustrating a Bayer arrangement in which four adjacent pixels have the same color according to an embodiment; FIG. 5 is a diagram illustrating a Bayer arrangement in which nine adjacent pixels have the same color according to an embodiment; FIG. 6 is a diagram illustrating an example of the configuration of a pixel according to an embodiment; FIG. 7 is a diagram illustrating a timing chart of pixel reset and transfer according to an embodiment; FIG. 8 is a diagram illustrating a timing chart of sampling exposure according to an embodiment; FIG. 9 is a diagram illustrating another timing chart of sampling exposure according to an embodiment; FIG. 10 is a diagram illustrating an exposure pattern for each frame of sampling exposure according to an embodiment; FIG. 11 is a diagram illustrating an example of signal processing according to an embodiment; FIG. 12 is a diagram illustrating sampling exposure of a blinking traffic light according to an embodiment; FIG. 13 is a diagram illustrating sampling exposure data according to an embodiment; A flowchart showing the flow of blink detection processing according to an embodiment; FIG. 14 is a diagram illustrating a target block and a scanning block according to an embodiment; FIG. 15 is a diagram illustrating exposure data of a blinking traffic light according to an embodiment; FIG. 16 is a diagram illustrating exposure data of contours and lines according to an embodiment; FIG. 17 is a diagram illustrating calculation of a Mix ratio according to an embodiment; FIG. 18 is a diagram illustrating an example of processing by a blinking correction processing unit according to an embodiment; FIG. 19 is a diagram illustrating an example of processing by a non-blinking correction processing unit according to an embodiment; FIG. 19 is a diagram illustrating an example of processing by a synthesis processing unit according to an embodiment; FIG. 20 is a diagram illustrating a timing chart of a first modified example of sampling exposure according to an embodiment; FIG. 1 is a diagram showing a timing chart of a modified example 2 of sampling exposure according to an embodiment. FIG. 2 is a diagram showing a detailed configuration example 1 of a part of a sensor according to an embodiment. FIG. 3 is a diagram showing a detailed configuration example 2 of a part of a sensor according to an embodiment. FIG. 4 is a diagram showing a detailed configuration example 2 of a part of a sensor according to an embodiment. FIG. 5 is a block diagram showing an example of a schematic configuration of a vehicle control system. FIG. 6 is an explanatory diagram showing an example of the installation positions of an outside-vehicle information detection unit and an imaging unit.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The embodiments include examples and modifications. Note that the technology according to the present disclosure is not limited to the embodiments. Furthermore, in the embodiments, essentially identical components are designated by the same reference numerals, and redundant explanations will be omitted.

[0011] The present disclosure will be described in the following order: 1. Embodiment 1-1. Configuration example of an imaging device 1-2. Configuration example of a sensor 1-3. Configuration example of a pixel array unit 1-4. Configuration example of a pixel 1-5. Exposure example of sampling exposure 1-6. Processing example of signal processing 1-6-1. Overall processing example 1-6-2. Processing example of a blinking detection unit 1-6-3. Processing example of a blinking correction processing unit 1-6-4. Processing example of a non-blinking correction processing unit 1-6-5. Processing example of a synthesis processing unit 1-7. Modification 1-7-1. Modification 1 1-7-2. Modification 2 1-8. Detailed configuration example of a sensor 1-8-1. Configuration example 1 1-8-2. Configuration example 2 1-9. Actions and effects 2. Other embodiments 3. Application examples 4. Supplementary notes

[0012] <1. Embodiment> <1-1. Configuration Example of Imaging Apparatus> An example of the configuration of an imaging apparatus 1 according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of an imaging apparatus 1 according to this embodiment.

[0013] As shown in FIG. 1, the imaging device 1 includes a sensor 10 and a processing unit 20 .

[0014] The sensor 10 is an image sensor that generates image data (e.g., sampling exposure data) of a subject. For example, the sensor 10 acquires the sampling exposure data using a multi-shutter technique. In the example of Figure 1, an optical system such as a lens that focuses light onto the sensor 10 is not shown, but it is desirable to provide such an optical system.

[0015] The processing unit 20 includes a blinking detection unit 21, a blinking correction processing unit 22, a storage unit 23, a non-blinking correction processing unit 24, and a synthesis processing unit 25. The processing unit 20 is an example of an information processing device.

[0016] The blinking detection unit 21 detects the presence or absence and coordinates of blinking of traffic signals, such as traffic lights (e.g., spatially regular changes in brightness), from the sampling exposure data. The blinking detection result includes, for example, information such as the presence or absence of blinking and the blinking coordinates. The blinking coordinates include, for example, the XY coordinates of the image. The presence or absence of blinking and the blinking coordinates are used, for example, to identify blinking areas, non-blinking areas, and other areas.

[0017] The flicker correction processing unit 22 performs flicker correction processing to correct pixel values ​​of the image so as to suppress flickering of traffic lights in the sampling exposure data. For example, the flicker correction processing unit 22 averages the pixel values ​​of the sampling exposure data. The storage unit 23 stores information necessary for the above-mentioned correction processing.

[0018] The non-blinking correction processor 24 performs a non-blinking correction process to correct pixel values ​​of the image so as to adjust the brightness of the image of the sampling exposure data (e.g., to increase the brightness). For example, the non-blinking correction processor 24 amplifies the pixel values ​​of the sampling exposure data.

[0019] The composition processing unit 25 performs composition processing to combine the image after the blinking correction processing and the image after the non-blinking correction processing based on the blinking detection result, and outputs the composite processed image to a subsequent development processing. For example, the composition processing unit 25 performs composition processing by changing the composition ratio depending on the blinking detection result. The composite processed image is, for example, raw data in a quad Bayer array in which traffic light blinking has been corrected.

[0020] The blinking detection unit 21, the blinking correction processing unit 22, the storage unit 23, the non-blinking correction processing unit 24, and the synthesis processing unit 25 will be described in detail later.

[0021] Note that various types of development processes may be used as the development process described above. As an example, a re-mosaic process may be performed to convert the pixel array of an input image into a different pixel array. In the re-mosaic process, for example, an image having a quad-Bayer array is converted into a Bayer array. In this conversion, for example, eight-directional pixel value gradients are detected at the position of a conversion target pixel that is the target of color conversion processing among the constituent pixels of the input image. Based on the eight-directional pixel value gradients, an interpolated pixel value calculation mode for the conversion target pixel is determined. An interpolated pixel value for the conversion target pixel position is calculated according to the determined processing mode, and the interpolated pixel value is used. This process is merely an example, and other processes may also be performed.

[0022] The processing unit 20 may be configured, for example, by hardware and / or software, and the configuration is not particularly limited. The processing unit 20 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a CPLD (Complex Programmable Logic Device). The processing unit 20 may also be realized by a computer such as a CPU (Central Processing Unit), an MCU (Micro Controller Unit), or an MPU (Micro Processor Unit) executing a program pre-stored in a ROM (Read Only Memory) using a RAM (Random Access Memory) as a work area.

[0023] The imaging device 1 is used in mobile objects such as automobiles, drones, and automatic delivery robots. Automatic delivery robots travel both outdoors and indoors, for example. An automatic delivery robot traveling autonomously outdoors travels and stops according to road traffic lights. An automatic transport robot traveling autonomously indoors (for example, in a factory) travels and stops according to a multi-dot LED sign (display). The multi-dot LED sign can display, for example, the same information as a road traffic light, or can display arrows or letters. The multi-dot LED sign is an example of a traffic light. The imaging device 1 can also be used to capture images of flashing displays in addition to traffic lights.

[0024] 1-2. Example of the configuration of the sensor An example of the configuration of the sensor 10 according to this embodiment will be described with reference to Fig. 2 to Fig. 5. Fig. 2 is a diagram showing an example of the configuration of the sensor 10 according to this embodiment.

[0025] As shown in FIG. 2, the sensor 10 includes a pixel array unit 11, a vertical drive unit 12, a column signal processing unit 13, and a control unit 14.

[0026] The pixel array unit 11 is configured by arranging a plurality of pixels 100. In the example of Fig. 2, the pixels 100 are arranged in a two-dimensional matrix. Each pixel 100 includes a photoelectric conversion unit that performs photoelectric conversion of incident light, and generates a pixel signal (pixel value) based on the incident light that is irradiated. For example, a photodiode is used as the photoelectric conversion unit.

[0027] A signal line 15 and a signal line 16 are connected to each pixel 100. The pixel 100 generates a pixel signal under control of a control signal transmitted through the signal line 15, and outputs the generated pixel signal to the column signal processing unit 13 via the signal line 16. The signal line 15 is arranged for each row of the two-dimensional matrix, and is commonly connected to, for example, multiple pixels 100 arranged in a row. The signal line 16 is arranged for each column of the two-dimensional matrix, and is commonly connected to, for example, multiple pixels 100 arranged in a column.

[0028] The vertical drive unit 12 generates control signals for the pixels 100. The vertical drive unit 12 generates a control signal for each row of the two-dimensional matrix of the pixel array unit 11, and outputs the generated control signals to each pixel 100 via each signal line 15.

[0029] The column signal processing unit 13 processes pixel signals generated by the pixels 100. The column signal processing unit 13 simultaneously processes pixel signals transmitted from each of a predetermined number of pixels 100 via signal lines 16. For example, the column signal processing unit 13 performs analog-to-digital conversion to convert the pixel signals generated by the pixels 100 from analog to digital, and correlated double sampling to remove offset errors in the pixel signals. The processed pixel signals, i.e., image data (e.g., sampling exposure readout data), are output to the processing unit 20.

[0030] The control unit 14 controls the vertical drive unit 12 and the column signal processing unit 13. The control unit 14 generates control signals for controlling the vertical drive unit 12 and the column signal processing unit 13 based on data instructing a clock, an operation mode, etc., input from an external circuit, etc. The control unit 14 outputs control signals to the vertical drive unit 12 via a signal line 17 and to the column signal processing unit 13 via a signal line 18, thereby controlling the vertical drive unit 12 and the column signal processing unit 13.

[0031] The sensor 10 described above basically receives light incident from a subject, performs photoelectric conversion, and outputs an electrical signal corresponding to the amount of light, i.e., an image signal (image data). The type of imaging device 1 is not particularly limited, and may be a front-illuminated type or a back-illuminated type. Furthermore, the imaging device 1 may be a CMOS (Complementary Metal Oxide Semiconductor) type or any other type.

[0032] <1-3. Configuration Example of Pixel Array Unit> A configuration example of the pixel array unit 11 according to this embodiment will be described with reference to FIGS. 3 to 5. FIG. 3 is a diagram for explaining a pixel block A1 of the pixel array unit 11 according to this embodiment. FIG. 4 is a diagram for explaining a Bayer arrangement in which four adjacent pixels (four adjacent pixels 100) according to this embodiment have the same color. FIG. 5 is a diagram for explaining a Bayer arrangement in which nine adjacent pixels (nine adjacent pixels 100) according to this embodiment have the same color.

[0033] As shown in FIG. 3 , a plurality of pixel blocks A1 are set in the pixel array unit 11. Each pixel block A1 is a block in which four adjacent pixels 100 are arranged in a 2×2 matrix. In the example of FIG. 3 , the four pixels 100 in the pixel block A1 are identified by the numbers (1) to (4). The horizontal direction in FIG. 3 corresponds to the row direction, and the vertical direction in FIG. 3 corresponds to the column direction (the same applies to other figures). The pixels 100 have color filters (e.g., red, green, blue, etc.). However, if color display is not required, the pixels 100 may not have color filters.

[0034] As shown in Fig. 4, the pixel array unit 11 has pixel block A1 configured in a Bayer array (quad-Bayer array) in which an RGGB array (arrangement of red, green, green, and blue) is repeated in a 2x2 matrix. The pixel block A1 is a 2x2 matrix block in which four adjacent pixels 100 of the same color form a group. In the example of Fig. 4, as in the example of Fig. 3, the four pixels 100 in pixel block A1 are distinguished by the numbers (1) to (4).

[0035] Although such a Bayer array is used, other Bayer arrays may also be used. For example, as shown in FIG. 5, the pixel array unit 11 may be configured in a Bayer array in which pixel block A1 is arranged in a 2×2 matrix, repeating an RGGB array (arrangement of red, green, green, and blue), as in FIG. 4. The pixel block A1 is a 3×3 matrix, with nine adjacent pixels 100 of the same color forming a group. In the example of FIG. 5, the nine pixels 100 in pixel block A1 are distinguished by the numbers (1) to (9). Such a Bayer array may also be used.

[0036] 1-4. Example of Pixel Configuration An example of the configuration of the pixel 100 according to this embodiment will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a diagram showing an example of the configuration of the pixel 100 according to this embodiment. Fig. 7 is a diagram showing a timing chart of resetting and transferring the pixel 100 according to this embodiment.

[0037] As shown in FIG. 6, the pixel 100 includes a photoelectric conversion unit 101 , a charge holding unit 102 , a reset element 103 , a transfer element 104 , and a selection element 105 .

[0038] The photoelectric conversion unit 101 performs photoelectric conversion of incident light. As the photoelectric conversion unit 101, for example, a photodiode is used.

[0039] The charge holding portion 102 holds the charge generated by the photoelectric conversion of the photoelectric conversion portion 101. As the charge holding portion 102, for example, a capacitor is used.

[0040] The reset element 103 resets the charge of the photoelectric conversion unit 101. The reset element 103 is connected to one end of the photoelectric conversion unit 101. The reset is performed, for example, by establishing electrical continuity between the photoelectric conversion unit 101 and a power supply line to drain the charge of the photoelectric conversion unit 101. A control signal (B1.Reset) to the reset element 103 is input from the above-mentioned vertical drive unit 12. For example, a transistor is used as the reset element 103.

[0041] The transfer element 104 transfers the charges generated by photoelectric conversion in the photoelectric conversion unit 101 from the photoelectric conversion unit 101 to the charge holding unit 102. The transfer element 104 is connected between the photoelectric conversion unit 101 and the charge holding unit 102. A control signal (B2. transfer) to the transfer element 104 is input from the above-mentioned vertical drive unit 12. As the transfer element 104, for example, a transistor is used.

[0042] The selection element 105 sends the charge held by the charge holding unit 102 to the signal line 16. This selection element 105 functions to select the pixel 100 to be read out. The selection element 105 is connected between the charge holding unit 102 and the signal line 16. A control signal (B3. Readout) to the selection element 105 is input from the above-mentioned vertical drive unit 12. For example, a transistor is used as the selection element 105.

[0043] According to this pixel 100, the presence of the charge holding portion 102 for accumulating charge allows exposure (accumulation of charge) and readout (readout of charge) to be controlled independently. The structure of this pixel 100 is an example of a pixel structure of the sensor 10 that achieves a multi-shutter. Note that the configuration of the pixel 100 shown in FIG. 6 is merely an example, and other configurations may be used.

[0044] As shown in FIG. 7 , exposure starts when the control signal (B1. Reset) changes from on to off, and ends when the control signal (B2. Transfer) changes on. The exposure time is from the end of the reset on pulse (start of exposure) to the beginning of the transfer on pulse (end of exposure). For example, continuous exposure can be achieved by controlling the reset and transfer at different times for multiple adjacent pixels 100. Note that readout is performed, for example, line by line sequentially in the vertical direction (column direction).

[0045] 1-5. Exposure Example of Sampling Exposure An exposure example of sampling exposure according to this embodiment will be described with reference to Figs. 8 to 10. Fig. 8 is a diagram showing a timing chart of sampling exposure according to this embodiment. Fig. 9 is a diagram showing another timing chart of sampling exposure according to this embodiment. Fig. 10 is a diagram showing an exposure pattern for each frame of sampling exposure according to this embodiment.

[0046] As shown in Figure 8, the flashing frequency of the traffic light is, for example, 120 Hz or 100 Hz. The flashing period is, for example, 1 / 120 (s) or 1 / 100 (s). The sampling exposure interval is 1 / 40 (s), the sampling exposure section is 1 / 80 (s), and the exposure pause section is 1 / 80 (s). The exposure operation is, for example, a repetition of sampling exposure, exposure pause, sampling exposure, and exposure pause. The numerical values ​​in Figure 8 are merely examples (the same applies to other figures).

[0047] In sampling exposure using multi-shutter technology, an exposure time (e.g., 2 ms) is set so that the blinking of traffic lights can be captured as a change in brightness. With multi-shutter technology, the timing of the start and end of sampling exposure differs for each pixel 100 in pixel block A1. The numbers (1) to (4) in FIG. 8 correspond to the numbers (1) to (4) in FIGS. 3 and 4 (the same applies to the other figures). In the example of FIG. 8, sampling exposure is performed in the order of pixels 100 (1) to (4). This is the same for all pixel blocks A1.

[0048] In all pixel blocks A1, the exposure times of the pixels 100 (1) to (4) do not overlap in time. For example, the exposure start timing for the pixel 100 (2) comes after the exposure end timing for the pixel 100 (1). Similarly, the exposure start timing for the pixel 100 (3) comes after the exposure end timing for the pixel 100 (2). The exposure start timing for the pixel 100 (4) comes after the exposure end timing for the pixel 100 (3).

[0049] Under these conditions, sampling exposure is performed on the pixels 100 (1) to (4) in the sampling exposure section (1 / 80 s), and sampling exposure data is read out from the sensor 10 in the exposure pause section (1 / 80 s). By performing sampling exposure multiple times for one cycle of the flashing of a traffic light, it is possible to sample the flashing of the traffic light. By using a multi-shutter function (multi-shutter technology) to independently control the start and end of exposure for each of the pixels 100 (1) to (4), it is possible to continuously expose all four pixels 100 in the pixel block A1 in all pixel blocks A1.

[0050] In the example of Fig. 8, the exposure times of the pixels 100 (1) to (4) do not overlap in time, but this is not limiting and they may overlap in time. As shown in Fig. 9, the exposure times of the pixels 100 (1) to (4) overlap in time. For example, the exposure start timing of the pixel 100 (2) arrives before the exposure end timing of the pixel 100 (1). Similarly, the exposure start timing of the pixel 100 (3) arrives before the exposure end timing of the pixel 100 (2). The exposure start timing of the pixel 100 (4) arrives before the exposure end timing of the pixel 100 (3).

[0051] That is, the timing of the sampling exposure is not limited to the timing shown in Fig. 8, and may be, for example, the timing shown in Fig. 9, or may be timing other than the timings shown in Fig. 8 and Fig. 9. Furthermore, the exposure times of the pixels 100 (1) to (4) are the same, but may be different, and may be set longer or shorter than those shown in Fig. 8 and Fig. 9. For example, when the brightness around the sensor 10 is darker than normal (predetermined brightness), the exposure time is set longer than the normal exposure time (predetermined exposure time).

[0052] As shown in Fig. 10, in each frame (e.g., frame 1, frame 2, frame 3, and frame 4), the same exposure pattern is repeated when the traffic light flashes at a frequency of 120 Hz. On the other hand, when the traffic light flashes at a frequency of 100 Hz, different exposure patterns are alternately repeated. Note that the frames are obtained at each sampling exposure interval (e.g., sampling exposure section) in Fig. 8 or 9.

[0053] As a result of sampling exposure, when a flashing traffic light is photographed, a spatial flickering pattern is obtained. Although the shape of the spatial flickering periodic exposure pattern differs between 100 Hz and 120 Hz traffic lights, the sensor 10 can detect the periodic flickering exposure pattern for both traffic lights. Through sampling exposure, the exposure data for a flashing traffic light has a spatially regular variation in luminance. By capturing this regular variation in luminance and correcting the variation in luminance, flicker can be suppressed. For example, flicker from both 120 Hz and 100 Hz traffic lights can be suppressed.

[0054] <1-6. Example of Signal Processing> <1-6-1. Example of Overall Processing> An example of overall signal processing according to this embodiment will be described with reference to Figs. 11 to 13. Fig. 11 is a diagram for explaining an example of signal processing according to this embodiment. Fig. 12 is a diagram for explaining sampling exposure of flashing traffic lights according to this embodiment. Fig. 13 is a diagram for explaining sampling exposure data according to this embodiment.

[0055] As shown in FIG. 11 , a sampling exposure C1 of the blinking of a traffic light is performed on the sensor 10. In response, sampling exposure data C2 is output from the sensor 10. The blinking detection unit 21 performs a blinking detection process on the sampling exposure data C2 to obtain a blinking detection result C3. The blinking detection result C3 includes information such as whether or not the traffic light is blinking and the blink coordinates. The blinking correction processing unit 22 performs a blinking correction process on the sampling exposure data C2 to obtain an image after the blinking correction process, and the non-blinking correction processing unit 24 performs a non-blinking correction process on the sampling exposure data C2 to obtain an image after the non-blinking correction process. The synthesis processing unit 25 synthesizes the image after the blinking correction process and the image after the non-blinking correction process to obtain a synthesized image C4. Then, in a subsequent development process, the synthesized image C4 is developed to obtain a synthesized image C5. Each of the processes constituting this example of processing will be described in detail below.

[0056] As shown in FIG. 12 , in the sampling exposure C1 of the flashing traffic light, exposure (1) is a sampling exposure using each pixel 100 (1) in each pixel block A1. Exposure (2) is a sampling exposure using each pixel 100 (2) in each pixel block A1. Exposure (3) is a sampling exposure using each pixel 100 (3) in each pixel block A1. Exposure (4) is a sampling exposure using each pixel 100 (4) in each pixel block A1. Sampling exposures are performed in this manner. Thinning is performed in each of exposures (1) to (4). When the thinned exposure data is developed, an image such as that shown in FIG. 12 is obtained.

[0057] In sampling exposure, not all pixels 100 are exposed at the same time, but a group of adjacent pixels 100 is treated as one unit (pixel block A1), and each pixel 100 in the pixel block A1 is exposed in sequence. Therefore, the number of pixels 100 used in one sampling (the number of exposed pixels 100) is 1 / the number of adjacent pixels 100. All pixels 100 are thinned out, and exposure is performed sequentially.

[0058] At this time, each sampling exposure uses a plurality of adjacent pixels 100 (pixel block A1) that are different in space. Therefore, when each sampling exposure data is collected as data for all pixels 100, if there is a brightness difference in each sampling exposure due to the blinking of a traffic light, it appears as a brightness difference in the spatial direction. The brightness difference that appears in the spatial direction forms a repeating pattern, with a plurality of adjacent pixels 100 as one unit (pixel block A1). The data collected from each sampling exposure data is treated as all pixel data for one frame (sampling exposure data C2).

[0059] As shown in Figure 13, the sampling exposure data C2 is a combination of the exposure data (data that has been thinned out) from each of exposures (1) to (4). A flickering pattern, which is a spatially regular change in brightness, appears. From the sampling exposure data C2, "areas where traffic lights are flashing (e.g., spatially regular changes in brightness)" within the shooting angle of view are detected, and a flicker detection result C3 is obtained.

[0060] <1-6-2. Processing Example of Blink Detection Unit> A processing example of the blink detection unit 21 according to this embodiment will be described with reference to FIGS. 14 to 18. FIG. 14 is a flowchart showing the flow of blink detection processing according to this embodiment. FIG. 15 is a diagram for explaining a target block D1 and a scanning block D2 according to this embodiment. FIG. 16 is a diagram for explaining exposure data of traffic light blinking according to this embodiment. FIG. 17 is a diagram for explaining exposure data of contours and lines according to this embodiment. FIG. 18 is a diagram for explaining calculation of a Mix ratio (Mix proportion) according to this embodiment.

[0061] 14, in step S11, the blinking detection unit 21 treats a plurality of adjacent pixels as a pixel block A1, and selects a target block D1 and surrounding blocks (pixel blocks A1) of size (N x N). In step S12, the blinking detection unit 21 finds the maximum and minimum pixel values ​​for each of the (N x N) blocks.

[0062] As shown in Fig. 15, the (N x N) block is a scanning block D2 that is scanned with respect to the image of the sampling exposure data C2. The scanning block D2 includes a target block D1 located in the center and multiple peripheral blocks that surround the target block D1. Each of the target block D1 and the peripheral blocks is a pixel block A1. In the example of Fig. 15, N = 3, and the number of blocks is 9.

[0063] 14 , in step S13, the blinking detection unit 21 obtains the maximum / minimum value from the maximum and minimum values ​​for each (N×N) block. In step S14, the blinking detection unit 21 sorts the (N×N) obtained maximum / minimum values ​​in descending order. In step S15, the blinking detection unit 21 selects the (N+1)th largest maximum / minimum value.

[0064] In step S16, the blinking detection unit 21 determines whether the selected maximum / minimum value is equal to or less than the first threshold value. If the blinking detection unit 21 determines that the selected maximum / minimum value is not equal to or less than the first threshold value (is greater than the first threshold value) (step S16: No), in step S17, the blinking detection unit 21 determines that the area of ​​interest, which is the block of interest D1, is a blinking area.

[0065] On the other hand, if the blinking detection unit 21 determines in step S16 that the selected maximum / minimum value is equal to or less than the first threshold value (step S16: Yes), the blinking detection unit 21 determines in step S18 whether the selected maximum / minimum value is equal to or greater than a second threshold value, which is smaller than the first threshold value.

[0066] In step S18, if the blinking detection unit 21 determines that the selected maximum / minimum value is not greater than the second threshold value (smaller than the second threshold value) (step S18: No), in step S19, it determines that the area of ​​interest is a non-blinking area.

[0067] On the other hand, in step S18, if the blinking detection unit 21 determines that the selected maximum / minimum value is greater than or equal to the second threshold value (step S18: Yes), in step S20, it determines that the area of ​​interest is a Mix area (synthesis area) and calculates the Mix ratio (synthesis ratio).

[0068] This process is performed for the entire area of ​​the image while shifting the scanning block D2 in the horizontal direction (X direction) and vertical direction (Y direction) (see Figure 15). This makes it possible to reduce false detection of blinking. As shown in Figure 16, the blinking of traffic lights is spatially spread. Therefore, the pixel values ​​of the target block D1 and each surrounding block are the same. On the other hand, as shown in Figure 17, contours and lines are not spatially spread. Therefore, the pixel values ​​of the target block D1 and each surrounding block are not the same. Due to these differences, it is possible to distinguish between contours and lines and blinking when detecting them.

[0069] In calculating the mix ratio in step S20, the blinking detection unit 21 calculates the mix ratio based on the mix ratio setting graph E1, as shown in FIG. 18 . The mix ratio is the mix ratio between the blinking area and the non-blinking area. In the example of FIG. 18 , the blinking detection unit 21 sets the mix ratio by varying it between 0% and 100% when the (N+1)th largest maximum / minimum value is between the first threshold and the second threshold. The mix ratio is used in the compositing process of the compositing processing unit 25. The mix ratio setting graph E1 is an example of mix ratio setting information (compositing ratio setting information).

[0070] As described above, the flicker detection unit 21 detects the presence or absence of spatially regular luminance changes and their coordinates due to the flickering of traffic signals, such as traffic lights, from the sampling exposure data (detection of flickering presence or absence and flickering coordinates). The sampling exposure data of a flickering traffic signal has spatially regular luminance changes. By capturing and correcting this regular luminance change, flicker can be suppressed. Flicker is a phenomenon in which brightness changes due to flickering, for example, are reflected in an image.

[0071] <1-6-3. Processing Example of Blinking Correction Processing Unit> A processing example of the blinking correction processing unit 22 according to this embodiment will be described with reference to Fig. 19. Fig. 19 is a diagram for explaining a processing example of the blinking correction processing unit 22 according to this embodiment.

[0072] 19 , the blinking correction processing unit 22 calculates the average value of each pixel value of adjacent pixels 100 in the pixel block A1 (step S31) and replaces each pixel value of adjacent pixels 100 in the pixel block A1 with the calculated average value (step S32). The blinking correction processing unit 22 then reads each pixel value (past value) of adjacent pixels 100 at the same coordinates as each pixel value (current value) of the replaced adjacent pixels 100, and calculates a composite value for each pixel 100 using the current and past values. The blinking correction processing unit 22 then uses the composite value as the pixel value of each adjacent pixel 100. The composite value is, for example, (current value + past value) / 2. This series of processes is performed for the entire pixel block A1.

[0073] In this way, the flickering correction processing unit 22 performs a conversion process to replace the spatially regular luminance change caused by the flickering of the traffic light with the average value of the four pixels 100 (four adjacent pixels 100) within the pixel block A1. In addition, in order to suppress temporal fluctuations in the calculated average values ​​of the four pixels 100, the flickering correction processing unit 22 performs a calculation process to calculate and output the average value of the current four pixels 100 and the average value of the corresponding past (previous) four pixels 100. The storage unit 23 stores the past (previous) average values ​​of the four pixels 100 for the above-mentioned calculation process.

[0074] <1-6-4. Processing Example of Non-Blinking Correction Processing Unit> A processing example of the non-blinking correction processing unit 24 according to this embodiment will be described with reference to Fig. 20. Fig. 20 is a diagram for explaining a processing example of the non-blinking correction processing unit 24 according to this embodiment.

[0075] 20, the non-blinking correction processing unit 24 has an amplifier 24a. This non-blinking correction processing unit 24 performs non-blinking correction processing by increasing the gain for each pixel block A1, i.e., for each pixel 100, using the amplifier 24a. As a result, the input signal is amplified and output. In other words, the brightness of the image after the non-blinking correction processing is increased.

[0076] <1-6-5. Processing Example of the Combining Processor> A processing example of the combining processor 25 according to this embodiment will be described with reference to Fig. 21. Fig. 21 is a diagram for explaining a processing example of the combining processor 25 according to this embodiment.

[0077] 21 , the composition processing unit 25 combines the image after the blinking correction processing (signal from the blinking correction processing unit 22) and the image after the non-blinking correction processing (signal from the non-blinking correction processing unit 24) based on the presence or absence of spatially regular luminance changes and their coordinates (blinking detection results) determined by the blinking detection unit 21. At this time, the composition processing unit 25 changes the mix ratio to combine the image after the blinking correction processing and the image after the non-blinking correction processing. The mix ratio is a mix ratio calculated by the blinking detection unit 21 based on the mix ratio setting graph E1.

[0078] The mix ratio is calculated for each pixel block A1 (block of interest D1) (see FIG. 14 ). For example, if the area of ​​interest, which is block of interest D1, is a blinking area, the mix ratio is determined to be 100% from the mix ratio setting graph E1. Also, if the area of ​​interest is a non-blinking area, the mix ratio is determined to be 0% from the mix ratio setting graph E1. If the area of ​​interest is a synthesis area, the mix ratio is determined to a predetermined ratio from the mix ratio setting graph E1. Using these mix ratios, the signal from the blinking portion correction process and the signal from the non-blinking portion correction process are synthesized for each pixel block A1 to generate a synthesized image.

[0079] In the blinking detection, if the (N+1)th largest maximum / minimum value falls between the first and second thresholds, the Mix ratio will change continuously between 0% and 100%. In the combining process, for example, the signal from the blinking portion correction process and the signal from the non-blinking portion correction process are combined based on the following conditions:

[0080] When the (N+1)th largest maximum / minimum value is equal to or greater than the first threshold, the Mix ratio is 1.0 (100%), and the corrected output is a signal from the blinking correction processing unit 22. When the (N+1)th largest maximum / minimum value is equal to or less than the second threshold, the Mix ratio is 0.0 (0%), and the corrected output is a signal from the non-blinking correction processing unit 24.

[0081] When the (N+1)th largest maximum / minimum value is between the first threshold and the second threshold, the mix ratio is (((N+1)th largest maximum / minimum value) - second threshold) / (first threshold - second threshold). The corrected output is (corrected output = mix ratio x signal from blinking correction processing unit 22 + (1.0 - mix ratio) x signal from non-blinking correction processing unit 24).

[0082] <1-7. Modifications> <1-7-1. Modification 1> Modification 1 of sampling exposure according to this embodiment will be described with reference to Fig. 22. Fig. 22 is a diagram showing a timing chart of modification 1 of sampling exposure according to this embodiment.

[0083] As shown in Fig. 22, an exposure (exposure for a purpose other than the sampling exposure) may be inserted in an exposure pause section, and exposure data corresponding to the exposure inserted in the exposure pause section may be read out in a sampling exposure section adjacent to the exposure pause section. Except for the insertion of this exposure and the reading out of the exposure data, the process is the same as the sampling exposure shown in Fig. 8.

[0084] The exposure times of the pixels 100 (1) to (4) in the exposure pause section overlap in time. For example, the exposure times of the pixels 100 (1) to (4) in the exposure pause section are the same. Furthermore, the exposure start timing and the exposure end timing of the pixels 100 (1) to (4) in the exposure pause section are the same.

[0085] 22 is the end time of the reading of the sampling exposure. The exposure end timing (the timing at which the transfer elements 104 are turned off) of the exposure inserted in the exposure pause section is, for example, the same as Tsamp_end or later than Tsamp_end.

[0086] According to the first modification, the amount of exposure of each pixel 100 is increased, so that it is possible to suppress insufficient exposure at night, for example. Furthermore, it is possible to appropriately deal with exposure other than that of traffic lights, for example.

[0087] <1-7-2. Modification 2> Modification 2 of sampling exposure according to this embodiment will be described with reference to Fig. 23. Fig. 23 is a diagram showing a timing chart of modification 2 of sampling exposure according to this embodiment.

[0088] As shown in Fig. 23, the exposure times of some of the pixels 100 (1) to (4) in the exposure pause section are different. Other than these different exposure times, the exposure is the same as the sampling exposure shown in Fig. 22. However, as the exposure in the exposure pause section, for example, HDR (high dynamic range) is inserted.

[0089] The exposure times of the pixels 100 (1) and (3) in the exposure pause section are the same, and the exposure times of the pixels 100 (2) and (4) in the exposure pause section are the same, but the exposure times of the pixels 100 (1) and (3) in the exposure pause section are longer than the exposure times of the pixels 100 (2) and (4) in the exposure pause section.

[0090] Furthermore, the exposure start timing of the pixels 100 (1) and (3) in the exposure pause section is the same, and the exposure start timing of the pixels 100 (2) and (4) in the exposure pause section is the same. However, the exposure start timing of the pixels 100 (1) and (3) in the exposure pause section is earlier than the exposure start timing of the pixels 100 (2) and (4) in the exposure pause section. The exposure end timing of the pixels 100 (1) to (4) in the exposure pause section is the same.

[0091] Note that Tsamp_end in Fig. 23 is the end time of the readout of the sampling exposure, as in Fig. 22. The exposure end timing (the disconnection timing of the transfer element 104) of the exposure inserted in the exposure pause section is, for example, the same as Tsamp_end or later than Tsamp_end.

[0092] According to Modification 2, as in Modification 1, the amount of exposure of each pixel 100 is increased, so that it is possible to suppress insufficient exposure at night, for example, and also to appropriately deal with exposure other than that of traffic lights. It is also possible to compensate for insufficient dynamic range, for example.

[0093] Note that the exposure time and exposure start timing of the pixels 100 (1) and (3) in the exposure pause section are the same, and the exposure time and exposure start timing of the pixels 100 (2) and (4) in the exposure pause section are the same, but this is not limited to this. For example, in other combinations, the exposure times and exposure start timings of some of the pixels 100 (1) to (4) in the exposure pause section may be the same or different.

[0094] <1-8. Detailed Configuration Examples of Sensor> <1-8-1. Configuration Example 1> Detailed Configuration Example 1 of the sensor 10 according to this embodiment will be described with reference to Fig. 24 and Fig. 25. Fig. 24 and Fig. 25 are diagrams each showing detailed Configuration Example 1 of a part of the sensor 10 according to this embodiment.

[0095] 24, the vertical drive unit 12 includes, for example, a vertical transfer timing circuit (read timing circuit) 12a and a shutter / reset control circuit 12b. The signal lines 15 include, for example, a plurality of control lines 15a.

[0096] The vertical transfer timing circuit 12a outputs a control signal (B3. read) to each selection element 105 of each pixel 100 for each row of the pixel array section 11. The shutter / reset control circuit 12b outputs a control signal (B1. reset) to the reset element 103 of each pixel 100 in the pixel block A1, and also outputs a control signal (B2. transfer) to the transfer element 104 of each pixel 100.

[0097] 25 , the column signal processing unit 13 includes, for example, a plurality of amplifiers 13 a and a plurality of ADCs (Analog to Digital Converters) 13 b. The amplifiers 13 a and the ADCs 13 b are connected to signal lines 16 and are connected in series with each other. The amplifiers 13 a and the ADCs 13 b are provided for each signal line 16.

[0098] 24 and 25 , the pixels 100 aligned in the vertical direction are connected to one signal line 16 for every two vertically adjacent pixels 100. The two vertically adjacent pixels 100 are, for example, the two vertically adjacent pixels 100 in pixel block A1.

[0099] According to this configuration example 1, the start and end of exposure of each pixel 100 in pixel block A1, i.e., reset and shutter, can be independently controlled using a small number of control lines 15a. The presence of the charge holding unit 102 allows exposure and readout to be performed independently, and readout is performed sequentially for each signal line 16 (line) in the vertical direction. Because the readout timing differs for each signal line 16, the pixels 100 aligned in the column direction can share the signal line 16.

[0100] <1-8-2. Configuration Example 2> A detailed configuration example 2 of the sensor 10 according to this embodiment will be described with reference to Fig. 26 and Fig. 27. Fig. 26 and Fig. 27 are diagrams each showing a detailed configuration example 2 of a part of the sensor 10 according to this embodiment.

[0101] 26 , the vertical drive unit 12 has a vertical transfer timing circuit (read timing circuit) 12a and a shutter / reset control circuit 12b, similar to the above-described configuration example 1. The signal line 15 also includes a plurality of control lines 15a, similar to the above-described configuration example 1. The vertical transfer timing circuit 12a and the shutter / reset control circuit 12b are similar to those in the above-described configuration example 1.

[0102] 27, the column signal processing unit 13 has a plurality of amplifiers 13a and a plurality of ADCs 13b, similar to the above-described configuration example 1. The amplifiers 13a and the ADCs 13b are similar to those in the above-described configuration example 1.

[0103] 26 and 27 , each pixel 100 in a pixel block A1 is connected to one signal line 16. The pixels 100 in the pixel block A1 are, for example, four pixels 100 that are adjacent to each other in the vertical and horizontal directions in the pixel block A1.

[0104] Configuration Example 2 as described above can provide the same effects as those of Configuration Example 1. Furthermore, compared to Configuration Example 1, the number of amplifiers 13 a and ADCs 13 b can be reduced, and the number of signal lines 16 can also be reduced.

[0105] <1-9. Actions and Effects> As described above, the information processing device (e.g., processing unit 20) according to the embodiment includes a blinking detection unit 21 that detects the presence or absence of blinking and its coordinates from an image based on exposure data (e.g., sampling exposure data), a blinking correction processing unit 22 that performs blinking correction processing to correct pixel values ​​of the image to suppress blinking of the image, a non-blinking correction processing unit 24 that performs non-blinking correction processing to correct pixel values ​​of the image to adjust the brightness of the image (e.g., to increase the brightness of the image), and a synthesis processing unit 25 that synthesizes the image that has undergone the blinking correction processing and the image that has undergone the non-blinking correction processing based on the presence or absence of blinking and the coordinates (see FIG. 1, etc.). As a result, the image that has undergone the blinking correction processing and the image that has undergone the non-blinking correction processing are synthesized to generate a synthesized image, thereby suppressing flicker in the captured image that corresponds to the blinking cycle of a traffic light or the like, for example.

[0106] Alternatively, the blinking detection unit 21 may detect the presence or absence of blinking and the coordinates for each pixel block A1 including a plurality of pixels 100 (see, for example, FIGS. 14 and 15 ). This allows for reliable detection of blinking in the image.

[0107] Alternatively, the blinking detection unit 21 may determine, for each pixel block A1, the area of ​​interest that is the pixel block A1 as a blinking area, a non-blinking area, or a synthesis area, and the synthesis processing unit 25 may synthesize, for each pixel block A1, an image that has undergone the blinking correction process and an image that has undergone the non-blinking correction process, depending on whether the area of ​​interest is a blinking area, a non-blinking area, or a synthesis area (see, for example, FIGS. 14 and 15 ). This allows appropriate synthesis of the images.

[0108] Furthermore, the composition processing unit 25 may compose an image that has undergone the blinking correction process and an image that has undergone the non-blinking correction process based on the composition ratio of each of the blinking area, the non-blinking area, and the composition area (see, for example, FIGS. 14 and 15 ), thereby achieving appropriate composition of the respective images.

[0109] Alternatively, the blinking detection unit 21 may calculate the maximum and minimum pixel values ​​for each pixel block A1 in a scanning block D2 having a plurality of pixel blocks A1, including a pixel block A1 (e.g., a block D1) that is an area of ​​interest, calculate the maximum and minimum pixel values ​​for each pixel block A1, sort the maximum and minimum pixel values ​​for each pixel block A1 in descending order, select a predetermined maximum or minimum value, and determine whether the area of ​​interest is a blinking area, a non-blinking area, or a composite area based on the selected predetermined maximum or minimum value (see FIGS. 14 and 15 ). This allows for reliable detection of blinking in an image.

[0110] Furthermore, the blinking detection unit 21 may determine the area of ​​interest as a blinking area if the selected maximum / minimum value of the predetermined number is greater than a first threshold, determine the area of ​​interest as a non-blinking area if the selected maximum / minimum value of the predetermined number is smaller than a second threshold that is smaller than the first threshold, and determine the area of ​​interest as a combined area if the selected maximum / minimum value of the predetermined number is between the first and second thresholds (see FIGS. 14 and 15 ). This allows for reliable detection of blinking in the image.

[0111] Furthermore, the pixel block A1 (for example, the block D1) serving as the area of ​​interest may be located in the center of the scanning block D2 (see FIGS. 14 and 15). This allows for reliable detection of flickering in the image.

[0112] Furthermore, when the blinking detection unit 21 determines the area of ​​interest as the combining area, it may set a combining ratio between the image that has undergone the blinking correction process and the image that has undergone the non-blinking correction process (see FIGS. 14 and 15 ), thereby realizing appropriate combining of the respective images.

[0113] The blinking detection unit 21 may also set the combining ratio based on combining ratio setting information (e.g., Mix ratio setting graph E1) indicating the relationship between a predetermined maximum / minimum value and the combining ratio (see FIG. 18 ). This allows the combining ratio to be reliably determined.

[0114] Furthermore, when the blinking detection unit 21 determines that the area of ​​interest is a blinking area, the combination processing unit 25 may set the combination ratio to, for example, 100% so that only images on which the blinking correction process has been performed are used, and when the blinking detection unit 21 determines that the area of ​​interest is a non-blinking area, the combination processing unit 25 may set the combination ratio to, for example, 0% so that only images on which the non-blinking correction process has been performed are used (see FIGS. 14 and 15 ). This allows for simplified processing.

[0115] Furthermore, the composition processing unit 25 may compose an image that has been subjected to the blinking correction process and an image that has been subjected to the non-blinking correction process based on a composition ratio (see FIG. 21 ), thereby making it possible to appropriately compose an image that has been subjected to the blinking correction process and an image that has been subjected to the non-blinking correction process.

[0116] Furthermore, when the area of ​​interest is a blinking area (for example, when the combination ratio is 100%), the combination processing unit 25 may use only the image on which the blinking correction process has been performed, when the area of ​​interest is a non-blinking area (for example, when the combination ratio is 0%), the combination processing unit 25 may use only the image on which the non-blinking correction process has been performed, and when the area of ​​interest is a combination area (for example, when the combination ratio is between 0% and 100%), the combination processing unit 25 may combine the image on which the blinking correction process has been performed and the image on which the non-blinking correction process has been performed based on the combination ratio (see FIG. 21). This allows for simplified processing.

[0117] Furthermore, when the area of ​​interest is a synthesis area, the synthesis processing unit 25 may synthesize the image that has undergone the blinking correction processing and the image that has undergone the non-blinking correction processing based on the formula: synthesis ratio × signal output from the blinking correction processing unit + (100% - synthesis ratio) × signal output from the non-blinking correction processing unit (see FIG. 21 ). This allows the image that has undergone the blinking correction processing and the image that has undergone the non-blinking correction processing to be appropriately synthesized.

[0118] Alternatively, as the blinking correction process, the blinking correction processing unit 22 may calculate an average value of the pixel values ​​of each of the pixels 100 for each pixel block A1 including a plurality of pixels 100, and replace the pixel values ​​of each of the pixels 100 in the pixel block A1 with the calculated average value (see FIG. 19 ). This allows for appropriate blinking correction.

[0119] Furthermore, the blinking correction processing unit 22 may combine current pixel values, which are the pixel values ​​of each of the pixels 100 in the replaced pixel block A1, with past pixel values, which are the pixel values ​​of each of the pixels 100 in a past pixel block A1 that has the same coordinates as the replaced pixel block A1 (see FIG. 19 ). This allows for appropriate blinking correction.

[0120] Furthermore, the blinking correction processing unit 22 may combine the current pixel value and the past pixel value based on the formula (current pixel value + past pixel value) / 2 (see FIG. 19 ), thereby making it possible to appropriately perform blinking correction.

[0121] Furthermore, the non-blinking correction processing unit 24 may increase the pixel values ​​of the image as the non-blinking correction processing (see FIG. 20), thereby making it possible to adjust the brightness of the image to be increased.

[0122] The exposure data may also be data acquired using a multi-shutter technique that sets the exposure start and exposure end for each pixel 100 in a pixel block A1 that includes a plurality of pixels 100 (see, for example, FIGS. 8 to 10).

[0123] 2. Other Embodiments The configurations and processes according to the above-described embodiments (including examples and modified examples) may be implemented in various different forms other than the above-described embodiments. For example, the configurations and processes are not limited to the above-described examples and may be implemented in various forms. Furthermore, for example, the configurations, processing procedures, specific names, or information including various data and parameters shown in the above documents and drawings may be changed arbitrarily unless otherwise specified.

[0124] Furthermore, the components and processes according to the above-described embodiments (including examples and modifications) do not necessarily have to be physically configured as shown in the drawings. In other words, the specific forms of distribution and integration of the components and processes are not limited to those shown in the drawings, and all or part of them may be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc.

[0125] Furthermore, the configurations and processes of the above-described embodiments (including examples and modified examples) may be combined as appropriate. For example, at least a part of an embodiment may be combined as appropriate with at least a part of another embodiment. Furthermore, the effects of the embodiments are merely examples and are not intended to be limiting, and other effects may also be obtained.

[0126] 3. Application Examples The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure may be realized as a device (e.g., electronic device) mounted on any type of moving object, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, a robot, construction machinery, or agricultural machinery (tractor). Furthermore, for example, the technology according to the present disclosure may be realized as a device (e.g., electronic device) mounted on an endoscopic surgery system, a microsurgery system, or the like.

[0127] 28 is a block diagram showing a schematic configuration example of a vehicle control system 7000, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied. The vehicle control system 7000 includes a plurality of electronic control units connected via a communication network 7010. In the example shown in FIG. 28 , the vehicle control system 7000 includes a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, an outside-vehicle information detection unit 7400, an inside-vehicle information detection unit 7500, and an integrated control unit 7600. The communication network 7010 connecting these multiple control units may be an in-vehicle communication network conforming to any standard, such as a Controller Area Network (CAN), a Local Interconnect Network (LIN), a Local Area Network (LAN), or FlexRay (registered trademark).

[0128] Each control unit includes a microcomputer that performs arithmetic processing according to various programs, a memory unit that stores the programs executed by the microcomputer or parameters used in various calculations, and a drive circuit that drives various controlled devices. Each control unit includes a network I / F for communicating with other control units via a communication network 7010, and a communication I / F for communicating with devices or sensors inside and outside the vehicle via wired or wireless communication. Figure 28 illustrates the functional configuration of the integrated control unit 7600, including a microcomputer 7610, a general-purpose communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon receiving unit 7650, an in-vehicle device I / F 7660, an audio / video output unit 7670, an in-vehicle network I / F 7680, and a memory unit 7690. Other control units also include a microcomputer, a communication I / F, a memory unit, and the like.

[0129] The drivetrain control unit 7100 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 7100 functions as a control device for a drive force generating device for generating drive force for the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating braking force for the vehicle. The drivetrain control unit 7100 may also function as a control device for an ABS (Antilock Brake System) or an ESC (Electronic Stability Control), etc.

[0130] A vehicle state detection unit 7110 is connected to the drivetrain control unit 7100. The vehicle state detection unit 7110 includes at least one of a gyro sensor that detects the angular velocity of the axial rotational motion of the vehicle body, an acceleration sensor that detects the acceleration of the vehicle, or a sensor that detects the amount of operation of the accelerator pedal, the amount of operation of the brake pedal, the steering angle of the steering wheel, the engine rotation speed, the rotation speed of the wheels, etc. The drivetrain control unit 7100 performs arithmetic processing using signals input from the vehicle state detection unit 7110, and controls the internal combustion engine, the drive motor, the electric power steering device, the brake device, etc.

[0131] The body system control unit 7200 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 7200 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 7200. The body system control unit 7200 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0132] The battery control unit 7300 controls the secondary battery 7310, which is the power supply source for the drive motor, in accordance with various programs. For example, information such as battery temperature, battery output voltage, or remaining battery capacity is input to the battery control unit 7300 from a battery device equipped with the secondary battery 7310. The battery control unit 7300 performs arithmetic processing using these signals, and controls the temperature regulation of the secondary battery 7310 or a cooling device or the like equipped in the battery device.

[0133] The outside vehicle information detection unit 7400 detects information outside the vehicle equipped with the vehicle control system 7000. For example, at least one of an imaging unit 7410 and an outside vehicle information detection unit 7420 is connected to the outside vehicle information detection unit 7400. The imaging unit 7410 includes at least one of a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. The outside vehicle information detection unit 7420 includes at least one of an environmental sensor for detecting the current weather or climate, or a surrounding information detection sensor for detecting other vehicles, obstacles, pedestrians, etc. around the vehicle equipped with the vehicle control system 7000.

[0134] The environmental sensor may be, for example, at least one of a raindrop sensor that detects rain, a fog sensor that detects fog, a sunshine sensor that detects the degree of sunshine, and a snow sensor that detects snowfall. The surrounding information detection sensor may be at least one of an ultrasonic sensor, a radar device, and a LIDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) device. The imaging unit 7410 and the outside vehicle information detection unit 7420 may each be provided as an independent sensor or device, or may be provided as a device in which multiple sensors or devices are integrated.

[0135] 29 shows an example of the installation positions of the imaging unit 7410 and the vehicle exterior information detection unit 7420. The imaging units 7910, 7912, 7914, 7916, and 7918 are provided, for example, at least one of the front nose, side mirrors, rear bumper, back door, and upper part of the windshield inside the vehicle cabin of the vehicle 7900. The imaging unit 7910 provided on the front nose and the imaging unit 7918 provided on the upper part of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 7900. The imaging units 7912 and 7914 provided on the side mirrors mainly acquire images of the sides of the vehicle 7900. The imaging unit 7916 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 7900. The imaging unit 7918 provided on the upper part of the windshield inside the vehicle cabin is mainly used to detect leading vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0136] 29 shows an example of the imaging ranges of the imaging units 7910, 7912, 7914, and 7916. Imaging range a indicates the imaging range of the imaging unit 7910 provided on the front nose, imaging ranges b and c indicate the imaging ranges of the imaging units 7912 and 7914 provided on the side mirrors, respectively, and imaging range d indicates the imaging range of the imaging unit 7916 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 7910, 7912, 7914, and 7916, a bird's-eye view image of the vehicle 7900 viewed from above can be obtained.

[0137] The outside vehicle information detection units 7920, 7922, 7924, 7926, 7928, and 7930 provided on the front, rear, sides, corners, and above the windshield inside the vehicle cabin of the vehicle 7900 may be, for example, ultrasonic sensors or radar devices. The outside vehicle information detection units 7920, 7926, and 7930 provided on the front nose, rear bumper, back door, and above the windshield inside the vehicle cabin of the vehicle 7900 may be, for example, LIDAR devices. These outside vehicle information detection units 7920 to 7930 are mainly used to detect preceding vehicles, pedestrians, obstacles, etc.

[0138] Returning to FIG. 28 , the explanation will be continued. The outside-vehicle information detection unit 7400 causes the imaging unit 7410 to capture an image outside the vehicle and receives the captured image data. The outside-vehicle information detection unit 7400 also receives detection information from the connected outside-vehicle information detection unit 7420. If the outside-vehicle information detection unit 7420 is an ultrasonic sensor, a radar device, or a LIDAR device, the outside-vehicle information detection unit 7400 emits ultrasonic waves or electromagnetic waves and receives information on the received reflected waves. Based on the received information, the outside-vehicle information detection unit 7400 may perform object detection processing or distance detection processing for people, vehicles, obstacles, signs, or text on the road surface. Based on the received information, the outside-vehicle information detection unit 7400 may also perform environmental recognition processing for recognizing rainfall, fog, road conditions, etc. Based on the received information, the outside-vehicle information detection unit 7400 may also calculate the distance to an object outside the vehicle.

[0139] The outside vehicle information detection unit 7400 may also perform image recognition processing or distance detection processing to recognize people, vehicles, obstacles, signs, or characters on the road surface based on the received image data. The outside vehicle information detection unit 7400 may perform processing such as distortion correction or alignment on the received image data, and may also generate an overhead image or a panoramic image by combining image data captured by different image capturing units 7410. The outside vehicle information detection unit 7400 may also perform viewpoint conversion processing using image data captured by different image capturing units 7410.

[0140] The interior information detection unit 7500 detects information inside the vehicle. A driver state detection unit 7510 that detects the driver's state is connected to the interior information detection unit 7500, for example. The driver state detection unit 7510 may include a camera that captures an image of the driver, a biosensor that detects the driver's biometric information, or a microphone that collects sound from within the vehicle cabin. The biosensor is provided, for example, on the seat or steering wheel, and detects the biometric information of a passenger sitting in the seat or the driver gripping the steering wheel. The interior information detection unit 7500 may calculate the driver's level of fatigue or concentration based on the detection information input from the driver state detection unit 7510, or may determine whether the driver is dozing off. The interior information detection unit 7500 may perform processing such as noise canceling on the collected audio signal.

[0141] The integrated control unit 7600 controls the overall operation of the vehicle control system 7000 according to various programs. An input unit 7800 is connected to the integrated control unit 7600. The input unit 7800 may be implemented by a device that can be operated by a passenger, such as a touch panel, a button, a microphone, a switch, or a lever. Data obtained by voice recognition of a voice input through a microphone may be input to the integrated control unit 7600. The input unit 7800 may be, for example, a remote control device using infrared or other radio waves, or an externally connected device such as a mobile phone or a personal digital assistant (PDA) that can operate the vehicle control system 7000. The input unit 7800 may be, for example, a camera, in which case the passenger can input information using gestures. Alternatively, data obtained by detecting the movement of a wearable device worn by the passenger may be input. Furthermore, the input unit 7800 may include, for example, an input control circuit that generates an input signal based on information input by the passenger using the input unit 7800 and outputs the input signal to the integrated control unit 7600. Passengers and the like operate this input unit 7800 to input various data to the vehicle control system 7000 and to instruct processing operations.

[0142] The storage unit 7690 may include a ROM (Read Only Memory) that stores various programs executed by the microcomputer, and a RAM (Random Access Memory) that stores various parameters, calculation results, sensor values, etc. The storage unit 7690 may also be realized by a magnetic storage device such as an HDD (Hard Disc Drive), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like.

[0143] The general-purpose communication I / F 7620 is a general-purpose communication I / F that mediates communication with various devices present in the external environment 7750. The general-purpose communication I / F 7620 may implement a cellular communication protocol such as GSM (Global System of Mobile communications), WiMAX (registered trademark), LTE (Long Term Evolution), or LTE-Advanced (LTE-A), or other wireless communication protocols such as a wireless LAN (also referred to as Wi-Fi (registered trademark)) or Bluetooth (registered trademark). The general-purpose communication I / F 7620 may connect to a device (e.g., an application server or a control server) present on an external network (e.g., the Internet, a cloud network, or an operator-specific network) via, for example, a base station or an access point. In addition, the general-purpose communication I / F 7620 may connect to a terminal located near the vehicle (for example, a terminal of a driver, pedestrian, or store, or an MTC (Machine Type Communication) terminal) using, for example, P2P (Peer To Peer) technology.

[0144] The dedicated communication I / F 7630 is a communication I / F that supports a communication protocol designed for use in vehicles. The dedicated communication I / F 7630 may implement a standard protocol such as WAVE (Wireless Access in Vehicle Environment), which is a combination of a lower layer IEEE 802.11p and an upper layer IEEE 1609, DSRC (Dedicated Short Range Communications), or a cellular communication protocol. The dedicated communication I / F 7630 typically performs V2X communication, which is a concept including one or more of vehicle-to-vehicle communication, vehicle-to-infrastructure communication, vehicle-to-home communication, and vehicle-to-pedestrian communication.

[0145] The positioning unit 7640 performs positioning by receiving, for example, GNSS signals from GNSS (Global Navigation Satellite System) satellites (for example, GPS signals from GPS (Global Positioning System) satellites), and generates position information including the latitude, longitude, and altitude of the vehicle. Note that the positioning unit 7640 may identify the current position by exchanging signals with a wireless access point, or may obtain position information from a terminal such as a mobile phone, PHS, or smartphone that has a positioning function.

[0146] The beacon receiving unit 7650 receives, for example, radio waves or electromagnetic waves transmitted from radio stations or the like installed on the road, and acquires information such as the current location, congestion, road closures, required travel time, etc. The function of the beacon receiving unit 7650 may be included in the dedicated communication I / F 7630 described above.

[0147] The in-vehicle device I / F 7660 is a communication interface that mediates connections between the microcomputer 7610 and various in-vehicle devices 7760 present in the vehicle. The in-vehicle device I / F 7660 may establish wireless connections using wireless communication protocols such as wireless LAN, Bluetooth (registered trademark), NFC (Near Field Communication), or WUSB (Wireless USB). Furthermore, the in-vehicle device I / F 7660 may establish a wired connection such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface), or MHL (Mobile High-Definition Link) via a connection terminal (and a cable, if necessary) not shown. The in-vehicle device 7760 may include, for example, at least one of a mobile device or a wearable device owned by a passenger, or an information device carried into or attached to the vehicle. The in-vehicle device 7760 may also include a navigation device that searches for a route to an arbitrary destination. The in-vehicle device I / F 7660 exchanges control signals or data signals with these in-vehicle devices 7760.

[0148] The in-vehicle network I / F 7680 is an interface that mediates communication between the microcomputer 7610 and the communication network 7010. The in-vehicle network I / F 7680 transmits and receives signals in accordance with a predetermined protocol supported by the communication network 7010.

[0149] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 in accordance with various programs based on information acquired via at least one of the general-purpose communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the in-vehicle network I / F 7680. For example, the microcomputer 7610 may calculate control target values ​​for the driving force generating device, the steering mechanism, or the braking device based on the acquired information inside and outside the vehicle, and output control commands to the drivetrain control unit 7100. For example, the microcomputer 7610 may perform cooperative control aimed at realizing functions of an Advanced Driver Assistance System (ADAS), including vehicle collision avoidance or impact mitigation, following driving based on the following distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc. In addition, the microcomputer 7610 may perform cooperative control for the purpose of autonomous driving, in which the vehicle travels autonomously without relying on driver operation, by controlling a driving force generating device, a steering mechanism, a braking device, etc. based on information acquired about the vehicle's surroundings.

[0150] The microcomputer 7610 may generate three-dimensional distance information between the vehicle and objects such as surrounding structures and people, and create local map information including information about the vicinity of the vehicle's current location, based on information acquired via at least one of the general-purpose communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the in-vehicle network I / F 7680. Furthermore, the microcomputer 7610 may predict dangers, such as a vehicle collision, the approach of a pedestrian, or entry into a closed road, based on the acquired information, and generate a warning signal. The warning signal may be, for example, a signal for generating a warning sound or turning on a warning lamp.

[0151] The audio / image output unit 7670 transmits at least one of audio and image output signals to an output device capable of visually or audibly notifying vehicle occupants or the outside of the vehicle of information. In the example of FIG. 28 , an audio speaker 7710, a display unit 7720, and an instrument panel 7730 are illustrated as output devices. The display unit 7720 may include, for example, at least one of an on-board display and a head-up display. The display unit 7720 may have an AR (Augmented Reality) display function. The output device may also be other devices, such as headphones, a wearable device such as an eyeglass-type display worn by the occupant, a projector, or a lamp. When the output device is a display device, the display device visually displays results obtained by various processes performed by the microcomputer 7610 or information received from other control units in various formats, such as text, images, tables, and graphs. When the output device is an audio output device, the audio output device converts audio signals, such as reproduced audio data or acoustic data, into analog signals and audibly outputs the analog signals.

[0152] In the example shown in FIG. 28 , at least two control units connected via the communication network 7010 may be integrated into a single control unit. Alternatively, each control unit may be composed of multiple control units. Furthermore, the vehicle control system 7000 may include another control unit not shown. In the above description, some or all of the functions performed by one of the control units may be assigned to another control unit. In other words, as long as information is transmitted and received via the communication network 7010, predetermined arithmetic processing may be performed by one of the control units. Similarly, a sensor or device connected to one of the control units may be connected to another control unit, and multiple control units may transmit and receive detection information to each other via the communication network 7010.

[0153] A computer program for realizing each function of the imaging device 1 according to this embodiment described with reference to FIG. 1 can be implemented in any control unit or the like. A computer-readable recording medium storing such a computer program can also be provided. Examples of the recording medium include a magnetic disk, an optical disk, a magneto-optical disk, and a flash memory. The computer program may also be distributed, for example, via a network without using a recording medium.

[0154] In the vehicle control system 7000 described above, the imaging device 1 according to this embodiment described with reference to Fig. 1 can be applied to the integrated control unit 7600 of the application example shown in Fig. 28. For example, the processing unit 20 of the imaging device 1 corresponds to the microcomputer 7610 of the integrated control unit 7600. However, the processing unit 20 of the imaging device 1 is not limited to this.

[0155] Furthermore, at least some of the components of the imaging device 1 according to the present embodiment described with reference to Fig. 1 may be realized in a module (for example, an integrated circuit module configured on a single die) for the integrated control unit 7600 shown in Fig. 28. Alternatively, at least some of the components of the imaging device 1 according to the present embodiment described with reference to Fig. 1 may be realized by a plurality of control units of the vehicle control system 7000 shown in Fig. 28.

[0156] <4. Supplementary Notes> The present technology may also be configured as follows. (1) An information processing device comprising: a blinking detection unit that detects the presence or absence of blinking and coordinates from an image based on exposure data; a blinking correction processing unit that performs blinking correction processing to correct pixel values ​​of the image to suppress blinking of the image; a non-blinking correction processing unit that performs non-blinking correction processing to correct pixel values ​​of the image to adjust brightness of the image; and a synthesis processing unit that synthesizes the image on which the blinking correction processing has been performed and the image on which the non-blinking correction processing has been performed based on the presence or absence of blinking and the coordinates. (2) The information processing device described in (1), wherein the blinking detection unit detects the presence or absence of blinking and the coordinates for each pixel block including a plurality of pixels. (3) The information processing device described in (2), wherein the blinking detection unit determines, for each pixel block, an area of ​​interest that is the pixel block as a blinking area, a non-blinking area, or a synthesis area, and the synthesis processing unit synthesizes, for each pixel block, the image on which the blinking correction processing has been performed and the image on which the non-blinking correction processing has been performed depending on whether the area of ​​interest is the blinking area, the non-blinking area, or the synthesis area. (4) The information processing device according to (3), wherein the synthesis processing unit synthesizes the image on which the flashing correction processing has been performed and the image on which the non-flashing correction processing has been performed based on synthesis ratios of the flashing area, the non-flashing area, and the synthesis area, respectively. (5) The information processing device according to (3) or (4), wherein the flashing detection unit calculates a maximum and minimum pixel value for each of the pixel blocks in a scanning block having a plurality of pixel blocks including the pixel block that is the area of ​​interest, calculates the maximum / minimum pixel value for each of the pixel blocks, sorts the maximum / minimum pixel values ​​for each pixel block in descending order, selects a predetermined maximum / minimum value, and determines the area of ​​interest as the flashing area, the non-flashing area, or the synthesis area according to the selected predetermined maximum / minimum value.(6) The information processing device according to (5), wherein the blinking detection unit determines the area of ​​interest as the blinking area if the selected maximum / minimum value of the predetermined number is greater than a first threshold, determines the area of ​​interest as the non-blinking area if the selected maximum / minimum value of the predetermined number is smaller than a second threshold that is smaller than the first threshold, and determines the area of ​​interest as the synthesis area if the selected maximum / minimum value of the predetermined number is between the first threshold and the second threshold. (7) The information processing device according to (5) or (6), wherein the pixel block that becomes the area of ​​interest is located at the center of the scanning block. (8) The information processing device according to any one of (3) to (7), wherein the blinking detection unit, when determining the area of ​​interest as the synthesis area, sets a synthesis ratio between the image that has been subjected to the blinking correction processing and the image that has been subjected to the non-blinking correction processing. (9) The information processing device according to (8), wherein the blinking detection unit sets the synthesis ratio based on synthesis ratio setting information that indicates the relationship between the maximum / minimum value of the predetermined number and the synthesis ratio. (10) The information processing device according to (8) or (9), wherein the blinking detection unit, when determining the area of ​​interest as the blinking area, sets the combination ratio so that the synthesis processing unit uses only the images on which the blinking correction processing has been performed, and when determining the area of ​​interest as the non-blinking area, sets the combination ratio so that the synthesis processing unit uses only the images on which the non-blinking correction processing has been performed. (11) The information processing device according to any one of (8) to (10), wherein the synthesis processing unit combines the images on which the blinking correction processing has been performed and the images on which the non-blinking correction processing has been performed based on the combination ratio. (12) The information processing device according to (11), wherein the synthesis processing unit, when the area of ​​interest is the blinking area, uses only the images on which the non-blinking correction processing has been performed, and when the area of ​​interest is the synthesis area, combines the images on which the blinking correction processing has been performed and the images on which the non-blinking correction processing has been performed based on the combination ratio.(13) The information processing device according to (12), wherein, when the area of ​​interest is the synthesis area, the synthesis processing unit synthesizes the image on which the flashing correction processing has been performed and the image on which the non-flashing correction processing has been performed based on an equation: the synthesis ratio x signal output from the flashing correction processing unit + (100% - the synthesis ratio) x signal output from the non-flashing correction processing unit. (14) The information processing device according to any one of (1) to (13), wherein, as the flashing correction processing, the flashing correction processing unit calculates, for each pixel block including a plurality of pixels, an average value of pixel values ​​of each of the plurality of pixels, and replaces the pixel values ​​of each of the plurality of pixels in the pixel block with the calculated average value. (15) The information processing device according to (14), wherein the flashing correction processing unit synthesizes current pixel values ​​that are the pixel values ​​of each of the plurality of pixels in the replaced pixel block, with past pixel values ​​that are the pixel values ​​of each of the plurality of pixels in the past pixel block that have the same coordinates as the replaced pixel block. (16) The information processing device according to (15), wherein the blinking correction processing unit combines the current pixel value and the past pixel value based on the formula (the current pixel value + the past pixel value) / 2. (17) The information processing device according to any one of (1) to (16), wherein the non-blinking correction processing unit increases a pixel value of the image as the non-blinking correction processing. (18) The information processing device according to any one of (1) to (17), wherein the exposure data is data acquired using a multi-shutter technology in which an exposure start and an exposure end are set for each pixel in a pixel block including a plurality of pixels. (19) An imaging device comprising: a sensor that generates exposure data; a blinking detection unit that detects the presence or absence of blinking and coordinates from an image based on the exposure data; a blinking correction processing unit that performs blinking correction processing to correct pixel values ​​of the image to suppress blinking of the image; a non-blinking correction processing unit that performs non-blinking correction processing to correct pixel values ​​of the image to adjust brightness of the image; and a synthesis processing unit that combines the image on which the blinking correction processing has been performed and the image on which the non-blinking correction processing has been performed, based on the presence or absence of blinking and the coordinates.(20) An information processing method including: an information processing device detecting the presence or absence of blinking and coordinates from an image based on exposure data; performing blinking correction processing to correct pixel values ​​of the image so as to suppress blinking of the image; performing non-blinking correction processing to correct pixel values ​​of the image so as to adjust brightness of the image; and combining the image that has been subjected to the blinking correction processing and the image that has been subjected to the non-blinking correction processing based on the presence or absence of blinking and coordinates. (21) An imaging device including the information processing device described in any one of (1) to (18). (22) A moving body including the imaging device described in (21).

[0157] REFERENCE SIGNS LIST 1 Imaging device 10 Sensor 11 Pixel array section 12 Vertical drive section 12a Vertical transfer timing circuit 12b Shutter / reset control circuit 13 Column signal processing section 13a Amplifier 13b ADC 14 Control section 15 Signal line 15a Control line 16 Signal line 17 Signal line 18 Signal line 20 Processing section 21 Blinking detection section 22 Blinking correction processing section 23 Storage section 24 Non-blinking correction processing section 24a Amplifier 25 Composition processing section 100 Pixel 101 Photoelectric conversion section 102 Charge holding section 103 Reset element 104 Transfer element 105 Selection element A1 Pixel block D1 Target block D2 Scanning block E1 Mix ratio setting graph

Claims

1. An information processing device comprising: a blinking detection unit that detects the presence or absence of blinking and coordinates from an image based on exposure data; a blinking correction processing unit that performs blinking correction processing to correct pixel values ​​of the image so as to suppress blinking of the image; a non-blinking correction processing unit that performs non-blinking correction processing to correct pixel values ​​of the image so as to adjust the brightness of the image; and a synthesis processing unit that synthesizes the image that has been subjected to the blinking correction processing and the image that has been subjected to the non-blinking correction processing based on the presence or absence of blinking and coordinates.

2. The information processing device according to claim 1, wherein the blinking detection unit detects the presence or absence of blinking and the coordinates for each pixel block including a plurality of pixels.

3. The information processing device described in claim 2, wherein the blinking detection unit determines, for each pixel block, the pixel block area of ​​interest as a blinking area, a non-blinking area, or a synthesis area, and the synthesis processing unit synthesizes, for each pixel block, the image on which the blinking correction process has been performed and the image on which the non-blinking correction process has been performed, depending on whether the area of ​​interest is the blinking area, the non-blinking area, or the synthesis area.

4. The information processing device described in claim 3, wherein the synthesis processing unit synthesizes the image on which the blinking correction processing has been performed and the image on which the non-blinking correction processing has been performed based on the respective synthesis ratios of the blinking area, the non-blinking area, and the synthesis area.

5. The information processing device of claim 3, wherein the blinking detection unit calculates the maximum and minimum pixel values ​​for each of a plurality of pixel blocks in a scanning block including the pixel block that is the area of ​​interest, calculates the maximum / minimum pixel values ​​for each pixel block, sorts the maximum / minimum pixel values ​​for each pixel block in descending order, selects a predetermined number of maximum / minimum values, and determines the area of ​​interest as the blinking area, the non-blinking area, or the composite area depending on the selected predetermined number of maximum / minimum values.

6. The information processing device of claim 5, wherein the blinking detection unit determines the area of ​​interest as the blinking area if the selected maximum / minimum value of the predetermined number is greater than a first threshold, determines the area of ​​interest as the non-blinking area if the selected maximum / minimum value of the predetermined number is smaller than a second threshold that is smaller than the first threshold, and determines the area of ​​interest as the synthesis area if the selected maximum / minimum value of the predetermined number is between the first threshold and the second threshold.

7. The information processing device according to claim 5, wherein the pixel block that is the area of ​​interest is located at the center of the scanning block.

8. The information processing device according to claim 3, wherein, when the area of ​​interest is determined as the synthesis area, the blinking detection unit sets a synthesis ratio of the image on which the blinking correction processing has been performed and the image on which the non-blinking correction processing has been performed.

9. The information processing device according to claim 8, wherein the blinking detection unit sets the combining ratio based on combining ratio setting information indicating the relationship between a predetermined number of maximum / minimum values ​​and a combining ratio.

10. The information processing device of claim 8, wherein, when the blinking detection unit determines that the area of ​​interest is the blinking area, the synthesis processing unit sets the synthesis ratio so that only the image on which the blinking correction processing has been performed is used, and when the area of ​​interest is determined that the non-blinking area, the synthesis processing unit sets the synthesis ratio so that only the image on which the non-blinking correction processing has been performed is used.

11. The information processing device according to claim 8, wherein the synthesis processing unit synthesizes the image on which the blinking correction processing has been performed and the image on which the non-blinking correction processing has been performed based on the synthesis ratio.

12. The information processing device described in claim 11, wherein the synthesis processing unit uses only the image on which the flashing correction processing has been performed when the area of ​​interest is the flashing area, uses only the image on which the non-flash correction processing has been performed when the area of ​​interest is the non-flash area, and synthesizes the image on which the flashing correction processing has been performed and the image on which the non-flash correction processing has been performed based on the synthesis ratio when the area of ​​interest is the synthesis area.

13. The information processing device of claim 12, wherein, when the area of ​​interest is the synthesis area, the synthesis processing unit synthesizes the image on which the blinking correction processing has been performed and the image on which the non-blinking correction processing has been performed based on the formula: synthesis ratio x signal output from the blinking correction processing unit + (100% - the synthesis ratio) x signal output from the non-blinking correction processing unit.

14. The information processing device according to claim 1, wherein the blinking correction processing unit calculates an average value of the pixel values ​​of each of the plurality of pixels for each pixel block containing the plurality of pixels, and replaces the pixel values ​​of each of the plurality of pixels in the pixel block with the calculated average value.

15. The information processing device according to claim 14, wherein the blinking correction processing unit combines current pixel values, which are the pixel values ​​of each of the plurality of pixels in the replaced pixel block, with past pixel values, which are the pixel values ​​of each of the plurality of pixels in the past pixel block at the same coordinates as the replaced pixel block.

16. The information processing device according to claim 15, wherein the blinking correction processing unit combines the current pixel value and the past pixel value based on the formula (the current pixel value + the past pixel value) / 2.

17. The information processing device according to claim 1, wherein the non-blinking correction processing unit increases pixel values ​​of the image as the non-blinking correction processing.

18. The information processing device according to claim 1, wherein the exposure data is data acquired using a multi-shutter technique that sets exposure start and exposure end for each pixel in a pixel block including a plurality of pixels.

19. An imaging device comprising: a sensor that generates exposure data; a blinking detection unit that detects the presence or absence of blinking and coordinates from an image based on the exposure data; a blinking correction processing unit that performs blinking correction processing to correct pixel values ​​of the image so as to suppress blinking of the image; a non-blinking correction processing unit that performs non-blinking correction processing to correct pixel values ​​of the image so as to adjust brightness of the image; and a synthesis processing unit that synthesizes the image that has been subjected to the blinking correction processing and the image that has been subjected to the non-blinking correction processing based on the presence or absence of blinking and coordinates.

20. An information processing method including an information processing device detecting the presence or absence of blinking and coordinates from an image based on exposure data; performing blinking correction processing to correct pixel values ​​of the image so as to suppress blinking of the image; performing non-blinking correction processing to correct pixel values ​​of the image so as to adjust the brightness of the image; and combining the image on which the blinking correction processing has been performed and the image on which the non-blinking correction processing has been performed based on the presence or absence of blinking and coordinates.

Images