Scene information generation device and imaging apparatus

Abstract

To generate scene information.SOLUTION: A scene information generation device comprises: a luminance map generation unit that, on the basis of pixel information picked up by a first image pick-up device that has a plurality of pixel blocks having a plurality of pixels and a floating diffusion to be shared, and a control circuit capable of individually controlling a drive mode for each pixel block or for the plurality of pixel blocks, generates a luminance map indicating the degree of brightness for each of the pixel blocks; a motion map generation unit that generates a motion map indicating the degree of change from an immediately previous frame image, on the basis of the pixel information picked up by a second image pick-up device different from the first image pick-up device; and a scene information generation unit that generates scene information indicating the degree of brightness or change for each of the pixel blocks, by employing, for each of the pixel blocks, the degree of brightness indicated in the luminance map generated by the luminance map generation unit or the degree of change indicated in the motion map generated by the motion map generation unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a scene information generating device and an imaging device. [Background technology]

[0002] In conventional imaging systems, the resolution, frame rate, and dynamic range characteristics are generally set to be constant within the screen. Here, in wide-field-of-view images that are intended for display on head-mounted displays, LED (Light Emitting Diode) domes, etc., and that significantly exceed the conventional display angle of view, it is expected that subjects with various characteristics will be simultaneously captured within the same image. Examples of subjects with various characteristics include subjects with fine details, subjects that move quickly, and subjects with large differences in brightness, such as between sunlight and shade.

[0003] To capture images of all subjects captured in such wide-field images using conventional imaging systems while suppressing aliasing, motion blur, blown-out highlights, blocked-up shadows, and degradation of the signal-to-noise ratio (SNR), an image sensor that simultaneously satisfies performance requirements such as high resolution, high frame rate, and high dynamic range is required. However, achieving high resolution and high frame rate requires high-speed readout of pixel signals within the image sensor, but the faster the pixel signals are readout, the more likely the dynamic range and SNR are to degrade. Furthermore, increasing the pixel signal readout speed also leads to problems such as increased readout circuit size and power consumption. Therefore, it is not easy to achieve the high performance described above with conventional image sensors, such as complementary metal oxide semiconductor (CMOS) image sensors.

[0004] On the other hand, from the perspective of improving subjective image quality, the imaging parameters of the image sensor do not necessarily need to be constant within the screen. For example, there is no need to capture a still subject at a high frame rate, and there is no need to capture a fast-moving subject at high resolution because motion blur reduces spatial frequency components. Furthermore, for highly luminous subjects, ensuring dark gradation and SNR is given lower priority, while for low-luminance subjects, ensuring bright gradation is given lower priority.

[0005] As described above, since the required imaging parameter levels differ depending on the characteristics of each of the multiple subjects included in the same image, it is not necessary to keep the imaging parameters of the imaging element constant within the screen. For example, in an area where a stationary subject exists, it is not necessary to capture the image at a high frame rate, and it is desirable to capture the image at a high resolution. In addition, in an area where a fast-moving object exists, the spatial frequency decreases due to motion blur, so it is not necessary to capture the image at a high resolution, and it is desirable to capture the image at a high frame rate.

[0006] Furthermore, by partially shortening the exposure time in areas where high-brightness objects exist and partially extending the exposure time in areas where low-brightness objects exist, the dynamic range of the entire image can be improved. In this way, by appropriately controlling imaging parameters according to the characteristics of each of multiple objects contained in the same image, it is possible to obtain image quality that is essentially equivalent to that achieved when high resolution, frame rate, and dynamic range characteristics are all achieved, even if they are not all achieved simultaneously. An image sensor using a pixel structure described in Patent Document 1 is known as a technology for achieving such imaging.

[0007] Furthermore, the imaging element described in Non-Patent Document 1 has a pixel structure similar to that of Patent Document 1, and discloses a technique for shortening or extending the exposure time for each pixel group. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Publication No. 2022-123539 [Non-patent literature]

[0009] [Non-Patent Document 1] Kohei Tomioka, et.al., “Feedback control of Block-wise-controlled Image Sensor Based on Brightness Distribution Analysis”, 2023 International Image Sensor Workshop (IISW 2023), P25, 2023 Summary of the Invention [Problem to be solved by the invention]

[0010] An image sensor using the pixel structure described in Patent Document 1 (hereinafter referred to as an area control image sensor) is capable of locally controlling the drive method (hereinafter referred to as the drive mode) of a pixel signal readout circuit. Using an image capture device using this area control image sensor, for example, if a unit pixel group (hereinafter referred to as a control block) capable of switching drive modes is 2×2 pixels, it is conceivable to switch between four drive modes: high-resolution mode, high-speed mode, high-brightness mode, and low-brightness mode. The high-resolution mode is a drive mode in which binning is disabled and 2×2 pixels are read out at a predetermined scanning speed. The high-speed mode is a drive mode in which binning of 2×2 pixels is enabled and the binned signals are read out at a predetermined scanning speed, thereby achieving image capture at half the horizontal and vertical resolution and four times the frame rate of the high-resolution mode. The high-brightness mode is a drive mode in which the exposure time of the 2×2 pixels is shortened using an electronic shutter to suppress pixel saturation and blooming. The low-brightness mode is a drive mode in which the exposure time is extended beyond the reference frame rate to improve the SNR.

[0011] These four drive modes can be continuously and appropriately switched in real time to match the characteristics of the subject, for example, using high-resolution mode for areas containing stationary, finely detailed objects, high-speed mode for areas containing fast-moving objects, high-brightness mode for areas containing high-brightness objects, and low-brightness mode for areas containing low-brightness objects. By switching drive modes within the same image in this way, it is possible to capture images with subjective image quality equivalent to that of an imager that simultaneously achieves high resolution, frame rate, and dynamic range, even if the imager does not simultaneously achieve all of these characteristics. In addition, the pixel readout speed of an area-control image sensor can be suppressed to less than the pixel readout speed of an imager that simultaneously achieves high resolution, frame rate, and dynamic range (in this example, it is less than one-quarter of the pixel readout speed of an imager that simultaneously achieves 2 x 2 pixels and a four-time frame rate). Because the resolution, frame rate, and sensitivity of an image captured in this way vary from region to region, displaying the image requires spatiotemporal interpolation and sensitivity normalization according to the resolution and frame rate of the display device.

[0012] In the area control image sensor described above, an image analysis unit is provided to select a drive mode for each control block. The image analysis unit selects the optimal drive mode for each control block by analyzing the brightness of each control block of the image from the area control image sensor, the characteristics of spatial frequency, and the arrangement tendency of areas with high spatial frequency. However, Patent Document 1 does not specify the specific analysis procedure.

[0013] Furthermore, the above-mentioned Patent Document 1 and Non-Patent Document 1 do not specify a specific analysis procedure for determining the high-speed mode. In order to suitably select a drive mode including the high-speed mode, there has been a demand for generating scene information that indicates the degree of brightness and the degree of movement.

[0014] In view of the above-mentioned problems, the present invention provides a scene information generating device and an imaging device that are capable of generating scene information. [Means for solving the problem]

[0015] [A1] In order to solve the above problem, one aspect of the present invention is a brightness map generating device comprising: a pixel information acquisition unit that acquires pixel information captured by an imaging element having a pixel block having a plurality of pixels and a floating diffusion shared by the plurality of pixels, and a control circuit that can independently control a drive mode for each of the pixel blocks or for each of a plurality of pixel blocks; and a brightness map generation unit that determines one of at least three levels of brightness depending on the number of pixels included in the pixel block whose pixel value is greater than a predetermined threshold, and generates a brightness map indicating the level of brightness for each of the pixel blocks.

[0016] [A2] Also, one aspect of the present invention is that in the brightness map generating device described in [A1] above, the brightness map generating unit determines that a pixel block is bright if, among the multiple pixels included in the pixel block, the number of pixels whose pixel values ​​are greater than a first threshold is a predetermined number or more, determines that the pixel block is dark if, among the multiple pixels included in the pixel block, the number of pixels whose pixel values ​​are less than a second threshold is a predetermined number or more, and determines that a pixel block that is neither bright nor dark is neutral.

[0017] [A3] Also, one aspect of the present invention is a brightness map generating device described in [A1] or [A2] above, wherein the brightness map generating unit corrects acquired pixel information according to information generated based on past frame images among frame images captured successively by the imaging element, and generates the brightness map according to the corrected pixel information.

[0018] [A4] Also, one aspect of the present invention is a brightness map generating device described in any one of [A1] to [A3] above, wherein the brightness map is generated for each of a plurality of frames to be binned, and the image sensor controls the driving mode when capturing images from the n+1th frame onwards in accordance with the brightness map generated in accordance with the frame image of the nth frame (n is a natural number greater than or equal to 1).

[0019] [A5] Also, one aspect of the present invention is a brightness map generating device as described in [A4] above, wherein the pixel information acquisition unit acquires at least green pixel information among red, green, and blue pixels, the brightness map generation unit generates the brightness map based on the acquired green pixel information, and the imaging element controls the driving mode when capturing images from the n+1th frame onwards in accordance with the brightness map generated based on the green pixel information for all red, green, and blue.

[0020] [A6] Also, one aspect of the present invention is a luminance map generating device as described in [A4] above, wherein the pixel information acquisition unit acquires pixel information for each of red, green, and blue pixels among the red, green, and blue pixels, and the luminance map generation unit generates provisional luminance maps for each of red, green, and blue based on the acquired pixel information for each of red, green, and blue, and generates final luminance maps based on the judgment for pixel blocks that have been identically determined in all of the provisional luminance maps for red, green, and blue, and generates final luminance maps by making intermediate judgments for pixel blocks that have been differently determined in the provisional luminance maps for red, green, and blue.

[0021] [A7] Also, one aspect of the present invention is a brightness map generating device described in [A4] above, wherein the pixel information acquisition unit acquires pixel information for each of red, green, and blue pixels among the red, green, and blue pixels, and the brightness map generation unit generates provisional brightness maps for each of red, green, and blue based on the acquired pixel information for each of red, green, and blue, and generates a final brightness map by determining that pixel blocks that are determined to be bright in any of the three channels of red, green, and blue are bright, determining that pixel blocks that are determined to be dark in all three channels of red, green, and blue are dark, and determining that pixel blocks that are neither bright nor dark are intermediate.

[0022] [A8] Another aspect of the present invention includes an imaging element having a pixel block having a plurality of pixels and a floating diffusion shared by the plurality of pixels, and a control circuit capable of independently controlling a drive mode for each pixel block or for each of a plurality of pixel blocks, and a brightness map generation device described in any of [A1] to [A7] above that generates the brightness map based on pixel information captured by the imaging element, wherein the imaging element controls the drive mode when capturing images from the (n+1)th frame onwards based on the brightness map generated in accordance with the frame image of the nth frame.

[0023] [B1] Another aspect of the present invention is a scene information generating device comprising: a brightness map generating unit that generates a brightness map indicating a degree of brightness for each pixel block based on pixel information captured by a first imaging element, the first imaging element having a plurality of pixel blocks, each pixel block having a plurality of pixels and a floating diffusion shared by the plurality of pixels, and a control circuit that can independently control a drive mode for each of the pixel blocks or for each of the plurality of pixel blocks; a motion map generating unit that generates a motion map indicating a degree of change from a previous frame image based on pixel information captured by a second imaging element different from the first imaging element; and a scene information generating unit that generates, for each pixel block, scene information indicating the brightness or degree of change indicated in the brightness map generated by the brightness map generating unit or the degree of change indicated in the motion map generated by the motion map generating unit.

[0024] [B2] Also, one aspect of the present invention is that, in the scene information generating device described in [B1] above, the first imaging element is an image sensor that converts incident visible light into an electrical signal, and the second imaging element is an infrared sensor that converts incident infrared light into an electrical signal.

[0025] [B3] Furthermore, in one aspect of the present invention, in the scene information generating device described in [B1] or [B2] above, the second imaging element is an event vision sensor that detects whether or not there is a change in movement in accordance with a change in the amount of incident light.

[0026] [B4] Furthermore, one aspect of the present invention is a scene information generating device described in any one of [B1] to [B3] above, wherein the motion map generating unit determines that pixels included in the pixel information captured by the second imaging element, which correspond to pixels included in a pixel block of the first imaging element, are moving if the number of pixels having a change in motion is a predetermined number or more, and determines that pixels are still if the number is less than the predetermined number.

[0027] [B5] Also, one aspect of the present invention is a scene information generating device described in any of [B1] to [B4] above, wherein the brightness map generating unit generates the brightness map indicating one of the states of bright, dark, or intermediate for each pixel block, and the scene information generating unit generates the motion map indicating one of the states of bright, dark, intermediate, or motion, and the motion determination is given priority over the bright, dark, and intermediate determinations.

[0028] [B6] Furthermore, one aspect of the present invention is a scene information generating device described in any one of [B1] to [B5] above, wherein the motion map generating unit generates the motion map using blocks whose side length is the least common multiple of the side length of the pixel blocks of the first imaging element and the side length of the pixels of the second imaging element.

[0029] [B7] Furthermore, one aspect of the present invention is that in the scene information generating device described in [B6] above, the scene information generating unit determines that, if there is at least a pixel in the block that has been determined to be moving, the pixels included in the block that includes that pixel are determined to be moving.

[0030] [B8] Another aspect of the present invention is an imaging device comprising a first imaging element that converts incident visible light into an electrical signal, a second imaging element that converts incident infrared light into an electrical signal, and a scene information generating device according to any one of [B1] to [B7] above, which generates the scene information based on the brightness map generated based on pixel information captured by the first imaging element and the motion map generated based on pixel information captured by the second imaging element.

[0031] [B9] Furthermore, one aspect of the present invention is an imaging device according to the above [B8], further comprising a prism that separates visible light from the incident light into the first imaging element and infrared light into the second imaging element. [Effects of the Invention]

[0032] According to the present invention, it is possible to provide a scene information generating device and an imaging device capable of generating scene information. [Brief explanation of the drawings]

[0033] [Figure 1] FIG. 1 is a functional configuration diagram illustrating an example of a functional configuration of an imaging system according to an embodiment. [Figure 2] FIG. 2 is a functional configuration diagram showing an example of the detailed functional configuration of a scene information / image acquisition board and a signal processing board according to the present embodiment. [Figure 3] 10 is a diagram for explaining the processing flow of a brightness map generation unit, a motion map generation unit, and a scene information generation unit according to the present embodiment. FIG. [Figure 4] FIG. 4 is a diagram for explaining an example of a luminance map according to the embodiment. [Figure 5] 10 is a flowchart showing an example of a series of steps in a brightness map generation method according to the present embodiment. [Figure 6] 10A and 10B are diagrams for explaining an example of a brightness mode determination method according to the embodiment. [Figure 7] 10 is a flowchart showing an example of a series of steps in a motion map generation method according to the present embodiment. [Figure 8] 10 is a flowchart showing an example of a sequence of steps in a scene information generating method according to the present embodiment. [Figure 9] FIG. 10 is a first diagram for explaining an example of processing when the image sensor according to the present embodiment and the sub-sensor have different sizes. [Figure 10] FIG. 2 is a second diagram for explaining an example of processing when the image sensor according to the present embodiment and the sub-sensor have different sizes. [Figure 11] FIG. 2 is a block diagram showing an example of the internal configuration of the imaging device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0034] [Embodiment] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that the present invention is not limited to these embodiments and includes various modifications and improvements. In other words, the components described below include those that would be easily conceivable to a person skilled in the art or that are substantially identical, and the components described below can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the present invention. Furthermore, in the drawings, the scale and number of components may differ from the scale and number of the actual structures to make each configuration easier to understand.

[0035] 1 is a functional configuration diagram showing an example of the functional configuration of an imaging system according to an embodiment. First, with reference to the diagram, an imaging system 1 according to this embodiment will be described. The imaging system 1 includes an imaging device 3 and a video display device 5.

[0036] The imaging device 3 captures an image. A single image captured by the imaging device 3 may contain multiple subjects with different characteristics. An example of multiple subjects with different characteristics is a stationary object and a fast-moving object. That is, a single image captured by the imaging device 3 may contain a mixture of areas where stationary objects exist and areas where fast-moving objects exist. The imaging device 3 sets suitable imaging conditions for each of these multiple areas, thereby optimally capturing images of multiple subjects with different characteristics.

[0037] Specifically, the imaging device 3 includes a lens 31, a prism 32, a sub-sensor 33, an image sensor 34, a sensor drive board 35, a sensor power supply board 36, a scene information / image acquisition board 37, a signal processing board 38, and a power supply module 39. The following description will be given of a case in which the imaging device 3 captures a color image. In this case, the image sensor 34, the sensor drive board 35, the sensor power supply board 36, and the scene information / image acquisition board 37 are provided for each of the colors R (Red), G (Green), and B (Blue). This embodiment is not limited to this example, and can also be applied to capturing monochrome images, grayscale images, and multispectral images.

[0038] The lens 31 guides incident light to the prism 32. The figure shows the optical axis OA of the light incident on the lens 31. The lens 31 is preferably a wide-angle lens or an ultra-wide-angle lens in order to simultaneously capture images of multiple subjects with different characteristics. In this embodiment, lenses 31 with various angles of view can be used, and the angle of view is not limited in any way. A turret-type structure may also be provided to allow selection of multiple lenses with different angles of view.

[0039] The prism 32 splits the incident light according to the wavelength of the light. For example, the prism 32 may split the incident light into infrared light and send it to the sub-sensor 33, and visible light and send it to the image sensor 34. The prism 32 may also split the light into multiple image sensors 34 according to the wavelengths of each of the R, G, and B components. In the example shown in the figure, the prism 32 guides light having a wavelength corresponding to the G component to the image sensor 34G, guides light having a wavelength corresponding to the B component to the image sensor 34B, and guides light having a wavelength corresponding to the R component to the image sensor 34R.

[0040] The sub-sensor 33 is a sensor provided auxiliary to the image sensor 34 that generates an image. The sub-sensor 33 may be, for example, an event detection sensor that detects changes in luminance of each pixel arranged on a two-dimensional coordinate system. The sub-sensor 33 may be, for example, a distance sensor (Time-of-Flight Sensor: ToF sensor) that detects distance according to the timing of receiving infrared light emitted from the imaging device 3.

[0041] In the illustrated example, a sensor that detects infrared light is used as the sub-sensor 33, but this embodiment is not limited to this example. For example, the sub-sensor 33 may be configured to detect visible light. By separating (assigning) infrared light that is not necessary for detecting an image to the sub-sensor 33, it is possible to obtain the effect of reducing image degradation, so it is preferable that the sub-sensor 33 be a sensor that detects changes in the luminance of infrared light.

[0042] The image sensor 34 is an image sensor that detects the brightness of visible light emitted from the prism 32. In the example shown in the figure, an image sensor 34 for each of the R, B, and C colors is provided. Specifically, image sensor 34G is shown as a sensor that detects green (G), image sensor 34B is shown as a sensor that detects blue (B), and image sensor 34R is shown as a sensor that detects red (R).

[0043] The image sensor 34 has multiple pixel blocks. Each pixel block includes at least multiple pixels and one floating diffusion. A pixel block may include, for example, 2×2=4 pixels or 4×4=16 pixels. According to this embodiment, it is possible to determine whether or not to perform binning for each pixel block. When binning is performed, i.e., when pixel values ​​of multiple pixels are simultaneously extracted, the resolution in the three-dimensional direction can be improved (i.e., a higher frame rate can be achieved) at the expense of the resolution in the two-dimensional direction. When binning is not performed, i.e., when pixel values ​​of multiple pixels are individually extracted, the resolution in the two-dimensional direction can be improved at the expense of the resolution in the three-dimensional direction (i.e., the frame rate). The image sensor 34 does not perform binning in areas where stationary objects exist, but performs binning in areas where fast-moving objects exist, thereby enabling imaging with a mixture of areas where binning is performed and areas where binning is not performed.

[0044] The sensor drive boards 35 are provided corresponding to the image sensors 34. In the example shown, a sensor drive board 35G is provided corresponding to the image sensor 34G, a sensor drive board 35B is provided corresponding to the image sensor 34B, and a sensor drive board 35R is provided corresponding to the image sensor 34R. The sensor drive boards 35 control the image sensors 34 in accordance with the drive mode and read out pixel values ​​from the image sensors 34.

[0045] Sensor power supply board 36 supplies power to sensor drive board 35. Sensor power supply board 36 may be provided corresponding to sensor drive board 35. In the example shown in the figure, sensor power supply board 36G is provided corresponding to sensor drive board 35G, sensor power supply board 36B is provided corresponding to sensor drive board 35B, and sensor power supply board 36R is provided corresponding to sensor drive board 35R.

[0046] The scene information and image acquisition boards 37 are provided corresponding to the sensor drive boards 35. In the example shown in the figure, a scene information and image acquisition board 37G is provided corresponding to the sensor drive board 35G, a scene information and image acquisition board 37B is provided corresponding to the sensor drive board 35B, and a scene information and image acquisition board 37R is provided corresponding to the sensor drive board 35R.

[0047] The scene information / image acquisition board 37 generates scene information based on pixel values ​​read by the sensor drive board 35. The scene information is generated from a brightness map and a motion map. The brightness map may be generated based on an image captured by the image sensor 34. For example, the brightness map may indicate the degree of brightness for each pixel block. Examples of the degree of brightness include bright determination (Bright), dark determination (Dark), and neutral determination (Normal). The motion map may also be generated based on an image captured by the sub-sensor 33. For example, the brightness map may indicate the presence or degree of motion for each pixel corresponding to a pixel block of the image sensor 34. Examples of the presence or degree of motion include moving determination (Fast) and still determination (Slow). The scene information may indicate the degree of brightness or the degree of motion for each pixel block.

[0048] For example, as an example of scene information, pixel blocks determined to be moving may be determined to be moving, and pixel blocks determined to be still may be determined to be bright, dark, or neutral based on a brightness map.

[0049] Note that one piece of scene information may be generated in common for all RGB colors, or different scene information may be generated for each RGB color. When one piece of scene information is generated in common for all RGB colors, the result of G, which has the greatest influence on the luminance component among the RGB colors, may be prioritized, or one piece of scene information common to all RGB colors may be generated using other logic. The luminance map and scene information may be shared among the RGB colors.

[0050] The signal processing board 38 generates video data based on the pixel values ​​read out by the sensor driving board 35 and the scene information generated by the scene information / video acquisition board 37. Note that the image information captured by the image sensor 34 according to this embodiment includes areas that have been binned and areas that have not been binned, so in order to display the video information on the video display device 5, it is preferable to perform spatiotemporal interpolation processing to generate the video information. However, the video data generated by the signal processing board 38 may be information in a state before the spatiotemporal interpolation processing is performed.

[0051] The power supply module 39 supplies power to each component included in the imaging device 3 .

[0052] The video display device 5 displays the video captured by the imaging device 3. The video display device 5 includes a time-space interpolation processing unit 51, a signal processing unit 52, and a display unit 53.

[0053] The spatiotemporal interpolation processing unit 51 acquires video data and scene information from the imaging device 3. The spatiotemporal interpolation processing unit 51 performs interpolation processing in time and space based on the acquired video data and scene information. Specifically, the spatiotemporal interpolation processing unit 51 interpolates missing pixels in two dimensions or three dimensions based on the video data and scene information. An example of an interpolation method is linear interpolation.

[0054] The signal processing unit 52 generates a video signal based on the information interpolated by the spatio-temporal interpolation processing unit 51 .

[0055] The display unit 53 performs display based on the video signal generated by the signal processing unit 52. The display unit 53 may be, for example, a liquid crystal display, an organic EL (Electroluminescence) display, or the like.

[0056] The video display device 5 does not need to be a device that includes all of the components of the spatiotemporal interpolation processing unit 51, the signal processing unit 52, and the display unit 53, but may be a device in which the spatiotemporal interpolation processing unit 51 and the signal processing unit 52 are added to an existing display unit 53.

[0057] FIG. 2 is a functional configuration diagram showing an example of the detailed functional configuration of the scene information / image acquisition board and signal processing board according to this embodiment. An example of the detailed functional configuration of the scene information / image acquisition board 37 and the signal processing board 38 will be described with reference to the same figure. It should be noted that a scene information / image acquisition board 37 exists for each of the RGB colors. Hereinafter, the configuration that processes the image sensor 34G that receives green light will be referred to as the scene information / image acquisition board 37G, the configuration that processes the image sensor 34B that receives blue light will be referred to as the scene information / image acquisition board 37B, and the configuration that processes the image sensor 34R that receives red light will be referred to as the scene information / image acquisition board 37R. The scene information / image acquisition board 37G, the scene information / image acquisition board 37B, and the scene information / image acquisition board 37R may have similar configurations. In the description given with reference to the same figure, only the scene information / image acquisition board 37G will be described as a representative. It should be noted that when the colors are not to be distinguished, they may simply be referred to as the scene information / image acquisition board 37.

[0058] The scene information / image acquisition board 37 includes a sensor data receiving unit 311, a binarization unit 312, a filtering / expansion processing unit 313, an affine transformation unit 314, a motion map generation unit 315, a sensor data receiving unit 321, an OB clamp unit 322, an FPN cancellation unit 323, a low-illumination mode correction unit 324, a linearity correction unit 325, a brightness map generation unit 331, a scene information generation unit 332, a drive mode transmission unit 333, and a drive mode out control unit 334.

[0059] The sensor data receiving unit 311, binarization unit 312, filtering and expansion processing unit 313, affine transformation unit 314, and motion map generation unit 315 may be provided in any one of the scene information and image acquisition boards 37R for each color. In the example shown in the figure, these components are provided in the scene information and image acquisition board 37G. Alternatively, these components may be provided on a board separate from the scene information and image acquisition board 37.

[0060] The sensor data receiving unit 311 receives sensor data from the sub-sensor 33. The sensor data acquired by the sensor data receiving unit 311 is, for example, information indicating changes in pixel values ​​on each two-dimensional coordinate. The sensor data includes information indicating which of three types of states exists for each coordinate, compared with the pixel values ​​of a past frame image: a change in the direction of increasing the pixel value (positive), a change in the direction of decreasing the pixel value (negative), or no change.

[0061] The binarization unit 312 converts the three types of states of the sensor data into two states: whether there has been a change or not. That is, the binarization unit 312 binarizes the sensor data. The binarization unit 312 may generate a binary event image, for example, in which pixels that have accumulated either positive or negative are colored white, and pixels that have accumulated neither are colored black.

[0062] The filtering and expansion processing unit 313 performs noise removal processing from the information binarized by the binarization unit 312. The filtering and expansion processing unit 313 may perform opening processing, median filter processing, and expansion processing on the event image, for example. The base and filter radius for each processing may be set from an external terminal or the like. Furthermore, the processing performed by the filtering and expansion processing unit 313 may be bypassable.

[0063] The affine transformation unit 314 performs a process of adjusting the size of the image captured by the sub-sensor 33 to the size of the image captured by the image sensor 34. The process performed by the affine transformation unit 314 can also be called an offset process or a trimming process. Furthermore, when the optical axis of the sub-sensor 33 and the optical axis of the image sensor 34 do not completely coincide with each other, the affine transformation unit 314 may adjust the position by performing an affine transformation in addition to or instead of adjusting the image size. The transformation parameters used in the affine transformation may be set from an external terminal or the like. Furthermore, the process performed by the affine transformation unit 314 may be bypassable.

[0064] The motion map generator 315 processes the sensor data captured by the sensor data receiver 311 and subjected to predetermined processing to generate a motion map. The motion map is a traveled map in which the number of events (the number of pixels with changes) is accumulated for each control block, and if the accumulated number of events is greater than a threshold, it is determined to be moving, and if it is less than a threshold, it is determined to be still. The motion map can also be said to be information indicating the degree of change from the immediately preceding frame image. The threshold used for the determination may be set from an external terminal or the like.

[0065] The sensor data receiving unit 321 receives sensor data from the image sensor 34. The sensor data acquired by the sensor data receiving unit 321 is, for example, information indicating changes in pixel values ​​on each two-dimensional coordinate system, i.e., image information. The sensor data includes pixel values ​​for each coordinate system. Here, the image sensor 34 has multiple pixel blocks, each including multiple pixels and a floating diffusion shared by the multiple pixels. Therefore, the sensor data may include information indicating the presence or absence of binning for each pixel block. In the following description, the sensor data receiving unit 321 may be referred to as a pixel information acquiring unit, and the process performed by the sensor data receiving unit 321 may be referred to as a pixel information acquiring process.

[0066] In the following description, the image sensor 34 may be referred to as the first imaging element, and the sub-sensor 33 may be referred to as the second imaging element. The first imaging element may be referred to as an image sensor that converts incident visible light into an electrical signal, and the second imaging element may be referred to as an infrared sensor that converts incident infrared light into an electrical signal. Furthermore, the second imaging element may be referred to as an event vision sensor (EVS) that detects whether or not there is a change in movement according to a change in the amount of incident light.

[0067] The OB clamp unit 322 cancels an increase in dark current caused by a temperature rise or the like by subtracting pixel values ​​based on pixel values ​​in the OB (Optical Black) region. The OB region may be provided in either or both of the vertical and horizontal portions of the image sensor 34. Note that the processing performed by the OB clamp unit 322 may be bypassable.

[0068] The FPN cancellation unit 323 performs a process of canceling fixed pattern noise (FPN) caused by dark current components. The FPN cancellation unit 323 performs an FPN correction process and an update process of FPN data used in the FPN correction process. Note that the process performed by the FPN cancellation unit 323 may be bypassable.

[0069] The low-illuminance mode correction unit 324 corrects image data loss that occurs when the exposure time is extended (the frame rate is reduced) in low-illuminance mode. The correction value used for correction in low-illuminance mode is the image data immediately before the image loss occurs. Note that the processing performed by the low-illuminance mode correction unit 324 may be bypassed.

[0070] The linearity correction unit 325 corrects the linearity of pixel values ​​for each drive mode relative to the amount of incident light of the image, and normalizes the sensitivity difference due to the length of exposure time in high-brightness mode and low-brightness mode. For the linearity correction, known techniques such as gamma correction or correction using a lookup table may be used. Note that the processing performed by the linearity correction unit 325 may be bypassed.

[0071] The luminance map generation unit 331 generates a luminance map. The luminance map is information indicating the degree of brightness of pixel values ​​included in each pixel block. Note that "each pixel block" includes each control block configured including multiple pixels. The degree of brightness indicated in the luminance map may be, for example, three degrees of brightness: bright determination, dark determination, and intermediate determination. The degree of brightness may be determined, for example, according to the number of pixels whose pixel values ​​are greater than a predetermined threshold value or the number of pixels whose pixel values ​​are smaller than a predetermined threshold value among the multiple pixels included in the pixel block to be determined. Note that in the following description, the process performed by the luminance map generation unit 331 may be referred to as a luminance map generation process.

[0072] Here, the degree of brightness in the luminance map may fluctuate from frame to frame when pixel values ​​are near a threshold value. When such fluctuations occur, problems such as flicker may occur in an image captured by the image sensor 34 controlled based on the fluctuating luminance map, resulting in degradation of subjective image quality. Therefore, the luminance map generation unit 331 may correct pixel information (specifically, pixel values) according to scene information of the immediately preceding frame and generate a luminance map according to the corrected pixel information. Specifically, the luminance map generation unit 331 may correct the acquired pixel information by multiplying it by a different coefficient depending on whether the luminance map of the corresponding pixel block in the immediately preceding frame is determined to be bright, dark, or neutral. For example, if the luminance map of the corresponding pixel block in the immediately preceding frame is determined to be bright, the correction coefficient may be multiplied by approximately 1.1 to 1.2; if the luminance map is determined to be dark, the correction coefficient may be multiplied by approximately 0.9 to 0.8; and if the luminance map is determined to be neutral, the correction coefficient may not be multiplied (or, alternatively, the correction coefficient may be multiplied by 1). The correction coefficient may be multiplied by a correction circuit (not shown), which can provide the effect of suppressing fluctuations in drive mode determination.

[0073] The scene information generation unit 332 generates scene information. Scene information is information indicating the brightness or degree of change for each pixel block. The brightness or degree of change included in the scene information may include the brightness levels included in the brightness map, i.e., light, dark, and neutral determinations, as well as a motion determination. The scene information generation unit 332 may generate scene information for each pixel block, for example, by employing the brightness level indicated in the brightness map generated by the brightness map generation unit 331 or the degree of change indicated in the motion map generated by the motion map generation unit 315. Note that in the following description, the process performed by the scene information generation unit 332 may be referred to as a scene information generation process.

[0074] The scene information generated by the scene information generation unit 332 is stored (buffered) in a predetermined temporary storage unit. The scene information generation unit 332 may generate scene information based additionally on scene information from one frame before. By performing correction based additionally on scene information from one frame before, the scene information generation unit 332 can prevent a situation in which the determination results differ from frame to frame when the degree of brightness or the degree of movement is near a threshold value.

[0075] The drive mode transmission unit 333 controls the drive of the image sensor 34 based on the scene information generated by the scene information generation unit 332. The drive mode transmission unit 333 may also control the drive of the image sensor 34 based on an instruction from the drive mode outside control unit 334.

[0076] The out-of-drive mode control unit 334 acquires an operation related to the control of the image sensor 34 from the user, transmits a control signal based on the acquired operation to the drive mode transmission unit 333, and controls the drive mode of the image sensor 34. Note that the out-of-drive mode control unit 334 is not limited to performing control based on user control, and may control the drive mode of the image sensor 34 by other methods.

[0077] It can also be said that the scene information used by the drive mode transmission unit 333 for drive control is information for one frame in the past. In other words, the image sensor 34 controls the drive mode when capturing images of the (n+1)th frame and thereafter (for example, the (n+1)th frame or the (n+2)th frame, etc.) in accordance with the scene information generated in accordance with the frame image of the nth frame. It should be noted that, as described above, the motion map and brightness map are referenced to generate the scene information.

[0078] Since the scene information / image acquisition board 37 generates a luminance map, a device having this function can also be called a luminance map generation device. Also, since the scene information / image acquisition board 37 generates scene information, a device having this function can also be called a scene information generation device.

[0079] The signal processing board 38 includes an event data output unit 381, a paint unit 382, ​​a linear matrix unit 383, a gamma correction unit 384, a detail unit 385, a video output unit 386, and a scene information output unit 387. By including these components, the signal processing board 38 adjusts the color tone, linearity, and contrast of the image.

[0080] The event data output unit 381 transmits the motion map generated by the motion map generation unit 315 to the video display device 5. The event data output unit 381 may convert the motion map generated by the motion map generation unit 315 into a predetermined format and transmit it. For example, the event data output unit 381 may convert the motion map into a video signal using a signal based on the 3G-SDI standard and transmit it.

[0081] The paint unit 382 acquires pixel information for each of the RGB colors from the scene information and image acquisition board 37G, the scene information and image acquisition board 37B, and the scene information and image acquisition board 37R. The paint unit 382 performs painting processing based on the acquired color information for each color.

[0082] The linear matrix unit 383 performs linear matrix correction, which is a correction for reproducing colors that cannot be expressed when dividing into three primary colors using a prism, and may be a known method.

[0083] The gamma correction unit 384 performs gamma correction. Gamma correction is a correction that corrects the gamma characteristics (gamma curve) to bring out details in shadows and highlights while maintaining the overall image quality. A known method may be used for the processing performed by the gamma correction unit 384.

[0084] The detail section 385 performs correction to bring out further details. The processing performed by the detail section 385 may be performed using a known method.

[0085] In addition to the above-mentioned corrections, the signal processing board 38 may also perform aperture correction to correct high-frequency components lost due to the MTF (Modulation Transfer Function) of the optical system or an optical low-pass filter inserted in front of the sensor, and shading correction to make the level uniform across the entire screen.

[0086] The video output unit 386 outputs the video signal obtained as a result of the above-described correction processing. For example, the video output unit 386 may transmit the video signal as a signal based on the 12G-SDI standard.

[0087] The scene information output unit 387 transmits the scene information generated by the scene information generation unit 332 to the video display device 5. The scene information output unit 387 may convert the scene information generated by the scene information generation unit 332 into a predetermined format and transmit it. For example, the event data output unit 381 may convert a motion map into a video signal using a signal based on the 3G-SDI standard and transmit it.

[0088] 3 is a diagram illustrating the processing flow of the brightness map generation unit, the motion map generation unit, and the scene information generation unit according to this embodiment. An example of the generation processing of the brightness map, the motion map, and the scene information will be described with reference to the same drawing.

[0089] First, in the luminance map generation process P11, a luminance map is generated. The luminance map may be generated for each RGB color. The luminance map is generated by determining the degree of brightness for each pixel block. For example, if there are 1040 x 564 pixel blocks, the degree of brightness for each pixel block is determined and a luminance map is generated. Note that the number of pixels included in a pixel block does not have to match the number of pixels included in a control block, which is the unit of the luminance map. In other words, the degree of brightness may be determined for each control block composed of multiple pixel blocks. Note that when luminance maps are generated for each RGB color, in the luminance map generation process P11, they are integrated into a single luminance map based on a predetermined logic, and the integrated luminance map is output to the next process.

[0090] Next, in a motion map generation step P12, a motion map is generated based on pixel information captured by the sub-sensor 33. The size of the control unit of the motion map does not have to match the size of the control unit of the brightness map.

[0091] Next, in a scene information generation step P13, scene information is generated. The scene information is generated based on the brightness map generated in the brightness map generation step P11, the motion map generated in the motion map generation step, and information generated based on past frame images (for example, scene information generated in the immediately preceding frame) among frame images captured successively by the image sensor 34. Note that in the case of the first frame, the immediately preceding scene information does not need to be referenced.

[0092] The process shown in the figure is completed before the timing signal for the final line of the pixel block of the third subframe is issued. In other words, the scene information is updated once every four times the frame period of the subframes (i.e., the normal frame period).

[0093] FIG. 4 is a diagram illustrating an example of a brightness map according to this embodiment. FIG. 4(A) schematically shows some of the pixels of the image sensor 34. The example shown in the figure shows 4×4=16 pixels. For example, in the case of hardware in which one pixel block is composed of 2×2=4 pixels, the degree of brightness is determined for each pixel block. FIG. 4(B) shows the degree of brightness determined for each pixel block using the pixels shown in FIG. 4(A). B indicates a bright judgment (Bright), D indicates a dark judgment (Dark), and N indicates a normal judgment (Normal).

[0094] For example, if the number of pixels in a pixel block whose pixel values ​​are greater than a predetermined first threshold is equal to or greater than a predetermined number, the pixel block may be determined to be bright. Similarly, if the number of pixels in a pixel block whose pixel values ​​are less than a predetermined second threshold is equal to or greater than a predetermined number, the pixel block may be determined to be dark. Furthermore, a pixel block that is neither determined to be bright nor dark may be determined to be neutral.

[0095] It should be noted that the image shown in the figure is merely an example, and a pixel block may be made up of a 3×3 pixel group, or a control block may be made up of a 2×2 pixel block.

[0096] 5 is a flowchart showing an example of a series of steps in the brightness map generation method according to this embodiment. With reference to this drawing, an example of a series of steps in the brightness map generation method performed by the scene information / image acquisition board 37 will be described.

[0097] (Step S11) First, the sensor data receiving unit 321 acquires pixel values ​​of each of the RGB colors.

[0098] (Step S12) Next, the brightness map generation unit 331 multiplies each pixel by a mode-specific correction coefficient based on the brightness map of the previous frame. The correction coefficient can be set arbitrarily. By increasing the correction coefficient, transition to another mode (for example, from dark judgment to medium judgment, or from bright judgment to medium judgment, etc.) can be made less likely to occur, and flicker can be further suppressed.

[0099] (Step S13) Next, the brightness map generation unit 331 calculates the number of pixels in the pixel block to be determined whose pixel value is greater than Bth. If the calculation result is greater than or equal to Bnum, the process proceeds to step S14. If the calculation result is not greater than or equal to Bnum, the process proceeds to step S15. Bth is, for example, a 14-bit integer range, and may be, for example, approximately 300. Bnum may be, for example, approximately 4 pixels out of 16 pixels. Note that Bth is preferably a value obtained by multiplying the digital value obtained when the amount of electrons corresponding to the saturation electron number of a pixel is output from the ADC circuit by an arbitrary coefficient smaller than 1.

[0100] (Step S14) If the number of pixels whose pixel values ​​are greater than Bth in the pixel block to be judged is equal to or greater than Bnum, the pixel block is judged to be bright (Bright).

[0101] (Step S15) Next, the brightness map generation unit 331 calculates the number of pixels in the pixel block to be determined whose pixel value is greater than Dth. If the calculation result is greater than or equal to Dnum, the process proceeds to step S16. If the calculation result is not greater than or equal to Dnum, the process proceeds to step S17. Dth is, for example, a 14-bit integer range, and may be, for example, about 5. Dnum may be, for example, about 4 pixels out of 16 pixels. Note that Dth is desirably a value greater than the value obtained by adding the average value of the dark noise of the pixels to the variance of the dark noise.

[0102] (Step S16) If the number of pixels whose pixel values ​​are greater than Dth in the pixel block to be determined is equal to or greater than Dnum, the pixel block is determined to be dark (Dark).

[0103] (Step S17) Blocks that are neither bright nor dark are determined to be normal.

[0104] (Step S18) Finally, the luminance map generating unit 331 generates a luminance map according to the degree of brightness (bright judgment, dark judgment, or intermediate judgment) determined for each pixel block.

[0105] FIG. 6 is a diagram for explaining an example of a luminance mode determination method according to this embodiment. An example of the luminance mode determination method will be described with reference to the same figure. When a luminance map is generated for each of the RGB colors, the luminance maps may be integrated according to a predetermined logic. The integrated luminance map may be used to generate scene information.

[0106] Below, three methods for integrating brightness maps will be explained.

[0107] A first method is to prioritize Gch (green). This is because green has the highest contribution to the luminance component. In this case, a luminance map is generated based on the Gch image, and each of the RGB colors is captured according to the luminance map (or according to scene information generated based on the luminance map). In other words, the sensor data receiving unit 321 acquires green pixel information for the nth frame from among the red, green, and blue pixels, the luminance map generating unit 331 generates a luminance map based on the acquired green pixel information, and the image sensor 34 controls the drive mode for capturing images from the n+1th frame onward according to the luminance map generated based on the green pixel information for all of the red, green, and blue colors.

[0108] A second method is to AND the results of each RGB color to generate an integrated luminance map. In this case, if all RGB colors are determined to be bright, the result is determined to be bright; if all RGB colors are determined to be dark, the result is determined to be dark; and otherwise, the result is determined to be intermediate. In other words, the sensor data receiving unit 321 acquires pixel information for each of the red, green, and blue pixels, and the luminance map generating unit 331 generates provisional luminance maps for each of the red, green, and blue colors based on the acquired pixel information for each of the red, green, and blue colors. For pixel blocks that are determined to be identical in all the provisional luminance maps for red, green, and blue, the final luminance map is generated based on the determination. For pixel blocks that are determined to be different in the provisional luminance maps for red, green, and blue, the final luminance map is generated by determining intermediate.

[0109] A third method is to generate an integrated luminance map by using OR for bright judgments and AND for dark stability. In this case, a bright judgment is made when one or more of the RGB channels are bright, a dark judgment is made when all of the RGB channels are dark, and an intermediate judgment is made otherwise. In other words, the sensor data receiving unit 321 acquires pixel information for each of the red, green, and blue pixels, and the luminance map generating unit 331 generates provisional luminance maps for each of the red, green, and blue pixels based on the acquired pixel information for each of the red, green, and blue channels. A pixel block that is brightly judged in any of the red, green, and blue channels is judged as bright, a pixel block that is darkly judged in all three red, green, and blue channels is judged as dark, and a pixel block that is neither bright nor dark is judged as intermediate, thereby generating a final luminance map. This method can prevent saturation of pixel values.

[0110] 6 shows the pixels of each of RGB, a luminance map, and an integrated luminance map. For example, if the upper left block of the luminance map is block 1, and blocks 2, 3, and 4 are arranged clockwise, then for Gch, block 1 is determined to be bright (B), block 2 is determined to be bright (B), block 3 is determined to be dark (D), and block 4 is determined to be bright (B). For Bch, block 1 is determined to be dark (D), block 2 is determined to be bright (B), block 3 is determined to be dark (D), and block 4 is determined to be bright (B). For Rch, block 1 is determined to be dark (D), block 2 is determined to be neutral (N), block 3 is determined to be dark (D), and block 4 is determined to be bright (B).

[0111] Consider the case where an integrated luminance map is generated by the second method described above when the luminance maps for each RGB color are as shown in the figure. Blocks 1 and 2 are determined differently for each RGB color, resulting in a neutral determination (N). Block 3 is determined to be dark (D), which is the same as block 1. Block 4 is determined to be bright (B), which is the same as block 4.

[0112] 7 is a flowchart showing an example of a series of steps in the motion map generation method according to this embodiment. With reference to this figure, an example of a series of steps in the motion map generation method performed by the scene information and video acquisition board 37 will be described.

[0113] (Step S21) First, the sensor data receiving unit 311 acquires sensor data from the sub-sensor 33. The sensor data can also be considered as event data indicating whether or not there is a change in pixel value.

[0114] (Step S22) Next, the motion map generator 315 determines whether the number of events (white) included in the block to be determined, i.e., the number of pixels indicating that there is motion, is equal to or greater than a predetermined threshold value Fnum. If it is equal to or greater than Fnum, the process proceeds to step S23. If it is not equal to or greater than Fnum, the process proceeds to step S24.

[0115] Here, there are cases where the number of pixels of the sub-sensor 33 does not match the number of pixels of the image sensor 34. When the number of pixels does not match, the motion map generator 315 performs motion determination using a range corresponding to a pixel block of the image sensor 34 as a block. The motion map generator 315 can also determine that, for pixels included in the pixel information captured by the sub-sensor 33 and corresponding to pixels included in the pixel blocks of the image sensor 34, the number of pixels with a change in motion is a predetermined number or more, and determines that the image is still if the number is less than the predetermined number.

[0116] (Step S23) A pixel block in which the number of pixels indicating that there is motion is equal to or greater than Fnum is determined to be motion.

[0117] (Step S24) A pixel block in which the number of pixels indicating that there is motion is not equal to or greater than Fnum is determined to be still.

[0118] (Step S25) Finally, the motion map generating unit 315 generates a motion map according to the degree of motion (motion determination or stillness determination) determined for each pixel block.

[0119] 8 is a flowchart showing an example of a sequence of steps in the scene information generating method according to this embodiment. With reference to the drawing, an example of a sequence of steps in the scene information generating method performed by the scene information / image acquisition board 37 will be described.

[0120] (Step S31) First, a motion map indicating whether each block is determined to be moving or still is generated by the motion map generation unit 315. The specific method for generating the motion map has been described with reference to FIG. 7 etc.

[0121] (Step S32) Next, a brightness map indicating whether each block is determined to be bright, dark, or intermediate is generated by the brightness map generation unit 331. The specific method for generating the brightness map has been described with reference to FIG. 5, etc.

[0122] (Step S33) Finally, the scene information generation unit 332 generates scene information based on the motion map and brightness map. The scene information indicates one of the following states: bright, dark, neutral, or motion. Here, the scene information may prioritize motion determination from the motion map, and for blocks that are static, determination may be made based on the brightness map. In other words, motion determination takes priority over bright, dark, and neutral determinations.

[0123] 9 is a first diagram for explaining an example of processing when the image sensor according to this embodiment and the sub-sensor have different sizes. Here, depending on the type of the sub-sensor 33 and the image sensor 34, the sizes may differ as shown in the figure.

[0124] Fig. 9(A) shows an example of the size of the image sensor 34. For example, the overall size of the sensor is 9.76 [mm] x 5.56 [mm], the pixel size is 2.5 [μm] x 2.5 [μm], and the number of pixels is 3904 x 2224. Fig. 9(B) shows an example of the size of the sub-sensor 33. For example, the overall size of the sensor is 9.6 [mm] x 7.2 [mm], the pixel size is 15 [μm] x 15 [μm], and the number of pixels is 640 x 480.

[0125] 9(C), offset processing is performed to align the centers of the entire sensors, thereby associating the pixels of the sub-sensor 33 with those of the image sensor 34. This processing may be performed by the affine transformation unit 314.

[0126] 10 is a second diagram for explaining an example of processing when the image sensor and the sub-sensor according to this embodiment have different sizes. When the pixel sizes are different as in the example shown in FIG. 9, it is preferable to consider the block size of the scene information in consideration of the pixel sizes of both sensors.

[0127] FIG. 10(A) shows an example of the pixel size and pixel block size of the image sensor 34. The pixel size is 2.5 μm × 2.5 μm, and a pixel block is composed of 4 × 4 = 16 pixels. FIG. 10(B) shows an example of the pixel size of the sub-sensor 33. The pixel size is 15 μm × 15 μm. In this case, scene information is generated in units of blocks whose side length is the least common multiple of the side length of the pixel blocks in the image sensor 34 and the side length of the pixels in the sub-sensor 33. A motion map may also be generated in units of these blocks. FIG. 10(C) shows an example of a block of scene information or a motion map.

[0128] When such a configuration is adopted, the scene information may contain a mixture of multiple blocks determined to be moving and blocks indicating the degree of brightness. In such a case, if a block contains at least one pixel determined to be moving, the scene information generation unit 332 may determine that the pixels included in the block containing that pixel are also determined to be moving.

[0129] FIG. 11 is a block diagram showing an example of the internal configuration of an imaging device according to this embodiment. At least some of the functions of the configuration of the imaging device 3 can be implemented using a computer. As shown in the figure, the computer includes a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, and a bus 906. The computer itself can be implemented using existing technology. The central processing unit 901 executes instructions contained in a program read from the RAM 902 or the like. In accordance with each instruction, the central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with external input / output devices. The input / output devices 904 and 905 are input / output devices. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used within the computer. For example, the central processing unit 901 reads and writes data from and to the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906. All or part of the functional units provided in the imaging device 3 may be realized using hardware such as an ASIC, a PLD, or an FPGA. All or part of the functional units may be realized by a combination of software and hardware.

[0130] [Summary of the embodiment] According to the embodiment described above, the brightness map generation device according to this embodiment includes a sensor data receiving unit 321 and a brightness map generating unit 331. The sensor data receiving unit 321 acquires pixel information captured by an image sensor 34 having a plurality of pixel blocks, each of which has a plurality of pixels and a floating diffusion shared by the plurality of pixels. The brightness map generating unit 331 determines one of at least three levels of brightness depending on the number of pixels, among the plurality of pixels included in a pixel block, whose pixel value is greater than a predetermined threshold, and generates a brightness map indicating the level of brightness for each pixel block.

[0131] According to this embodiment, it is possible to achieve, in a simple manner, appropriate drive mode determination and switching according to the luminance of the subject within the control block, thereby expanding the dynamic range of the imaging device 3. Furthermore, by multiplying by a correction coefficient before drive mode determination, if there is no significant change in luminance information after the drive mode has been switched once, control operates to maintain the current drive mode, making it possible to suppress image quality degradation such as flicker caused by fluctuations in the drive mode.

[0132] Furthermore, according to the above-described embodiment, the scene information generating device according to this embodiment includes a luminance map generating unit 331, a motion map generating unit 315, and a scene information generating unit 332. The luminance map generating unit 331 generates a luminance map indicating the degree of brightness for each pixel block based on pixel information captured by an image sensor 34 having a plurality of pixel blocks, each pixel block having a plurality of pixels and a floating diffusion shared by the plurality of pixels. The motion map generating unit 315 generates a motion map indicating the degree of change from the previous frame image based on pixel information captured by a sub-sensor 33 different from the image sensor 34. The scene information generating unit 332 generates scene information indicating the brightness or degree of change for each pixel block by adopting the degree of brightness indicated in the luminance map generated by the luminance map generating unit 331 or the degree of change indicated in the motion map generated by the motion map generating unit 315 for each pixel block.

[0133] According to this embodiment, it is possible to easily determine and switch the appropriate drive mode according to the brightness and movement of the subject within the control block, thereby improving the dynamic range and dynamic resolution of the imaging device.

[0134] In the past, background subtraction has been well known as a method for determining the movement of a subject, i.e., a method for detecting a moving object within a screen. However, because background subtraction involves calculating the difference between multiple frames, a delay of one or more frames occurs in detecting the position of a moving object. In this case, the high-speed mode of the control block is switched according to the delayed moving object position information, and the mode switching cannot keep up with a fast-moving object. As a result, problems such as the inability to suppress the occurrence of motion blur and the erroneous reduction of spatial resolution in the background area (stationary objects) have occurred. Furthermore, because the driving mode of an area control image sensor can change from frame to frame, the background subtraction method may calculate the difference between different driving modes, leading to the problem of erroneous detection of a moving object. This embodiment can also solve these problems.

[0135] It should be noted that the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects in addition to or in place of the above-described effects that would be apparent to those skilled in the art from the description of this specification. Furthermore, the present invention is not limited to these embodiments, and various modifications and substitutions can be made within the scope of the present invention. [Explanation of symbols]

[0136] 1. Imaging system 3. Imaging device 5. Video display devices 31 Lens 32 Prism 33 Sub-sensor 34 Image Sensor 35 Sensor drive board 36 Sensor power supply board 37 Scene information and image acquisition board 38 Signal Processing Board 39 Power Supply Module 51 Spatiotemporal Interpolation Processing Unit 52 Signal processing section 53 Display section

Claims

1. a brightness map generator that generates a brightness map indicating the degree of brightness for each pixel block based on pixel information captured by a first imaging element, the first imaging element having a plurality of pixel blocks, each pixel block having a plurality of pixels and a floating diffusion shared by the plurality of pixels, and a control circuit that can independently control a drive mode for each pixel block or for each of the plurality of pixel blocks; a motion map generator that generates a motion map indicating the degree of change from a previous frame image based on pixel information captured by a second imaging element different from the first imaging element; a scene information generation unit that generates, for each pixel block, scene information indicating the brightness or the degree of change indicated in the brightness map generated by the brightness map generation unit or the degree of change indicated in the motion map generated by the motion map generation unit; A scene information generating device comprising:

2. the first imaging element is an image sensor that converts incident visible light into an electrical signal; The second imaging element is an infrared sensor that converts incident infrared light into an electrical signal. The scene information generating device according to claim 1 .

3. The second imaging element is an event vision sensor that detects whether or not there is a change in movement according to a change in the amount of incident light. The scene information generating device according to claim 2 .

4. the motion map generation unit determines, for pixels included in the pixel information captured by the second imaging element and corresponding to pixels included in a pixel block of the first imaging element, that the number of pixels having a change in motion is a predetermined number or more, and determines the pixels to be still if the number is the predetermined number or less; The scene information generating device according to claim 3 .

5. the luminance map generation unit generates the luminance map indicating one of a bright determination, a dark determination, and an intermediate determination for each of the pixel blocks; the scene information generation unit generates the motion map indicating any one of the bright determination, the dark determination, the intermediate determination, and the motion determination; The motion judgment has priority over the light judgment, the dark judgment, and the middle judgment. The scene information generating device according to claim 4 .

6. the motion map generation unit generates the motion map in units of blocks having a side length that is the least common multiple of the side length of the pixel blocks included in the first imaging element and the side length of the pixels included in the second imaging element. The scene information generating device according to claim 5 .

7. When at least a pixel determined to be moving is present in the block, the scene information generation unit determines that pixels included in the block including the pixel are moving. The scene information generating device according to claim 6 .

8. a first imaging element that converts incident visible light into an electrical signal; a second image sensor that converts incident infrared light into an electrical signal; a scene information generating device according to claim 1 , which generates the scene information based on the luminance map generated based on pixel information captured by the first imaging element and the motion map generated based on pixel information captured by the second imaging element; and An imaging device comprising:

9. a prism that separates visible light from the incident light into the first image sensor and infrared light into the second image sensor; The imaging device according to claim 8 .

Images