Sensor device and method for operating a sensor device

Abstract

A sensor device (100) comprises an imaging unit (110) that is configured to capture a plurality of raw images of a scene with different sensitivities for brightness and an image processing unit (120) that is configured to generate high dynamic range, HDR, images from the raw images. The imaging unit (110) is configured to capture a first set (112) of raw images of the scene with different sensitivities for brightness using a first exposure period (t1) and to capture a second set of raw images (114) of the scene with different sensitivities for brightness using a second exposure period (t2). The image processing unit (120) is configured to generate a first HDR image (122) from the first set (112) of raw images and a second HDR image (124) from the second set (114) of raw images. The first exposure period (t1) is longer than the second exposure period (t2). The highest sensitivity for brightness of the raw images of the first set is higher than the highest sensitivity for brightness of the raw images of the second set.

Description

[0001] SENSOR DEVICE AND METHOD FOR OPERATING A SENSOR DEVICE

[0002] FIELD OF THE INVENTION

[0003] The present technology relates to a sensor device, an image processing system, and a method for operating a sensor device, in particular, to a sensor device and a method for operating a sensor device that allows an improved generation of high dynamic range, HDR, images.

[0004] BACKGROUND

[0005] Presently, the generation of HDR images does not take into account situations where the generated HDR images are intended to be used for human vision, HV, tasks, i.e. tasks that involve inspection of the HDR image by a human, as well as computer vision, CV, tasks, i.e. tasks that are automatically processed by a computer without the involvement of a human. This typically means that HDR images are provided that are either suitable for HV tasks or for CV tasks. Improved sensor devices and methods for operating these sensor devices are desirable that mitigate this problem.

[0006] SUMMARY OF INVENTION

[0007] To this end, a sensor device is provided that comprises an imaging unit that is configured to capture a plurality of raw images of a scene with different sensitivities for brightness and an image processing unit that is configured to generate high dynamic range, HDR, images from the raw images. Here, the imaging unit is configured to capture a first set of raw images of the scene with different sensitivities for brightness using a first exposure period and to capture a second set of raw images of the scene with different sensitivities for brightness using a second exposure period. The image processing unit is configured to generate a first HDR image from the first set of raw images and a second HDR image from the second set of raw images. The first exposure period is longer than the second exposure period and the highest sensitivity for brightness of the raw images of the first set is higher than the highest sensitivity for brightness of the raw images of the second set.

[0008] Further, a method for operating such a sensor device is provided, the method comprising: capturing, by the imaging unit, a first set of raw images of the scene with different sensitivities for brightness using a first exposure period and a second set of raw images of the scene with different sensitivities for brightness using a second exposure period; and generating, by the image processing unit, a first HDR image from the first set of raw images and a second HDR image from the second set of raw images. Here, the first exposure period is longer than the second exposure period and the highest sensitivity for brightness of the raw images of the first set is higher than the highest sensitivity for brightness of the raw images of the second set.

[0009] In the above, two different HDR images are generated for a single scene by using different exposure periods and different raw images. The first HDR image is generated based on an exposure period that is longer than the exposure period used to generate the second HDR image. This means that the first HDR image provides more information for dark regions but contains more blurring than the second HDR image. In addition, the first HDR image is based on a set of raw images that comprises at least one image with a higher sensitivity for brightness, i.e. a higher capability to obtain information from dark regions, than any of the raw images from which the second HDR image is generated. This makes the first HDR image particularly useful for HV tasks, since human vision relies on a certain sensitivity for dark regions, while a given amount of blurring is usually acceptable. The second HDR image will be very sharp due to the smaller exposure time and will be particularly sensitive to bright regions due to the comparably reduced brightness sensitivity of the raw images from which the second HDR image is constituted. This makes the second HDR image suited for computer vision tasks that need sharp images for edge detection, object detection, segmentation and the like. Thus, HDR images of a single scene that are suited for HV tasks and CV tasks can be provided in parallel.

[0010] BRIEF DESCRIPTION OF DRAWINGS

[0011] Fig. 1 is a schematic diagram of a sensor device.

[0012] Fig. 2 is a schematic diagram showing the relation between different brightness ranges.

[0013] Fig. 3 is a schematic diagram of an imaging unit of a sensor device.

[0014] Fig. 4 is another schematic diagram of a sensor device.

[0015] Figs. 5A and 5B are schematic diagrams referring to binning in an imaging unit.

[0016] Fig. 6 is another schematic diagram of a sensor device.

[0017] Fig. 7 is a schematic diagram of an image processing system.

[0018] Fig. 8 is a schematic process flow of a method for operating a sensor device.

[0019] Fig. 9 is a schematic block diagram of a vehicle control system.

[0020] Fig. 10 is a diagram of assistance in explaining an example of installation positions of an outside-vehicle information detecting section and an imaging section.

[0021] Figs. HA and 11B are schematic illustrations of a mobile device and a head mounted display comprising a sensor device.

[0022] DETAILED DESCRIPTION

[0023] The present disclosure is directed to mitigating problems related to the generation and processing of HDR images. Fig. 1 is a schematic block diagram of a sensor device 100 for generating a plurality of HDR images from a single scene. The sensor device 100 comprises as basic components an imaging unit 110 and an image processing unit 120. The sensor device 100 may also comprise further components, like e.g. control unit(s) for controlling the operation of the imaging unit 110 and the image processing unit 120. However, since such components and their functions are either known to a skilled person or of no relevance for the present disclosure they will not be described in detail.

[0024] The imaging unit 110 is configured to capture a plurality of raw images of a scene with different sensitivities for brightness. That is, during a single image capturing incoming light is converted into electrical signals such as to generate a plurality of sets of image data, each set of image data constituting a (raw) image. The different raw images differ by their sensitivity for brightness. Some raw images are capable to resolve details of darker regions of the scene but do not resolve bright regions well. Other raw images resolve bright regions well but do not resolve dark regions well. Further, intermediate raw images will resolve intermediate brightness regions well while not being able to resolve very dark and very bright regions.

[0025] The imaging unit 110 may in principle be constituted by any device that is capable to convert incoming light into electrical signals with the possibility to change the brightness sensitivity, e.g. by a change of exposure time, readout gain, aperture size or the like. In the following it will often be assumed that the imaging unit 110 is a CMOS sensor. However, also other types of imaging devices may be used such as e.g. a CCD sensor.

[0026] The image processing unit 120 is configured to generate high dynamic range, HDR, images from the raw images. In particular, by combining raw images with different brightness sensitivity, an image can be obtained in which dark, intermediate, and bright regions of the scene are equally well visible. The process of generating HDR images from raw images is in principle well known. Therefore, a detailed discussion of this process can be omitted here.

[0027] The image processing unit 120 may be constituted by any processing component that is capable to generated HDR images from the incoming raw images. The image processing unit 120 may e.g. be a computer, a processor, a CPU, a GPU, an FPGA or an ASIC. Parts or all of the functions of the image processing unit 120 may be constituted by software, hardware or a mixture of both. The image processing unit 120 may be located on the same chip or within the same packaging or housing as the imaging unit 110. The image processing unit 120 may also be located in a different device than the imaging unit 110, like e.g. an external personal computer or the like.

[0028] As illustrated in Fig. 1, the imaging unit 110 is configured to capture a first set 112 of raw images of the scene with different sensitivities for brightness using a first exposure period tl. A second set of raw images 114 of the scene with different sensitivities for brightness is captured while using a second exposure period t2, which is shorter than the first exposure period tl. In addition, the highest sensitivity for brightness of the raw images of the first set 112 is higher than the highest sensitivity for brightness of the raw images of the second set 114.

[0029] Thus, during one image capture the imaging unit 110 obtains light during two exposure periods of different lengths. Here, the first exposure period tl may be between 11 ms and 30 ms and the second exposure period t2 may be between 10 ps and 3 ms. Preferably, the first exposure tl may be 11 ms and the second exposure period t2 may be 1 ms. This means that the raw images obtained during the first exposure period tl will be generally more sensitive to brightness than the raw images obtained during the second exposure period t2, since light accumulate longer on the imaging unit 110 during the first exposure period tl. On the other hand, due to the longer exposure period the raw images of the first set 112 will show more blurring than the raw images of the second set 114.

[0030] Accordingly, amongst the raw images of the first set 112 will be images that show more details of dark regions of the imaged scene than any of the raw images of the second set 114. Therefore, amongst all raw images the highest sensitivity for brightness will be among the raw images of the first set 112. On the other hand, also the first set 112 will contain raw image(s) with a reduced brightness sensitivity that will show details of bright regions of the scene.

[0031] The raw images of the second set 114 will generally not provide as many details on dark regions of the imaged scene as the raw images of the first set 112. However, due to the shorter exposure period the raw images of the second set 114 will be sharper than the raw images of the first set 112.

[0032] The imaging unit 110 provides the data of the raw images of the first set 112 and the second set 114 to the image processing unit 120. As shown in Fig. 1 this may be done via analog digital converters 116, 118 that convert the analog electrical signals generated within the imaging unit 110 into digital signals, which are then provided to the image processing unit 120. However, the imaging unit 110 may also provide analog signals to the image processing unit 120, e.g. when analog to digital conversion is carried out in the image processing unit 120.

[0033] The image processing unit 120 generates then a first HDR image 122 from the first set 112 of raw images and a second HDR image 124 from the second set 114 of raw images, e.g. by using an - in principle known - mobile interface processor interface, MIPI, 126.

[0034] That is, from the raw images of the first set 112 a first HDR image 122 is generated, which first HDR image 122 is well adapted to correctly show dark regions of the imaged scene due to the high brightness sensitivity of at least one of the raw images of the first set 112. Nevertheless, also the first HDR image 122 will be able to represent bright regions correctly. But in comparison to the second HDR image 124, the first HDR image 122 will show reduced sharpness.

[0035] Thus, the first HDR image 122 is suited to convey information on comparably dark regions of a scene with a certain amount of blur. Since human vision, HV, is on the one hand adapted to also resolve dark regions and on the other capable to deal with (comparably) blurred images, the first HDR image 122 will be suitable for HV applications such as displaying the first HDR image 122 to a human user. Further, the first HDR image 122 may be used for image based automated processing and decision making. For example, in an autonomously driving car, recognition of objects that are not within the headlight of the car is important to prevent accidents. This object recognition will work comparably well based on first HDR images 122, which present sufficient details on dark objects that are outside the headlight.

[0036] In contrast, the second HDR image 124 is a sharp representation of bright regions of the imaged scene. Typically, a high resolution / sharpness and a sufficient brightness is preferably for computer vision, CV, applications that are usually based on edge detection, segmentation or the like. In the example of the autonomously driving car, the second HDR image 124 can provide information that is necessary for navigating the car. For example, the texture of the road, boundaries of the road, road signs, traffic lights or the like can be recognized in the second HDR image 124 by CV tools. Such objects are usually brightly illuminated by the headlights of the car and will therefore be well recognizable in the second HDR image 124.

[0037] The sensor device 100 allows therefore to provide two HDR images with different properties with a single image capture. While one HDR image is suitable for HV applications, the other HDR image can be used for CV applications. This eliminates the need to capture the two HDR images with two cameras. Accordingly, costs, area, and power consumption can be reduced compared to a two-camera solution. In addition, since both HDR images are obtained during the same image capture, the two HDR images are automatically aligned. This allows carrying out two different tasks (HV task and CV task) in parallel that regard the same field of view.

[0038] As shown in Fig. 1, the imaging unit 110 may further be configured to start the second exposure period t2 directly after capturing of the first set 112 of raw images has been ended. This means that after the readout process of the last raw image of the first set 112 has been ended and the imaging unit 110 has been reset, the second exposure period t2 starts immediately. This ensures that there is as little temporal shift between obtaining the raw images of the first set 112 and the raw images of the second set 114.

[0039] Also, the imaging unit 110 may immediately forward the raw images of the first set 112 to the image processing unit 120, i.e. without buffering them. That is, the imaging unit 110 does not delay output of data of the first set 112 of raw images in order to temporally align output of the data of the first set 112 and the data of the second set 114 to the imaging processing unit 120. Accordingly, the image processing unit 120 is configured to start processing of the first set 112 of raw images for generating the first HDR image 122 before starting processing of the second set 114 of raw images for generating the second HDR image 124.

[0040] This ensures that the first HDR image 122 and the second HDR image 124 are provided as soon as possible. This provides the possibility to operate on the obtained HDR images in real time. In particular in the field of autonomously driving cars even a delay by several microseconds may cause fatal accidents, which makes a real time processing desirable. Fig. 2 shows a schematic diagram for explaining the different sensitivities for brightness of the different raw images captured by the imaging unit 110. In Fig. 2 there is shown a response of the imaging unit 110 (e.g. an intensity detected by a pixel of the imaging unit 110) to varying brightness levels for three different sensitivities for brightness.

[0041] For raw images directed to a first brightness range si, indicated by a dotted line in Fig. 2, the response of the imaging unit 110 rises linearly from zero to an upper boundary B 1 while the brightness increases from zero. Then, the response saturates at Bl, which means that regions of the scene that are brighter will only be recorded with the response Bl irrespective of their content, i.e. the respective regions of the scene will be overexposed.

[0042] For raw images directed to a second brightness range s2, indicated by a dashed line in Fig. 2, there is a constant response up to a given brightness. Then, the response increases linearly up to an upper boundary B2, where saturation starts. Here, only regions in the brightness range s2 will be resolved in the corresponding raw image, while regions with brightness out of the range s2 will be represented by constant responses (underexposure or overexposure). The same holds analogously for raw images directed to a third brightness range s3 with a higher upper boundary B3 (illustrated with a dash-dotted line in Fig. 2.

[0043] Although Fig. 2 only shows three brightness ranges si, s2, s3 there could be more or less brightness ranges for the raw images of the first set 112 and the second set 114. Also, the increase of response with brightness is not necessarily linear. A linear example has been chosen merely for the ease of description.

[0044] As can be seen in Fig. 2, the different brightness ranges si, s2, and s3 overlap in overlap ranges ol and o2. As is in principle known, when forming an HDR image from raw images having sensitivities for brightness as illustrated in Fig. 2 the HDR image is formed by combining parts of the raw images such as to provide an optimal response to brightness changes for all levels of brightness. Thus, dark regions of the scene will be taken from raw images directed to brightness range si, intermediately bright regions from raw images directed to brightness range s2 and bright regions from raw images directed to brightness range s3. In addition, for brightness levels in the overlapping regions ol, o2 (or in parts thereof), two raw images may be blended.

[0045] Here, the first set 112 of raw images may comprise images that show a scene without underexposure or overexposure in the first brightness range si that includes zero brightness, the second brightness range s2, and the third brightness range s3, where for the upper boundaries Bl, B2, and B3 of these brightness ranges Bl < B2 < B3 holds. Thus, the first set 112 of raw images contains raw images that are directed to brightness ranges including zero brightness up to brightness ranges including high brightness levels. Here, it is also possible that the first brightness range si has a lower level above zero.

[0046] On the other hand, the second set 114 of raw images may comprise images that show a scene without underexposure or overexposure in a fourth brightness range and a fifth brightness range, where for the upper boundary B4 of the fourth brightness range and the upper boundary B5 of the fifth brightness range B4 < B5 hold, and where the lower boundary of the fourth brightness range is larger than zero (or larger than the lower boundary of the first brightness region, if this boundary is larger than zero). Thus, while in principle directed to different brightness ranges than the raw images of the first set 112, the raw images of the second set 114 are less sensitive for dark regions, since their lowest resolvable brightness value lies above zero brightness.

[0047] This provides an example for the distribution of sensitivities for brightness discussed above with respect to Fig. 1. That the second set does not contain raw images directed to brightness ranges around zero brightness is uncritical in the given context, since the resulting second HDR image 124 is to be used for CV applications that are focused on bright regions of a scene.

[0048] While in principle different brightness ranges may be used for the first set 112 and the second set 114, the second brightness range s2 used in the first set 112 may be equal to the fourth brightness range used in the second set 114, and the third brightness range s3 used in the first set 112 may be equal to the fifth brightness range used in the second set 114.

[0049] The above can also be understood by the following considerations. In Fig. 2 the brightness corresponds to all the light received during a given exposure period. Thus, assuming that Fig. 2 shows the relation between response and brightness for the first exposure period, the line indicating said relation for the second, shorter exposure period will be located at a position right to the line of Fig. 2. This is due to the fact that during the shorter, second exposure period saturation occurs later, i.e. more light is necessary to reach the same brightness. Shifting the brightness ranges si, s2, s3 together with the line indicates new brightness ranges for the capturing of the second set 114 of images that are higher than the brightness ranges used for the capturing of the first set 114 of images. Accordingly, dark regions in a captured scene can be less well resolved during capturing of the second set 114 of images, as was also explained above.

[0050] One particular example will be described with respect to Fig. 3. Here, the imaging unit 110 is constituted by a CMOS sensor and comprises a plurality of imaging pixels 111 that are configured to convert light into electrical signals. Of course, the described example is not limiting. Its structure may deviate from the structure described below. Moreover, the same effects are also achievable by other types of image sensors.

[0051] In the example of Fig. 3, each pixel 111 may include a photoelectric conversion element PD, preferably a photodiode, onto which light impinging on the imaging unit 110 is guided in an in principle known manner by an optical system of the imaging unit 110. Electrical charges generated by the photoelectric conversion element PD are transferred to a floating diffusion FD of the pixel 111 based on a signal TRG to a transfer transistor I l la. As long as the transfer transistor I l la is in an on-state charges that are generated at the photoelectric conversion element PD are transferred to the floating diffusion FD. Thus, via the signal TRG an effective exposure period of the pixel 111 can be adjusted.

[0052] The floating diffusion FD is connected via a reset transistor 11 lb to a power supply voltage VDD. When the reset transistor 11 lb is switched on, charges accumulated at the floating diffusion FD are absorbed by this power supply voltage, i.e. the floating diffusion FD is emptied via the reset transistor 11 lb.

[0053] The charges accumulated at the floating diffusion FD control a current flow through amplification transistor 111c. The more charge is accumulated, the larger the current will be. Readout of this current is triggered via selection transistor 11 Id. Once the selection signal SEL switches on the selection transistor 11 Id a current is flowing through the signal line VSL that is related via the accumulated charges at the floating diffusion FD to the light that did impinge on the photoelectric conversion element PD during the respective exposure period.

[0054] Thus, the different exposure periods tl and t2 can be obtained by setting transfer periods of charges via the transfer transistor I l la. The different raw images within the first set 112 and the second set 114 can then be obtained by reading out the charges accumulated in all floating diffusions of all pixels 111 with different readout parameters. Here, it should be noted that for each of the raw images of the first set 112 the same charges are read out, however, with different readout settings. Just the same, also for the raw images of the second set 114 the same charges are read out. Accordingly, the underlying information on the scene is the same for all raw images contained in one or the two sets 112, 114.

[0055] In particular, the first set 112 of raw images may be obtained by reading out electrical signals generated in the imaging pixels 111 during the first exposure period tl with different gains. That is, the gain of an amplifier 11 If that is used to further amplify the signal coming from the signal line VSL is adjusted differently for different raw images of the first set 112. This is illustrated schematically in Fig. 3 by the arrow pointing to the amplifier 11 If to set the gain and the arrow pointing away from the amplifier 11 If to indicate the output of the according pixel value of the respective raw image.

[0056] Then, after readout of the charges at the floating diffusion FD has been carried out as often as necessary to generate all raw images of the first set 112, the floating diffusion FD is reset and the second exposure period is started by turning on the transfer transistor I l la. Afterwards, also the second set 114 of raw images is obtained by reading out electrical signals generated in the imaging pixels 111 during the second exposure period t2 with different gains.

[0057] Thus, by varying the gain used in the imaging unit 110 to amplify imaging signals during readout, different ranges of brightness sensitivity can be obtained. For example, in Fig. 2 the different brightness ranges si, s2, and s3 may be obtained by successively reducing the gain during readout. While sensitivity range s 1 is obtained by using a high gain, sensitivity range s2 is obtained by using a lower gain, and the gain is further decreased for sensitivity range s3. Of course, more or different sensitivity ranges may be set by introducing readout cycles with other gains.

[0058] According to one example illustrated schematically in Fig. 4 the first set 112 of raw images may comprise a high conversion gain, HCG, image, a low conversion gain, LCG, image, and an overflow, OVF, image, while the second set 114 of raw images does not comprise a HCG gain image, i.e. it comprises merely a LCG image and an OVF image. Here, the HCG images are read out with a higher gain than LCG images and the LCG images are read out with a higher gain than OVF images. Thus, the first HDR image 122 is generated based on three images with different brightness sensitivities which provide also a good representation of dark regions of the captured scene. For the second HDR image 124 the HCG image is omitted, i.e. the highest brightness sensitivity is omitted. However, this is not detrimental due to the intended use of the second HDR image 124 for CV applications.

[0059] For both sets 112, 114 of raw images the same LCG and OVF settings may be used, i.e. the readout of the LCG images of both sets 112, 114 of raw images is carried out with the same gain and the readout of the OVF images of both sets 112, 114 of raw images is carried out with the same gain. This makes the readout settings less complex since only three different gains need to be used. Of course, also different gains may be used in reading out the raw images of the different sets 112, 114. For example, HCG may correspond to 1,000 to 3,000, e.g. 2,000 pV / e", LCG to 25 to 75, e.g. 50 pV / e", and OVF to 0.5 to 1.5, e.g. 1 pV / e". This accounts for the total gain, i.e. the pixel conversion gain times the analog readout gain. Also, it might be possible to use a gain of 16 to 20, e.g. 18 dB for HCG and gains in the range from 0 dB to 5 dB for LCG and OVG.

[0060] Above, five raw images are generated for the two HDR images 122, 124. In order to further reduce the number of raw images that need to be processed, the imaging unit 110 may be configured to bin a plurality of adjacent pixels 111, preferably four pixels arranged in a 2 x 2 matrix. Binning means here to allow a plurality of pixels 111 (or of respective photoelectric conversion elements PD) to transfer their photoelectrically generated charges to a common floating diffusion FD and to read out the thus combined signals of these pixels 111 together. This increases the effective sensitive area that corresponds to one floating diffusion FD and hence to one readout line. Thus, binning of pixels 111 leads to an increased brightness sensitivity. However, it reduces the saturation level per pixel 111, since the saturation occurs when the floating diffusion FD is saturated with charges.

[0061] As is in principle well known, binning can be switched on an off by either supplying all pixel signals at the same time to the common floating diffusion FD for common readout (binning on) or by supplying pixel signals consecutively to the floating diffusion and by reading out the pixel signals consecutively (binning off). Alternatively, each pixel 111 may have its own floating diffusion FD which is used for readout without binning, while additional transfer transistors can be used to connect several pixels to one common floating diffusion. Since the principles of binning are well known a more detailed description can be omitted here.

[0062] As schematically shown in Fig. 5 A during capturing of the first set 112 of raw images the pixels 111 are binned. As shown in Fig. 5B the pixels 111 are not binned during capturing of the second set 114 of raw images. Thus, the brightness sensitivity of the raw images of the first set 112 is further increased as compared to the brightness sensitivity of the raw images of the second set 114. In addition, the resolution of the first set 112 is reduced compared to the resolution of the second set 114.

[0063] Accordingly, the second HDR image 124 that is used for CV tasks remains unchanged, while the resolution of the first HDR image 122 is further reduced. This is, however, still uncritical for the intended usage of the first HDR image 122 for HV.

[0064] However, due to the increase of brightness sensitivity due to binning, the HCG readout may be omitted. That is, as shown in Fig. 6, the first set 112 of raw images merely comprises an LCG image and an OVF image, just as the second set 114 of raw images. This allows generation of two HDR images for CV applications and HV application based on only four input raw images. Thus, provision of two HDR images for one image capturing can be achieved with a further reduction in processing power and cost.

[0065] Of course, also during capturing of the second set 114 of raw images the pixels 111 can be binned, if this is useful. Also, in specific circumstances it might also be helpful to only bin the pixels 111 during capturing of the second set 114 of raw images, but not during capturing of the first set 112.

[0066] The generated HDR images may be used in an image processing system 200 as schematically illustrated in Fig. 7. The image processing system 200 comprises the sensor device 100 as it was described above. Further, the system may comprise a display 210 and / or a computer vision unit 220. Here, the display 210 may be used for presenting the first, HV HDR image 122 to a user of the image processing system 200. Alternatively or additionally the second, CV HDR image 124 is fed into the computer vision unit 220 that is configured to execute predetermined processing based on the second HDR image 124.

[0067] The computer vision unit 220 may here be a computer, processor or the like that is capable to process the second HDR image 124 such as to carry out a specific task. For example, the computer vision unit 220 may determine a vehicle speed and steering direction based on a stream of second HDR images 124 provided from the sensor device 100. The computer vision unit 220 may carry out its tasks rule-based or based on machine learning. Further, the computer vision unit 220 may also receive the first HDR images 122 in order to carry out different tasks based on these first HDR images 122, such as detection of objects on sides of a road or the like.

[0068] In general the sensor device 100 allows to carry out different, parallel processing of different tasks based on the two differing HDR images that are obtained from a single image capturing.

[0069] The above-described method for operating a sensor device 10 can be summarized as illustrated in Fig. 8.

[0070] At S 101 the sensor device 100 may capture by the imaging unit 110 a first set 112 of raw images of the scene with different sensitivities for brightness using a first exposure period tl. At SI 02 the sensor device 100 may capture by the imaging unit 110 a second set 114 of raw images of the scene with different sensitivities for brightness using a second exposure period t2.

[0071] At S103 the image processing unit 120 generates a first HDR image 122 from the first set 112 of raw images and at SI 04 it generates a second HDR image 124 from the second set 114 of raw images. Since the first exposure period tl is longer than the second exposure period t2, and since the highest sensitivity for brightness of the raw images of the first set 112 is higher than the highest sensitivity for brightness of the raw images of the second set 114, the resulting HDR images are particularly suited for HV applications and CV applications as described above.

[0072] The technology according to the above (i.e. the present technology) is applicable to various products. For example, the technology according to the present disclosure may be realized as a device that is installed on any kind of moving bodies, for example, vehicles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobilities, airplanes, drones, ships, and robots.

[0073] Fig. 9 is a block diagram depicting an example of schematic configuration of a vehicle control system as an example of a mobile body control system to which the technology according to an embodiment of the present disclosure can be applied.

[0074] The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. In the example depicted in Fig. 9, the vehicle control system 12000 includes a driving system control unit 12010, a body system control unit 12020, an outside-vehicle information detecting unit 12030, an in-vehicle information detecting unit 12040, and an integrated control unit 12050. In addition, a microcomputer 12051, a sound / image output section 12052, and a vehicle-mounted network interface (I / F) 12053 are illustrated as a functional configuration of the integrated control unit 12050.

[0075] The driving system control unit 12010 controls the operation of devices related to the driving system of the vehicle in accordance with various kinds of programs. For example, the driving system control unit 12010 functions as a control device for a driving force generating device for generating the driving force of the vehicle, such as an internal combustion engine, a driving motor, or the like, a driving force transmitting mechanism for transmitting the driving force to wheels, a steering mechanism for adjusting the steering angle of the vehicle, a braking device for generating the braking force of the vehicle, and the like.

[0076] The body system control unit 12020 controls the operation of various kinds of devices provided to a vehicle body in accordance with various kinds of programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various kinds of lamps such as a headlamp, a backup lamp, a brake lamp, a turn signal, a fog lamp, or the like. In this case, radio waves transmitted from a mobile device as an alternative to a key or signals of various kinds of switches can be input to the body system control unit 12020. The body system control unit 12020 receives these input radio waves or signals, and controls a door lock device, the power window device, the lamps, or the like of the vehicle.

[0077] The outside-vehicle information detecting unit 12030 detects information about the outside of the vehicle including the vehicle control system 12000. For example, the outside-vehicle information detecting unit 12030 is connected with an imaging section 12031. The outside-vehicle information detecting unit 12030 makes the imaging section 12031 image an image of the outside of the vehicle, and receives the imaged image. On the basis of the received image, the outside-vehicle information detecting unit 12030 may perform processing of detecting an object such as a human, a vehicle, an obstacle, a sign, a character on a road surface, or the like, or processing of detecting a distance thereto.

[0078] The imaging section 12031 is an optical sensor that receives light, and which outputs an electric signal corresponding to a received light amount of the light. The imaging section 12031 can output the electric signal as an image, or can output the electric signal as information about a measured distance. In addition, the light received by the imaging section 12031 may be visible light, or may be invisible light such as infrared rays or the like.

[0079] The in-vehicle information detecting unit 12040 detects information about the inside of the vehicle. The in-vehicle information detecting unit 12040 is, for example, connected with a driver state detecting section 12041 that detects the state of a driver. The driver state detecting section 12041, for example, includes a camera that images the driver. On the basis of detection information input from the driver state detecting section 12041, the in-vehicle information detecting unit 12040 may calculate a degree of fatigue of the driver or a degree of concentration of the driver, or may determine whether the driver is dozing.

[0080] The microcomputer 12051 can calculate a control target value for the driving force generating device, the steering mechanism, or the braking device on the basis of the information about the inside or outside of the vehicle which information is obtained by the outside-vehicle information detecting unit 12030 or the in-vehicle information detecting unit 12040, and output a control command to the driving system control unit 12010. For example, the microcomputer 12051 can perform cooperative control intended to implement functions of an advanced driver assistance system (ADAS) which functions include collision avoidance or shock mitigation for the vehicle, following driving based on a following distance, vehicle speed maintaining driving, a warning of collision of the vehicle, a warning of deviation of the vehicle from a lane, or the like.

[0081] In addition, the microcomputer 12051 can perform cooperative control intended for automatic driving, which makes the vehicle to travel autonomously without depending on the operation of the driver, or the like, by controlling the driving force generating device, the steering mechanism, the braking device, or the like on the basis of the information about the outside or inside of the vehicle which information is obtained by the outside-vehicle information detecting unit 12030 or the in-vehicle information detecting unit 12040.

[0082] In addition, the microcomputer 12051 can output a control command to the body system control unit 12020 on the basis of the information about the outside of the vehicle which information is obtained by the outside-vehicle information detecting unit 12030. For example, the microcomputer 12051 can perform cooperative control intended to prevent a glare by controlling the headlamp so as to change from a high beam to a low beam, for example, in accordance with the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detecting unit 12030.

[0083] The sound / image output section 12052 transmits an output signal of at least one of a sound and an image to an output device capable of visually or auditorily notifying information to an occupant of the vehicle or the outside of the vehicle. In the example of Fig. 9, an audio speaker 12061, a display section 12062, and an instrument panel 12063 are illustrated as the output device. The display section 12062 may, for example, include at least one of an on-board display and a head-up display.

[0084] Fig. 10 is a diagram depicting an example of the installation position of the imaging section 12031.

[0085] In Fig. 10, the imaging section 12031 includes imaging sections 12101, 12102, 12103, 12104, and 12105.

[0086] The imaging sections 12101, 12102, 12103, 12104, and 12105 are, for example, disposed at positions on a front nose, sideview mirrors, a rear bumper, and a back door of the vehicle 12100 as well as a position on an upper portion of a windshield within the interior of the vehicle. The imaging section 12101 provided to the front nose and the imaging section 12105 provided to the upper portion of the windshield within the interior of the vehicle obtain mainly an image of the front of the vehicle 12100. The imaging sections 12102 and 12103 provided to the sideview mirrors obtain mainly an image of the sides of the vehicle 12100. The imaging section 12104 provided to the rear bumper or the back door obtains mainly an image of the rear of the vehicle 12100. The imaging section 12105 provided to the upper portion of the windshield within the interior of the vehicle is used mainly to detect a preceding vehicle, a pedestrian, an obstacle, a signal, a traffic sign, a lane, or the like.

[0087] Incidentally, Fig. 10 depicts an example of photographing ranges of the imaging sections 12101 to 12104. An imaging range 12111 represents the imaging range of the imaging section 12101 provided to the front nose. Imaging ranges 12112 and 12113 respectively represent the imaging ranges of the imaging sections 12102 and 12103 provided to the sideview mirrors. An imaging range 12114 represents the imaging range of the imaging section 12104 provided to the rear bumper or the back door. A bird’s-eye image of the vehicle 12100 as viewed from above is obtained by superimposing image data imaged by the imaging sections 12101 to 12104, for example.

[0088] At least one of the imaging sections 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging sections 12101 to 12104 may be a stereo camera constituted of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0089] For example, the microcomputer 12051 can determine a distance to each three-dimensional object within the imaging ranges 12111 to 12114 and a temporal change in the distance (relative speed with respect to the vehicle 12100) on the basis of the distance information obtained from the imaging sections 12101 to 12104, and thereby extract, as a preceding vehicle, a nearest three-dimensional object in particular that is present on a traveling path of the vehicle 12100 and which travels in substantially the same direction as the vehicle 12100 at a predetermined speed (for example, equal to or more than 0 km / hour). Further, the microcomputer 12051 can set a following distance to be maintained in front of a preceding vehicle in advance, and perform automatic brake control (including following stop control), automatic acceleration control (including following start control), or the like. It is thus possible to perform cooperative control intended for automatic driving that makes the vehicle travel autonomously without depending on the operation of the driver or the like. For example, the microcomputer 12051 can classify three-dimensional object data on three-dimensional objects into three-dimensional object data of a two-wheeled vehicle, a standard-sized vehicle, a largesized vehicle, a pedestrian, a utility pole, and other three-dimensional objects on the basis of the distance information obtained from the imaging sections 12101 to 12104, extract the classified three-dimensional object data, and use the extracted three-dimensional object data for automatic avoidance of an obstacle. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as obstacles that the driver of the vehicle 12100 can recognize visually and obstacles that are difficult for the driver of the vehicle 12100 to recognize visually. Then, the microcomputer 12051 determines a collision risk indicating a risk of collision with each obstacle. In a situation in which the collision risk is equal to or higher than a set value and there is thus a possibility of collision, the microcomputer 12051 outputs a warning to the driver via the audio speaker 12061 or the display section 12062, and performs forced deceleration or avoidance steering via the driving system control unit 12010. The microcomputer 12051 can thereby assist in driving to avoid collision.

[0090] At least one of the imaging sections 12101 to 12104 may be an infrared camera that detects infrared rays. The microcomputer 12051 can, for example, recognize a pedestrian by determining whether or not there is a pedestrian in imaged images of the imaging sections 12101 to 12104. Such recognition of a pedestrian is, for example, performed by a procedure of extracting characteristic points in the imaged images of the imaging sections 12101 to 12104 as infrared cameras and a procedure of determining whether or not it is the pedestrian by performing pattern matching processing on a series of characteristic points representing the contour of the object. When the microcomputer 12051 determines that there is a pedestrian in the imaged images of the imaging sections 12101 to 12104, and thus recognizes the pedestrian, the sound / image output section 12052 controls the display section 12062 so that a square contour line for emphasis is displayed so as to be superimposed on the recognized pedestrian. The sound / image output section 12052 may also control the display section 12062 so that an icon or the like representing the pedestrian is displayed at a desired position.

[0091] An example of the vehicle control system to which the technology according to the present disclosure is applicable has been described above. The technology according to the present disclosure is applicable to the imaging section 12031 among the above-mentioned configurations. Specifically, the sensor device 10 is applicable to the imaging section 12031. The imaging section 12031 to which the technology according to the present disclosure has been applied flexibly acquires event data and performs data processing on the event data, thereby being capable of providing appropriate driving assistance.

[0092] Further possible implementations of the sensor device 100 are mobile devices 3000 such as cell phones, tablets, smart watches and the like as shown in Fig. 11A or head-mounted displays 4000 as shown in Fig. 11B. Further, the sensor device 10 is useable in augmented and / or virtual reality applications / cameras or in surveillance systems like 360° cameras.

[0093] Note that, the embodiments of the present technology are not limited to the above-mentioned embodiment, and various modifications can be made without departing from the gist of the present technology. Further, the effects described herein are only exemplary and not limited, and other effects may be provided.

[0094] Note that, the present technology can also take the following configurations.

[0095] [1] A sensor device (100) comprising: an imaging unit (110) that is configured to capture a plurality of raw images of a scene with different sensitivities for brightness; and an image processing unit (120) that is configured to generate high dynamic range, HDR, images from the raw images; wherein the imaging unit (110) is configured to capture a first set (112) of raw images of the scene with different sensitivities for brightness using a first exposure period (tl) and to capture a second set of raw images (114) of the scene with different sensitivities for brightness using a second exposure period (t2); the image processing unit (120) is configured to generate a first HDR image (122) from the first set (112) of raw images and a second HDR image (124) from the second set (114) of raw images; the first exposure period (tl) is longer than the second exposure period (t2); and the highest sensitivity for brightness of the raw images of the first set is higher than the highest sensitivity for brightness of the raw images of the second set.

[0096] [2] The sensor device (100) according to [1], wherein the imaging unit (110) comprises a plurality of imaging pixels (111) that are configured to convert light into electrical signals; the first set (112) of raw images is obtained by reading out electrical signals generated in the imaging pixels (111) during the first exposure period (tl) with different gains; and the second set (114) of raw images is obtained by reading out electrical signals generated in the imaging pixels (111) during the second exposure period (t2) with different gains.

[0097] [3] The sensor device (100) according to [1] or [2], wherein the first set (112) of raw images comprises a high conversion gain, HCG, image, a low conversion gain, LCG, image, and an overflow, OVF, image; the second set (114) of raw images comprises a LCG image and an OVF image;

[0098] HCG images are read out with a higher gain than LCG images; and

[0099] LCG images are read out with a higher gain than OVF images.

[0100] [4] The sensor device (100) according to any one of [1] to [3], wherein the first set (112) of raw images comprise images that show a scene without underexposure or overexposure in a first brightness range (si) including zero brightness, a second brightness range (s2), and a third brightness range (s3), where for the upper boundary Bl of the first brightness range (si), the upper boundary B2 of the second brightness range (s2) and the upper boundary B3 of the third brightness range (s3) Bl < B2 < B3 holds; the second set (114) of raw images comprises images that show a scene without underexposure or overexposure in a fourth brightness range and a fifth brightness range, where for the upper boundary B4 of the fourth brightness range and the upper boundary B5 of the fifth brightness range B4 < B5 holds; and the lower boundary of the fourth brightness range is larger than zero.

[0101] [5] The sensor device (100) according to [4], wherein the second brightness range (s2) is equal to the fourth brightness range and the third brightness range (s3) is equal to the fifth brightness range.

[0102] [6] The sensor device (100) according to [2], wherein the imaging unit (110) is configured to bin a plurality of adjacent pixels (111), preferably four pixels arranged in a 2 x 2 matrix, during capturing of the first set (112) of raw images and to not bin the plurality of adjacent pixels (111) during capturing of the second set (114) of raw images.

[0103] [7] The sensor device (100) according to [6], wherein the first set (112) of raw images comprises an LCG image and an OVF image; the second set (114) of raw images comprises an LCG image and an OVF image; and LCG images are read out with a higher gain than OVF images.

[0104] [8] The sensor device (100) according to any one of [1] to [7], wherein the first exposure period is between 11 ms and 30 ms and the second exposure period is between 10 ps and 3 ms.

[0105] [9] The sensor device (100) according to any one of [1] to [8], wherein the imaging unit (110) is configured to start the second exposure period (t2) directly after capturing of the first set (112) of raw images has been ended; and the image processing unit (120) is configured to start processing of the first set (112) of raw images for generating the first HDR image (122) before starting processing of the second set (114) of raw images for generating the second HDR image (124).

[0106]

[0010] The sensor device (100) according to any one of [1] to [9], wherein the imaging unit (110) is not configured to delay output of data of the first set (112) of raw images in order to temporally align output of the data of the first set (112) of raw images and the data of the second set (114) of raw images to the imaging processing unit (120).

[0107]

[0011] An image processing system (200) comprising the sensor device (100) according to any one of [1] to

[0010] ; a display (210) for presenting the first HDR image (122) to a user of the image processing system (200); and / or a computer vision unit (220) that is configured to execute predetermined processing based on the second HDR image (124).

[0012] A method for operating the sensor device (100) according to any one of [1] to

[0010] or the image processing system (200) according to

[0011] , the method comprising: capturing, by the imaging unit (110), a first set (112) of raw images of the scene with different sensitivities for brightness using a first exposure period (tl) and a second set (114) of raw images of the scene with different sensitivities for brightness using a second exposure period (t2); generating, by the image processing unit (120), a first HDR image from the first set (112) of raw images and a second HDR image from the second set (114) of raw images; wherein the first exposure period (tl) is longer than the second exposure period (t2); and the highest sensitivity for brightness of the raw images of the first set (112) is higher than the highest sensitivity for brightness of the raw images of the second set (114).

Claims

CLAIMS1. A sensor device comprising: an imaging unit that is configured to capture a plurality of raw images of a scene with different sensitivities for brightness; and an image processing unit that is configured to generate high dynamic range, HDR, images from the raw images; wherein the imaging unit is configured to capture a first set of raw images of the scene with different sensitivities for brightness using a first exposure period and to capture a second set of raw images of the scene with different sensitivities for brightness using a second exposure period; the image processing unit is configured to generate a first HDR image from the first set of raw images and a second HDR image from the second set of raw images; the first exposure period is longer than the second exposure period; and the highest sensitivity for brightness of the raw images of the first set is higher than the highest sensitivity for brightness of the raw images of the second set.

2. The sensor device according to claim 1, wherein the imaging unit comprises a plurality of imaging pixels that are configured to convert light into electrical signals; the first set of raw images is obtained by reading out electrical signals generated in the imaging pixels during the first exposure period with different gains; and the second set of raw images is obtained by reading out electrical signals generated in the imaging pixels during the second exposure period with different gains.

3. The sensor device according to claim 1, wherein the first set of raw images comprises a high conversion gain, HCG, image, a low conversion gain, LCG, image, and an overflow, OVF, image; the second set of raw images comprises a LCG image and an OVF image;HCG images are read out with a higher gain than LCG images; andLCG images are read out with a higher gain than OVF images.

4. The sensor device according to claim 1, wherein the first set of raw images comprise images that show a scene without underexposure or overexposure in a first brightness range including zero brightness, a second brightness range, and a third brightness range, where for the upper boundary B 1 of the first brightness range, the upper boundary B2 of the second brightness range and the upper boundary B3 of the third brightness range Bl < B2 < B3 holds; the second set of raw images comprises images that show a scene without underexposure or overexposure in a fourth brightness range and a fifth brightness range, where for the upper boundary B4 of the fourth brightness range and the upper boundary B5 of the fifth brightness range B4 < B5 holds; and the lower boundary of the fourth brightness range is larger than zero.

5. The sensor device according to claim 4, whereinthe second brightness range is equal to the fourth brightness range and the third brightness range is equal to the fifth brightness range.

6. The sensor device according to claim 2, wherein the imaging unit is configured to bin a plurality of adjacent pixels, preferably four pixels arranged in a 2 x 2 matrix, during capturing of the first set of raw images and to not bin the plurality of adjacent pixels during capturing of the second set of raw images.

7. The sensor device according to claim 6, wherein the first set of raw images comprises an LCG image and an OVF image; the second set of raw images comprises an LCG image and an OVF image; and LCG images are read out with a higher gain than OVF images.

8. The sensor device according to claim 1, wherein the first exposure period is between 11 ms and 30 ms and the second exposure period is between 10 ps and 3 ms.

9. The sensor device according to claim 1, wherein the imaging unit is configured to start the second exposure period directly after capturing of the first set of raw images has been ended; and the image processing unit is configured to start processing of the first set of raw images for generating the first HDR image before starting processing of the second set of raw images for generating the second HDR image.

10. The sensor device according to claim 1, wherein the imaging unit is not configured to delay output of data of the first set of raw images in order to temporally align output of the data of the first set of raw images and the data of the second set of raw images to the imaging processing unit.

11. An image processing system comprising the sensor device according to claim 1 ; a display for presenting the first HDR image to a user of the image processing system; and / or a computer vision unit that is configured to execute predetermined processing based on the second HDR image.

12. A method for operating the sensor device of claim 1, the method comprising: capturing, by the imaging unit, a first set of raw images of the scene with different sensitivities for brightness using a first exposure period and a second set of raw images of the scene with different sensitivities for brightness using a second exposure period; generating, by the image processing unit, a first HDR image from the first set of raw images and a second HDR image from the second set of raw images; wherein the first exposure period is longer than the second exposure period; andthe highest sensitivity for brightness of the raw images of the first set is higher than the highest sensitivity for brightness of the raw images of the second set.

Images