Image generation device, image generation method, and image generation program

Abstract

An image generation device according to the present disclosure includes a light source unit, a pixel array unit, and an image generation unit. The light source unit is capable of changing the brightness of light in a plurality of wavelength regions. In the pixel array unit, a plurality of pixels including photoelectric conversion units are arranged in a column direction and a row direction. The pixel array unit receives light from an object to which light of a plurality of wavelength regions is emitted. The image generation unit generates an image having light components of the plurality of wavelength regions based on timing at which brightness of light of each of the plurality of wavelength regions of the light source unit changes and a detection result of the pixel array unit when the brightness of the light source unit changes.

Description

Technical Field

[0001] The present disclosure relates to an image generation device, an image generation method, and an image generation program. Background Art

[0002] For solid-state imaging elements using complementary metal oxide semiconductors (CMOS) or the like, asynchronous solid-state imaging elements have been proposed. The asynchronous solid-state imaging elements detect the luminance change of each pixel in real time as an event signal (for example, Patent Document 1). A solid-state imaging element that detects the event signal of each pixel in this way is also called an event-based vision sensor (EVS).

[0003] Citation List

[0004] Patent Document

[0005] Patent Document 1: JP 2017-535999 A Summary of the Invention

[0006] Technical Problem

[0007] The present disclosure provides an image generation device, an image generation method, and an image generation program that can improve the convenience of a solid-state imaging element.

[0008] Solution to the Technical Problem

[0009] According to the present disclosure, there is provided an image generation device. The image generation device includes a light source unit, a pixel array unit, and an image generation unit. The light source unit is configured to change the light intensity of each of the lights in a plurality of wavelength bands. The pixel array unit includes a plurality of pixels each having a photoelectric conversion unit arranged in a row direction and a column direction, and receives light from an object irradiated with the lights in the plurality of wavelength bands. The image generation unit generates an image having components of the lights in the plurality of wavelength bands based on the timing of the change in the light intensity of each of the lights in the plurality of wavelength bands of the light source unit and the detection result of the pixel array unit when the light intensity of the light source unit changes. Brief Description of the Drawings

[0010] Figure 1 is a diagram showing an exemplary schematic configuration of an image generation device according to a first embodiment.

[0011] Figure 2 is a block diagram showing an exemplary system configuration of an image generation device according to a first embodiment.

[0012] Figure 3 is a block diagram showing an exemplary schematic configuration of an EVS according to a first embodiment.

[0013] Figure 4It is a diagram showing an exemplary process executed by an image generation device according to a first embodiment.

[0014] Figure 5 It is a graph showing a mechanism configured to reproduce grayscale in an image generation device according to a first embodiment.

[0015] Figure 6 It is a graph showing a mechanism configured to reproduce grayscale in an image generation device according to a first embodiment.

[0016] Figure 7 It is a diagram showing an exemplary process executed by an image generation device according to a first embodiment.

[0017] Figure 8 It is a diagram showing an exemplary process executed by an image generation device according to a first embodiment.

[0018] Figure 9 It is a diagram showing an exemplary operation of an image generation device according to a first embodiment.

[0019] Figure 10 It is a block diagram showing an exemplary system configuration of an image generation device according to a first modification of a first embodiment.

[0020] Figure 11 It is a diagram showing an exemplary operation of an image generation device according to a first modification of a first embodiment.

[0021] Figure 12 It is a block diagram showing an exemplary system configuration of an image generation device according to a second modification of a first embodiment.

[0022] Figure 13 It is a diagram showing an exemplary operation of an image generation device according to a second modification of a first embodiment.

[0023] Figure 14 It is a block diagram showing an exemplary system configuration of an image generation device according to a third modification of a first embodiment.

[0024] Figure 15 It is a block diagram showing an exemplary system configuration of an image generation device according to a second embodiment.

[0025] Figure 16 It is a block diagram showing an exemplary schematic configuration of a black-and-white image sensor according to a second embodiment.

[0026] Figure 17 It is a diagram showing an exemplary process executed by an image generation device according to a second embodiment.

[0027] Figure 18 is a diagram showing an exemplary process performed by an image generation device according to the second embodiment.

[0028] Figure 19 is a diagram showing an exemplary process performed by an image generation device according to the second embodiment.

[0029] Figure 20 is a flowchart showing an exemplary process of image generation processing performed by an image generation device according to the first embodiment.

[0030] Figure 21 is a flowchart showing an exemplary process of image generation processing performed by an image generation device according to the second embodiment. Detailed Embodiments

[0031] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the present disclosure is not limited to the following embodiments. In addition, the embodiments can be appropriately combined within a range where there is no contradiction in the processing content. In addition, in the following embodiments, the same parts are denoted by the same reference numerals, and their descriptions will not be repeated.

[0032] For solid-state imaging elements using complementary metal oxide semiconductors (CMOS) or the like, asynchronous solid-state imaging elements have been proposed, which detect luminance changes of each pixel in real time as event signals. A solid-state imaging element that detects event signals of each pixel in this way is also called an event-based vision sensor (EVS).

[0033] However, the above-described conventional technology only allows the generation of binary event signals for each pixel, and therefore, it is difficult to generate an image with grayscale by the same solid-state imaging element. In other words, in the above-described conventional technology, there is room for further improving the convenience of solid-state imaging elements such as EVS.

[0034] Therefore, it is desirable to implement a technology that can overcome the above problems and improve the convenience of solid-state imaging elements.

[0035] <First Embodiment>

[0036] First, the first embodiment will be described in detail with reference to the drawings.

[0037] <Exemplary Configuration of Image Generation Device>

[0038] Figure 1 is a schematic diagram showing an exemplary schematic configuration of an image generation device 1 according to the first embodiment, Figure 2 is a block diagram showing an exemplary system configuration of an image generation device 1 according to the first embodiment.

[0039] As Figure 1 shown, the image generation device 1 according to the first embodiment includes a light source unit 10, an illumination lens 20, an imaging lens 30, an EVS 40, and a system control unit 50. The system control unit 50 is an example of an image generation unit.

[0040] As Figure 2 shown, the light source unit 10 includes, for example, a red light-emitting diode (LED) light source 11, a green LED light source 12, a blue LED light source 13, and a light source driving unit 14. Alternatively, each of the LED light sources may use a laser diode (LD) instead of an LED to emit light that is scattered / diffused as needed by an optical system such as a diffuser plate.

[0041] The red LED light source 11 is an LED light source that emits light within the red wavelength band (hereinafter, also referred to as red light L R (see Figure 1 ). The green LED light source 12 is an LED light source that emits light within the green wavelength band (hereinafter also referred to as green light L G (see Figure 1 ).

[0042] The blue LED light source 13 is an LED light source that emits light within the blue wavelength band (hereinafter, also referred to as blue light L B (see Figure 1 ). The light source driving unit 14 is configured to independently drive the red LED light source 11, the green LED light source 12, and the blue LED light source 13.

[0043] As Figure 1 shown, the illumination lens 20 is disposed on the emission surface side of the light source unit 10 to convert each of the red light L R , green light L G , and blue light L B emitted from the light source unit 10 into illumination light having a predetermined divergence angle. Note that the illumination lens 20 may be combined with a folded optical system using a prism or a mirror, or diffused light from the light source may be directly used as illumination light without the illumination lens 20.

[0044] The imaging lens 30 is disposed on the light receiving surface side of the EVS 40 to form an image of the incident light on the light receiving surface of the EVS 40. The incident light may also include reflected light L x emitted from the light source unit 10 and reflected by the object T.

[0045] Although it will be described in detail later, as Figure 2As shown, the EVS 40 includes, for example, a pixel array unit 41 and a sensor control unit 42. In the pixel array unit 41, pixels that each detect an event (hereinafter referred to as event pixels 41a (see Figure 3 )) are arranged to form a two-dimensional grid. The sensor control unit 42 drives the pixel array unit 41 to generate event data including event signals S detected by each of the event pixels 41a (see Figure 5 ).

[0046] The system control unit 50 includes, for example, a processor (CPU) and drives a red LED light source 11, a green LED light source 12, and a blue LED light source 13 via a light source drive unit 14.

[0047] In addition, the system control unit 50 controls the EVS 40 in parallel with the control of the light source unit 10, thereby acquiring event data including event signals S detected according to changes in the light intensity in the light source unit 10.

[0048] For example, the irradiation light emitted from the light source unit 10 is projected onto the object T through the irradiation lens 20. The projected light is reflected by the object T. Then, the light reflected by the object T passes through the imaging lens 30 and enters the EVS 40.

[0049] The EVS 40 receives the light reflected by the object T, generates an event signal S, and generates event data including the generated event signal S. The event data generated by the EVS 40 is provided to the system control unit 50. The system control unit 50 performs processing on the event data input from the EVS 40, which will be described later.

[0050] Figure 3 is a block diagram showing an exemplary schematic configuration of the EVS 40 according to the first embodiment. As Figure 3 shown, the EVS 40 includes a pixel array unit 41, a sensor control unit 42, an X arbiter 43 and a Y arbiter 44, an event signal processing circuit 45, and an output interface (I / F) 46.

[0051] The pixel array unit 41 has a configuration in which a plurality of event pixels 41a that each detect an event based on a change in the brightness of incident light are arranged to form a two-dimensional grid. Each of the event pixels 41a is an example of a pixel.

[0052] Note that in the following description, the row direction indicates the arrangement direction of pixels in a pixel row (the horizontal direction in the drawings), and the column direction indicates the arrangement direction of pixels in a pixel column (the vertical direction in the figures).

[0053] The event pixel 41a includes a photoelectric conversion element that generates charge according to the luminance of incident light. When the event pixel 41a detects a change in the luminance of incident light based on the photocurrent flowing out of the photoelectric conversion element, the event pixel 41a outputs a read request from itself to the X arbiter 43 and the Y arbiter 44.

[0054] Then, based on the discrimination of the X arbiter 43 and the Y arbiter 44, the event pixel 41a outputs an event signal S indicating that an event has been detected (see Figure 5 ).

[0055] The event pixel 41a detects the presence or absence of an event based on whether a change exceeding a predetermined threshold width W (see Figure 5 ) has occurred in the photocurrent according to the luminance of incident light. For example, the event pixel 41a detects a luminance change exceeding the predetermined threshold width W (positive event) or a luminance change less than the predetermined threshold width W (negative event) as an event.

[0056] When an event is detected, the event pixel 41a outputs a request to each of the X arbiter 43 and the Y arbiter 44 to permit the output of the event signal S indicating the occurrence of the event.

[0057] Then, when the event pixel 41a receives a response indicating permission to output the event signal S from each of the X arbiter 43 and the Y arbiter 44, the event pixel 41a outputs the event signal S to the event signal processing circuit 45.

[0058] The X arbiter 43 and the Y arbiter 44 discriminate requests to output the event signal S provided from each of the plurality of event pixels 41a. Then, the X arbiter 43 and the Y arbiter 44 send a response based on the discrimination result (permission / non - permission to output the event signal S) and a reset signal for resetting the detection of the event to the event pixel 41a that has output the request.

[0059] The event signal processing circuit 45 performs predetermined signal processing on the event signal S input from the event pixel 41a to generate and output event data.

[0060] As described above, the change in the photocurrent generated in the event pixel 41a can also be regarded as a change in the amount of light (luminance change) entering the photoelectric conversion unit of the event pixel 41a. Therefore, it can also be said that the change in the amount of light (luminance change) of the event pixel 41a exceeds the predetermined threshold width W.

[0061] The event data indicating the occurrence of an event includes at least position information such as coordinates, which indicate the position of the event pixel 41a where the change in the amount of light as an event has occurred. In addition to the position information, the event data may include the polarity of the change in the amount of light.

[0062] Event data output from event pixel 41a at the timing of event occurrence implicitly includes time information indicating the relative time of event occurrence as long as the intervals between the event data are maintained unchanged from the time of event occurrence.

[0063] However, when the intervals between the event data are not maintained unchanged from the time of event occurrence but are changed due to storage of event data in a memory or the like, the time information implicitly included in the event data is lost.

[0064] Therefore, until the intervals between the event data are maintained unchanged from the time of event occurrence, the event signal processing circuit 45 can include in the event data time information such as a timestamp indicating the relative time or absolute time when the event occurred.

[0065] The sensor control unit 42 includes a timing generator that generates various timing signals. The sensor control unit 42 controls the driving of the X arbiter 43, Y arbiter 44, event signal processing circuit 45, etc. based on the various timings generated by the timing generator.

[0066] The output I / F 46 sequentially outputs the event data output from the event signal processing circuit 45 to the system control unit 50 (see Figure 2 ).

[0067] <Details of image generation processing>

[0068] Next, details of the image generation processing according to the first embodiment will be described with reference to Figures 4 to 9 FIG. Figure 4 FIG. is a diagram showing an exemplary process executed by the image generation apparatus 1 according to the first embodiment.

[0069] As Figure 4 shown, in the first embodiment, the system control unit 50 first controls the light source unit 10 to pulse-irradiate the object T with red light L R from the red LED light source 11 (step S01).

[0070] Note that in the present disclosure, "pulse irradiation" may be light irradiation that repeatedly turns on and off the light, or light irradiation that repeatedly alternates between high light intensity (e.g., 100% light intensity) and low light intensity (e.g., 50% light intensity).

[0071] In addition, in parallel with the processing of step S01, the system control unit 50 controls the EVS 40 to acquire the cumulative number of event signals S of each event pixel 41a (see Figure 5 ), where the event signal S is generated by the reflected light L R caused by the red light L x (step S02).

[0072] Here, reference will be made to Figure 5 and Figure 6 to describe the mechanism of the image generation process according to the first embodiment. Figure 5 and Figure 6 are graphs each showing the mechanism for reproducing grayscale in the image generation apparatus 1 according to the first embodiment.

[0073] Generally, the grayscale of an image is dark at a portion of the object T having a low reflectance, medium at a portion having a medium reflectance, bright at a portion having a high reflectance, and very bright at a portion having a very high reflectance.

[0074] Therefore, as Figure 5 represented by the pulse on the left side, for example, two event signals S are generated in a single pulse irradiation at a portion having a medium reflectance. Further, as shown by the pulse on the Figure 5 right side of, for example, the event signal S may not be randomly generated at a location having a low reflectance.

[0075] Meanwhile, as Figure 6 shown, at a portion having a very high reflectance, for example, more than two event signals S (six signals S in the drawing) are generated in a single pulse irradiation. This is because, as Figure 6 shown in, the brightness of the irradiation light of the LED (i.e., the brightness of the reflected light L x ) actually changes over time, and thus, when the amount of reflected light is very large, a large number of events occur continuously.

[0076] In addition, the process of amplifying the analog signal from the photoelectric conversion element and extracting the analog signal is a transient phenomenon having a time constant above a predetermined value, and this transient phenomenon gives an effect equivalent to the effect of the change in the amount of reflected light over a long period of time, contributing to the continuous occurrence of a large number of events.

[0077] Therefore, in the first embodiment, for each event pixel 41a (see Figure 3 ), obtaining the cumulative number of event signals S generated by irradiating with a red light L R pulse makes it possible to reproduce the red grayscale for each event pixel 41a.

[0078] Figure 7 and Figure 8 are diagrams each showing an exemplary process performed by the image generation apparatus 1 according to the first embodiment. After the process of Figure 4 , in the first embodiment, the system control unit 50 controls the light source unit 10 to irradiate the object T with a green light L G pulse from the green LED light source 12 (step S03).

[0079] In addition, in parallel with the processing of step S03, the system control unit 50 controls the EVS 40 to acquire the cumulative number of event signals S (see G ) generated by the reflected light L caused by the green light L x (step S04). Figure 5 )

[0080] Next, as shown in Figure 8 , the system control unit 50 controls the light source unit 10 to pulse-irradiate the object T with the blue light L from the blue LED light source 13 (step S05). B

[0081] In addition, in parallel with the processing of step S05, the system control unit 50 controls the EVS 40 to acquire the cumulative number of event signals S (see B ) generated by the reflected light L caused by the blue light L x (step S06). Figure 5 )

[0082] Next, the system control unit 50 converts the cumulative number of event signals S in each color into RGB luminance values, and the event signals S in each color are acquired in the above processing (step S07). Then, the system control unit 50 generates an image based on the RGB luminance values (step S08), and ends a series of steps of the image generation process.

[0083] Note that in the above image generation process, an example has been described in which the object T is irradiated with light in the order of red light L R , green light L G , and blue light L B , but the present disclosure is not limited to this example, and the irradiation with light of the corresponding color can be performed in any order.

[0084] In this way, in the first embodiment, it is possible to use the EVS 40 to generate an image having gray levels of multiple colors (here, three RGB colors), and the EVS is configured to detect only the event signal S as a binary signal for each event pixel 41a (see Figure 3 ). Therefore, according to the first embodiment, the convenience of the EVS 40 can be improved.

[0085] In addition, in the first embodiment, the gray levels of multiple colors can be detected in all event pixels 41a that can be set in the pixel array unit 41. This configuration makes it possible to reduce the occurrence of false colors that cause problems in an image sensor configured to detect only a single color for each pixel.

[0086] ​In addition, in the first embodiment, the brightness of the ambient light around the object T does not change significantly in a short period of time, and thus, the ambient light does not contribute to the occurrence of events in the EVS 40. In other words, in the first embodiment, regardless of the difference in ambient light, similar color images can be generated, and thus, color images that are not affected by the color temperature of the ambient light or the like can be generated.

[0087] In addition, in the first embodiment, compared with existing multispectral cameras, color images can be generated at a relatively low cost. In addition, when a laser such as an LD is used instead of an LED for the light source, the laser having a very narrow frequency width can generate color images with high spectral resolution.

[0088] In addition, in the first embodiment, the system control unit 50 preferably irradiates the object T with pulsed light of red light L R , green light L G and blue light L B , rather than continuous light of red light L R , green light L G and blue light L B . This configuration allows more event signals S to be generated with a large number of light pulses, and thus, color images with good gray levels can be generated.

[0089] In addition, in the first embodiment, the light source driving unit 14 preferably applies the light of each color exclusively to the object T. This is because it is very difficult to determine which applied light causes the generation of the event signal S when lights of multiple colors are applied simultaneously.

[0090] In addition, in the above first embodiment, an example in which the light source unit 10 and the EVS 40 operate while being synchronized by the system control unit 50 has been described, but the present disclosure is not limited to this example.

[0091] For example, the start time and the end time of the emission from each LED light source in the light source unit 10 are recorded as absolute times, and the progress of the occurrence of the event signal S in the EVS 40 is recorded with a time stamp indicating the absolute time.

[0092] Then, after the operations of the light source unit 10 and the EVS 40 are completed, the system control unit 50 can generate an image by obtaining the progress of the operation of each LED light source and the progress of the occurrence of the event signal S in the EVS 40 and respectively calculating the cumulative number of the event signals S in each color.

[0093] This configuration also enables an image with multiple colors of grayscale to be generated by using an EVS 40 configured to detect only an event signal S as a binary signal for each event pixel 41a. Therefore, according to the first embodiment, the convenience of the EVS 40 can be improved.

[0094] Note that at this time, for example, the object T can be irradiated with red light L R , green light L G and blue light L B , where the values of the pulse frequencies, etc. of the red LED light source 11, the green LED light source 12, and the blue LED light source 13 are different from each other.

[0095] This configuration enables the system control unit 50 to easily obtain, from the trigger signal pulse queue, the timing of irradiation with each of the red light L R , green light L G and blue light L B when confirming the progress of the operation of each LED light source after the operation of the light source unit 10 ends. Therefore, according to the first embodiment, the image generation process can be easily performed.

[0096] Figure 9 is a diagram showing an exemplary operation of the image generation device 1 according to the first embodiment. In Figure 9 , application examples of the technology of the present disclosure in underwater research will be described.

[0097] First, as shown in (a) of Figure 9 , for an environment with poor visibility, the system control unit 50 (see Figure 1 ) generates an EVS image using one LED light source (i.e., monochromatic light) or an appropriate combination of multiple LED light sources (step S11). Note that in the process of step S11, a light source other than monochromatic light such as a white LED can also be used as needed.

[0098] Then, when an object is detected in the capture area (step S12), the system control unit 50 sets the area where the object is detected as the region of interest (ROI) (step S13).

[0099] In addition, as shown in (b) of Figure 9 , the system control unit 50 generates an EVS image of the set ROI by using multiple LED light sources (i.e., multi-color light) (step S14). Therefore, an EVS image of the area of interest can be generated with multi-color light.

[0100] In Figure 9In the example, an EVS image of multicolor light is generated only in the area of interest at necessary timings, and an EVS image of multicolor light is not generated at other timings and areas, so that the power consumption of the image generation device 1 can be reduced and the capacity of the generated image can be reduced.

[0101] <First Modification Example>

[0102] Next, various modification examples of the first embodiment will be described with reference to Figures 10 to 14 FIG. Figure 10 is a block diagram showing an exemplary system configuration of the image generation device 1 according to the first modification example of the first embodiment.

[0103] As Figure 10 shown, in the first modification example, the configuration of the light source unit 10 is different from the configuration of the above-described first embodiment. Therefore, in the following examples, parts similar to those of the above-described embodiment and the like are denoted by the same reference numerals, and their detailed descriptions will not be repeated.

[0104] As Figure 10 shown, in the first modification example, the light source unit 10 includes a red LED light source 11 and a near-infrared LED light source 15. The near-infrared LED light source 15 is an LED light source that emits light in the near-infrared wavelength band (hereinafter, also referred to as near-infrared light). Then, the light source drive unit 14 is configured to independently drive the red LED light source 11 and the near-infrared LED light source 15.

[0105] Figure 11 is a diagram showing an exemplary operation of the image generation device 1 according to the first modification example of the first embodiment. In Figure 11 this section, an application example of the technology of the present disclosure in the vegetation evaluation of the farmland F will be described.

[0106] In Figure 11 the example, for example, the image generation device 1 is mounted on the unmanned aerial vehicle D. Then, the system control unit 50 of the image generation device 1 mounted on the unmanned aerial vehicle D (see Figure 10 ) first images the farmland F with red light L R (see Figure 1 ) from above the farmland F (step S21).

[0107] Next, the system control unit 50 images the farmland F with near-infrared light (step S22). Then, the system control unit 50 generates an image indexed by the normalized difference vegetation index (NDVI) based on the processes of steps S21 and S22 (step S23).

[0108] NDVI is an index represented by the following formula (1), and a value closer to +1 indicates better vegetation.

[0109] NDVI = (NIR - RED) / (NIR + RED) (1)

[0110] NIR: The amount of light reflection in the near-infrared region

[0111] RED: The amount of light reflection in the red region

[0112] Then, in the technology of the present disclosure, as described above, an image that is not affected by the color temperature of ambient light or the like can be generated. In other words, in the first modification example, NDVI can be evaluated without being affected by ambient light. Therefore, according to the first modification example, the vegetation of the farmland F can be accurately evaluated.

[0113] <Second Modification Example>

[0114] In the above-described first embodiment and the first modification example, examples of the light source unit 10 configured to emit light in a plurality of wavelength bands have been described, but the present disclosure is not limited to these examples. Figure 12 It is a block diagram showing an exemplary system configuration of the image generation device 1 according to a second modification example of the first embodiment.

[0115] As Figure 12 shown, in the second modification example, the light source unit 10 includes an ultraviolet LED light source 16. The ultraviolet LED light source 16 is an LED light source that emits light in the ultraviolet wavelength band (hereinafter also referred to as ultraviolet light). The light source drive unit 14 is configured to drive the ultraviolet LED light source 16.

[0116] Figure 13 It is a diagram showing an exemplary operation of the image generation device 1 according to a second modification example of the first embodiment. In Figure 13 , application examples of the technology of the present disclosure in evaluating the fluorescent protein introduced into the organism I will be described.

[0117] In Figure 13 the example, the system control unit 50 of the image generation device 1 (see Figure 12 ) first irradiates the organism I with ultraviolet light and receives the fluorescence generated thereby to generate an EVS image of the organism I (step S31). When the EVS 40 is sensitive to ultraviolet light, an optical filter that blocks ultraviolet light can be installed as needed. Then, based on the generated EVS image, the fluorescence of the fluorescent protein in the organism I is evaluated (step S32).

[0118] Then, in the technology of the present disclosure, as described above, an image that is not affected by the color temperature of ambient light or the like can be generated. In other words, in the second modification example, the presence or absence of fluorescence in the organism I can be evaluated without being affected by ambient light. Therefore, according to the second modification example, the fluorescence of the fluorescent protein can be accurately evaluated.

[0119] Note that in the second modification example, an example using ultraviolet light has been described. However, depending on the characteristics of the fluorescent substance, the light source that generates fluorescence can be selected not only from ultraviolet light but also from blue light or light within other wavelength bands.

[0120] <Third Modification Example>

[0121] In the above-described first embodiment and the first and second modification examples, an example in which the light source unit 10 includes various LED light sources has been described. However, the present disclosure is not limited to this example. Figure 14 FIG. is a block diagram showing an exemplary system configuration of the image generation apparatus 1 according to a third modification example of the first embodiment.

[0122] As Figure 14 shown, in the third modification example, the light source unit 10 includes a red laser diode (LD) light source 11A, a green LD light source 12A, and a blue LD light source 13A.

[0123] The red LD light source 11A is an LD light source that emits red light L R . The green LD light source 12A is an LD light source that emits green light L G . The blue LD light source 13A is an LD light source that emits blue light L B . The light source drive unit 14 (see Figure 1 ) is configured to independently drive the red LD light source 11A, the green LD light source 12A, and the blue LD light source 13A.

[0124] In the third modification example, all the light sources include LDs, and the irradiation light (i.e., laser) has high directivity. Therefore, in the third modification example, while the laser light emitted from the red LD light source 11A, the green LD light source 12A, and the blue LD light source 13A is plane-scanned by the optical system 20A, the object T is irradiated with light.

[0125] The optical system 20A includes optical members 21 to 23, a horizontal scanning member 24, and a vertical scanning member 25. The optical member 21 bends the red light L R toward a predetermined optical path. The optical member 22 causes the red light L R to be transmitted on the predetermined optical path and bends the green light L G toward a predetermined optical path. The optical member 23 causes the red light L R or the green light L G to be transmitted on the predetermined optical path and bends the blue light L B toward a predetermined optical path.

[0126] The horizontal scanning member 24 causes the red light L R , the green light L G or the blue light L BBend so that the red light L on the predetermined optical path is scanned horizontally in the object T R , the green light L G or the blue light L B .

[0127] The vertical scanning member 25 bends the red light L R , the green light L G or the blue light L B so that the red light L bent in the horizontal scanning member 24 is scanned vertically in the object T R , the green light L G or the blue light L B .

[0128] Then, in the third modification, the system control unit 50 (see Figure 1 ) controls the light source unit 10 and the optical system 20A to scan the red light L as a laser in a plane R , the green light L G or the blue light L B while irradiating the object T. The reflected light L reflected by the object T x passes through the imaging lens 30 and enters the EVS 40.

[0129] This configuration enables the EVS 40 to generate an image with gray levels of multiple colors (here, three RGB colors). Therefore, according to the third modification, the convenience of the EVS 40 can be improved.

[0130] In addition, in the third modification, it is allowed to generate an EVS image by using a laser with a very narrow frequency width, and a color image with high spectral resolution can be generated.

[0131] <Second Embodiment>

[0132] First, the second embodiment will be described in detail with reference to the accompanying drawings.

[0133] <Exemplary Configuration of Image Generation Device>

[0134] Figure 15 is a block diagram showing an exemplary system configuration of the image generation device 1 according to the second embodiment. As Figure 15 shown, the image generation device 1 according to the second embodiment includes a light source unit 10, an illumination lens 20 (see Figure 1 ), an imaging lens 30 (see Figure 1 ), a black-and-white image sensor 140, and a system control unit 50.

[0135] The configurations of the light source unit 10, the illumination lens 20, and the imaging lens 30 are similar to those of the above-described first embodiment, and thus, a detailed description thereof will not be repeated. Note that, as in the first embodiment, the illumination lens 20 may be combined with a folding optical system, or the illumination lens 20 itself may be removed.

[0136] As Figure 15 shown, a black-and-white image sensor 140, which will be described in detail later, includes, for example, a pixel array unit 141 and a sensor control unit 142. In the pixel array unit 141, pixels 141a (see Figure 16 ) each having a photoelectric conversion element are arranged to form a two-dimensional grid.

[0137] The system control unit 50 includes, for example, a processor (CPU) and drives a red LED light source 11, a green LED light source 12, and a blue LED light source 13 via a light source driving unit 14.

[0138] In addition, the system control unit 50 controls the black-and-white image sensor 140 in parallel with the control of the light source unit 10, thereby acquiring image data captured according to changes in the light intensity in the light source unit 10.

[0139] Figure 16 is a block diagram showing an exemplary schematic configuration of a black-and-white image sensor 140 according to the second embodiment.

[0140] As Figure 16 shown, the black-and-white image sensor 140 has, for example, a stacked structure in which a semiconductor chip on which a pixel array unit 141 is formed and a semiconductor chip on which a peripheral circuit is formed are stacked. The peripheral circuit may include, for example, a sensor control unit 142, a vertical drive circuit 143, a column processing circuit 144, and a horizontal drive circuit 145.

[0141] The black-and-white image sensor 140 further includes a signal processing unit 146 and a data storage unit 147. The signal processing unit 146 and the data storage unit 147 may be provided on the same semiconductor chip on which the peripheral circuit is provided, or may be provided on different semiconductor chips.

[0142] The pixel array unit 141 has a configuration in which pixels 141a are arranged in a row direction and a column direction (i.e., in a matrix) to form a two-dimensional grid, and these pixels each have a photoelectric conversion element that generates and accumulates charges according to the amount of received light.

[0143] Note that, in the pixel array unit 141 according to the second embodiment, color filters and the like are not provided in at least some of the pixels 141a. Therefore, the black-and-white image sensor 140 generates monochromatic image data at the positions of some pixels.

[0144] In the pixel array unit 141, for the pixel array in the matrix, pixel driving lines LD are routed in the row direction for each pixel row, and vertical signal lines VSL are routed in the column direction for each pixel column. The pixel driving lines LD transmit driving signals for driving signals read from the pixels.

[0145] In Figure 16 , the pixel driving lines LD are shown as being wired one by one, but are not limited to being wired one by one. One end of the pixel driving lines LD has an output end corresponding to each row connected to the vertical driving circuit 143.

[0146] The vertical driving circuit 143 includes a shift register, an address decoder, etc., and drives all the pixels 141a of the pixel array unit 141 or the pixels 141a in each row at the same time. In other words, the vertical driving circuit 143 and the sensor control unit 142 that controls the vertical driving circuit 143 together constitute a driving unit that controls the operation of each pixel of the pixel array unit 141.

[0147] Although the specific configuration of the vertical driving circuit 143 is not shown, a general vertical driving circuit includes two scanning systems: a readout scanning system and a sweep scanning system.

[0148] In order to read signals from the pixels 141a, the readout scanning system sequentially and selectively scans the pixels 141a of the pixel array unit 141 for each row. The signals read from each pixel 141a are analog signals. The sweep scanning system performs a sweep scan on the read rows to be read-scanned by the readout scanning system with an exposure time earlier than the readout scan.

[0149] Unnecessary charges are swept out from the photoelectric conversion elements of the pixels 141a in the read rows through the sweep scan of the sweep scanning system, whereby the photoelectric conversion elements are reset. Then, a so-called electronic shutter operation is performed by sweeping out (resetting) unnecessary charges in this sweep scanning system. Here, the electronic shutter operation refers to an operation of discharging the charges of the photoelectric conversion elements and newly starting exposure (starting the accumulation of charges).

[0150] The signals read by the readout scanning system in the readout operation correspond to the amount of light received after the last readout operation or the electronic shutter operation. Then, the period from the read timing of the last readout operation or the sweep timing of the electronic shutter operation to the read timing of this readout operation is the charge accumulation period (also called the exposure period) in the pixels 141a.

[0151] For each pixel column, each of the signals output from the corresponding pixels 141a in the pixel rows selectively scanned by the vertical driving circuit 143 is input to the column processing circuit 144 through each of the vertical signal lines VSL. For each pixel column of the pixel array unit 141, the column processing circuit 144 performs predetermined signal processing on the signals output from each pixel in the selected row through the vertical signal lines VSL, and temporarily stores the pixel signals obtained after the signal processing.

[0152] Specifically, the column processing circuit 144 performs at least noise removal processing as signal processing, such as correlated double sampling (CDS) processing or double data sampling (DDS) processing.

[0153] For example, fixed pattern noise inherent in pixels, such as reset noise and threshold variations of the amplification transistors in the pixels, is removed by CDS processing. The column processing circuit 144 also has, for example, an analog-to-digital (AD) conversion function of converting the analog pixel signals read from the photoelectric conversion elements into digital signals and outputting the digital signals.

[0154] The horizontal driving circuit 145 includes a shift register, an address decoder, etc., and sequentially selects the readout circuits (hereinafter referred to as pixel circuits) of the column processing circuits 144 corresponding to the pixel columns. This selective scanning by the horizontal driving circuit 145 causes the pixel signals obtained by signal processing to be sequentially output for each pixel circuit in the column processing circuit 144.

[0155] The sensor control unit 142 includes a timing generator that generates various timing signals, etc. The sensor control unit 142 controls the driving of the vertical driving circuit 143, the column processing circuit 144, the horizontal driving circuit 145, etc. based on the various timings generated by the timing generator.

[0156] The signal processing unit 146 has at least an arithmetic processing function, and performs various signal processing such as arithmetic processing on the pixel signals output from the column processing circuit 144. For the signal processing in the signal processing unit 146, the data storage unit 147 temporarily stores the data required for the signal processing.

[0157] <Details of image generation processing>

[0158] Next, refer to Figures 17 to 19 Describe the details of the image generation processing according to the second embodiment. Figures 17 to 19 are diagrams each showing an exemplary process performed by the image generation device 1 according to the second embodiment.

[0159] As Figure 17As shown, in the second embodiment, the system control unit 50 first controls the light source unit 10 to change the light intensity of the red light L emitted from the red LED light source 11 toward the object T R (step S41).

[0160] Note that in the present disclosure, "changing the light intensity of the light for irradiation" may be light irradiation in which the light intensity changes between the turning on and turning off of the light, or may be light irradiation in which the light intensity changes between a high light intensity (e.g., 100% light intensity) and a low light intensity (e.g., 50% light intensity).

[0161] In addition, in parallel with the processing of step S41, the system control unit 50 controls the black-and-white image sensor 140 to acquire the brightness of the red light L having a high light intensity R and the brightness of the red light L having a low light intensity. R Then, the system control unit 50 acquires the brightness difference of the red light L between the high light intensity and the low light intensity R (step S42).

[0162] Here, in the region of the object T having a red component, the brightness increases as the light intensity of the red light L R increases, and thus, the difference in the brightness of the red light L between the high light intensity and the low light intensity R increases. On the other hand, in the region of the object T without a red component, even if the light intensity of the red light L R increases, the brightness does not increase, and thus, the difference in the brightness of the red light L between the high light intensity and the low light intensity R decreases.

[0163] Therefore, in the second embodiment, even the black-and-white image sensor 140 configured to acquire only monochromatic image data allows the reproduction of red grayscale for each pixel 141a.

[0164] Next, as Figure 18 shown, the system control unit 50 controls the light source unit 10 to change the light intensity of the green light L emitted from the green LED light source 12 toward the object T G (step S43).

[0165] In addition, in parallel with the processing of step S43, the system control unit 50 controls the black-and-white image sensor 140 to acquire the brightness of the green light L having a high light intensity G and the brightness of the green light L having a low light intensity. G Then, the system control unit 50 acquires the brightness difference of the green light L between the high light intensity and the low light intensity G (step S44).

[0166] Next, as Figure 19As shown, the system control unit 50 controls the light source unit 10 to change the blue light L emitted from the blue LED light source 13 toward the object T B in light intensity (step S45).

[0167] In addition, in parallel with the processing of step S45, the system control unit 50 controls the black-and-white image sensor 140 to acquire the brightness of the blue light L with high light intensity B and the brightness of the blue light L with low light intensity B Then, the system control unit 50 obtains the brightness difference of the blue light L between the high light intensity and the low light intensity B (step S46).

[0168] Next, the system control unit 50 converts the brightness difference of each color obtained in the above processing into an RGB brightness value (step S47). Then, the system control unit 50 generates an image based on the RGB brightness value (step S48), and ends a series of steps of the image generation process.

[0169] In this way, in the second embodiment, it is allowed to generate an image with gray levels of multiple colors (here three RGB colors) using the black-and-white image sensor 140 configured to generate only monochromatic images. Therefore, according to the second embodiment, the convenience of the black-and-white image sensor 140 can be improved.

[0170] In addition, in the second embodiment, the gray levels of multiple colors can be detected in all the pixels 141a provided in the pixel array unit 141. This configuration enables reduction of the occurrence of false colors that cause problems in an image sensor configured to detect only a single color for each pixel.

[0171] Furthermore, in the second embodiment, the brightness of the ambient light around the object T does not change significantly in a short time, and thus, a large change in the brightness of the object T is not caused. In other words, in the second embodiment, similar color images can be generated regardless of the difference in ambient light, and thus, color images not affected by the color temperature of the ambient light and the like can be generated.

[0172] In addition, in the second embodiment, color images can be generated at a relatively low cost compared with existing multispectral cameras. In addition, when a laser such as an LD is used instead of an LED for the light source, the laser having a very narrow frequency width can generate a color image with high spectral resolution.

[0173] In the above embodiments, examples of generating a multicolor image by using the EVS 40 or the black-and-white image sensor 140 have been described, but the present disclosure is not limited to this example.

[0174] For example, an avalanche photodiode (APD) sensor or a single-photon avalanche diode (SPAD) sensor having a pixel array unit with pixels made of APD or SPAD can be used to generate a multi-color image. In addition, in the present disclosure, an infrared sensor can be used to form a multi-color image. This configuration also makes it possible to improve the convenience of these infrared sensors, APD sensors, and SPAD sensors.

[0175] In addition, in the present disclosure, the pixel array unit may be included in a sensor in which a plurality of sensors selected from an EVS, a black-and-white image sensor, an image sensor with a color filter, an infrared sensor, an APD sensor, and an SPAD sensor are combined and arranged.

[0176] Then, the sensor in which the plurality of sensors are combined and arranged can be used to generate a multi-color image. This configuration makes it possible to improve the convenience of the sensor in which the plurality of sensors are combined and arranged.

[0177] In the above-described embodiment, an example in which the system control unit 50 is provided inside the image generation device 1 has been described, but the present disclosure is not limited to this example. For example, the system control unit may be provided outside the image generation device provided with the light source unit, the pixel array unit, etc., such that the image generation device and the system control unit can be connected to each other via a network or the like.

[0178] <Steps of Image Generation Processing>

[0179] Next, with reference to Figure 20 and Figure 21 the steps of the image generation processing according to the embodiment will be described. Figure 20 is a flowchart showing exemplary steps of the image generation processing performed by the image generation device 1 according to the first embodiment.

[0180] First, the system control unit 50 controls the light source unit 10 to irradiate the object T with light pulses within a specific wavelength band (step S101). Then, the system control unit 50 controls the EVS 40 to acquire the cumulative number of event signals S generated due to the light applied to the object T (step S102).

[0181] Next, the system control unit 50 determines whether all the light within all the wavelength bands has been applied to the object T (step S103). Then, when the object T has not been irradiated with all the light within all the wavelength bands (step S103, NO), the system control unit 50 irradiates the object T with light within another wavelength band that has not been applied (step S104), and returns to the process of step S102.

[0182] On the other hand, when light in all wavelength bands is applied to the object T (step S103, YES), the system control unit 50 converts the cumulative number of event signals S generated from the emitted light in all wavelength bands into the brightness value of the corresponding wavelength band (step S105).

[0183] Finally, the system control unit 50 generates an image based on the converted brightness value (step S106), and ends a series of steps of the image generation process.

[0184] Figure 21 FIG. 7 is a flowchart showing an exemplary process of the image generation process executed by the image generation apparatus 1 according to the second embodiment.

[0185] First, the system control unit 50 controls the light source unit 10 to irradiate the object T with light in a specific wavelength band at a high light intensity (step S201). Then, the system control unit 50 controls the black-and-white image sensor 140 to acquire the brightness when the light is applied at a high light intensity (step S202).

[0186] Next, the system control unit 50 controls the light source unit 10 to irradiate the object T with the light being used at a low light intensity (step S203). Then, the system control unit 50 controls the black-and-white image sensor 140 to acquire the brightness when the light is applied at a low light intensity (step S204).

[0187] Note that in the present disclosure, the processes of steps S203 and S204 described above can be executed before the processes of steps S201 and S202.

[0188] Next, the system control unit 50 acquires the brightness difference of the light in the specific wavelength band between the high light intensity and the low light intensity (step S205).

[0189] Next, the system control unit 50 determines to irradiate the object T with light in all wavelength bands (step S206). Then, when the object T is not irradiated with light in all wavelength bands (step S206, NO), the system control unit 50 applies light in another wavelength band that has not been applied to the object T at a high light intensity (step S207), and returns to the process of step S202.

[0190] On the other hand, when the object T is irradiated with light in all wavelength bands (step S206, YES), the system control unit 50 converts the brightness difference of the light emitted in all wavelength bands into the brightness value of the corresponding wavelength band (step S208).

[0191] Finally, the system control unit 50 generates an image based on the converted brightness value (step S209), and ends a series of steps of the image generation process.

[0192] [Effect]

[0193] The image generation device 1 according to the embodiment includes a light source unit 10, a pixel array unit 41 (141), and an image generation unit (system control unit 50). The light source unit 10 is configured to change the light intensity of each of the lights in a plurality of wavelength bands. The pixel array unit 41 (141) includes a plurality of pixels having photoelectric conversion units arranged in a row direction and a column direction, and receives light from an object T irradiated with light in a plurality of wavelength bands. The image generation unit (system control unit 50) generates an image having components of light in a plurality of wavelength bands based on the timing of changing the light intensity of each of the lights in the plurality of wavelength bands in the light source unit 10 and the detection result of the pixel array unit 41 when the light intensity of the light source unit 10 changes.

[0194] This configuration enables the convenience of the solid-state imaging element to be improved.

[0195] In addition, in the image generation device 1 according to the embodiment, the pixel array unit 41 (141) includes any one of an EVS 40, a black-and-white image sensor 140, an infrared sensor, an APD sensor, and an SPAD sensor.

[0196] This configuration enables the convenience of the solid-state imaging element to be improved.

[0197] In addition, in the image generation device 1 according to the embodiment, the pixel array unit is included in a sensor in which a plurality of sensors selected from an EVS, a black-and-white image sensor, an image sensor having a color filter, an infrared sensor, an APD sensor, and an SPAD sensor are combined and arranged.

[0198] This configuration enables the convenience of the solid-state imaging element to be improved.

[0199] In addition, in the image generation device 1 according to the embodiment, the pixel array unit 41 is included in any one of an EVS 40, an APD sensor, and an SPAD sensor. The image generation unit generates an image having components of light in a plurality of wavelength bands based on the timing of changing the light intensity of each of the lights in the plurality of wavelength bands of the light source unit and the cumulative number of event signals detected by any one of an EVS sensor, an APD sensor, and an SPAD sensor when the light intensity of the light from the light source unit changes.

[0200] This configuration enables the convenience of the EVS 40 etc. to be improved.

[0201] In addition, in the image generation device 1 according to the embodiment, the pixel array unit 141 is included in the black-and-white image sensor 140. Further, the image generation unit generates an image having components of light in a plurality of wavelength bands based on the timing of changing the light intensity of each of the lights in the plurality of wavelength bands in the light source unit 10 and the luminance difference of the light detected by the black-and-white image sensor 140 when the light intensity of the light source unit 10 is changed.

[0202] This configuration enables the convenience of the black-and-white image sensor 140 to be improved.

[0203] In addition, in the image generation device 1 according to the embodiment, the light source unit 10 is configured to change the light intensity of each of the light in the red region, the light in the green region, and the light in the blue region.

[0204] This configuration enables a color image of three RGB colors to be generated.

[0205] In addition, in the image generation device 1 according to the embodiment, the light source unit 10 is configured to change the light intensity of each of the light in the red region and the light in the near-infrared region.

[0206] This configuration enables the vegetation of, for example, the farmland F to be accurately evaluated.

[0207] In addition, the image generation device 1 according to the embodiment includes a light source unit 10, an EVS 40, and an image generation unit (system control unit 50). The light source unit 10 is configured to change the light intensity of the light in one wavelength band. The EVS 40 includes a plurality of pixels (event pixels 41a) each of which detects a change in the luminance of the incident light and outputs an event signal S, and these pixels are arranged in a row direction and a column direction and receive the light from an object T irradiated with the light in one wavelength band. The image generation unit (system control unit 50) generates an image based on the timing of changing the light intensity of the light in one wavelength band in the light source unit 10 and the cumulative number of the event signals S respectively detected by the EVS 40 when the light intensity of the light source unit 10 is changed.

[0208] This configuration enables the convenience of the EVS 40 to be improved.

[0209] In addition, in the image generation device 1 according to the embodiment, the light source unit 10 is configured to change the light intensity of the light in the ultraviolet region.

[0210] This configuration enables the fluorescence of, for example, the fluorescent protein in the living body I to be accurately evaluated.

[0211] In addition, the image generation device 1 according to the embodiment includes a light source unit 10, a pixel array unit 41 (141), and an image generation unit (system control unit 50). The light source unit 10 is configured to change the light intensity of light within one wavelength band. The pixel array unit 41 (141) includes a plurality of pixels having photoelectric conversion units arranged in a row direction and a column direction, and receives light from an object T irradiated with light within one wavelength band. The image generation unit (system control unit 50) generates an image based on the timing of changing the light intensity of light within one wavelength band in the light source unit 10 and the detection result in the pixel array unit 41 (141) when the light intensity of the light source unit 10 is changed.

[0212] This configuration enables the convenience of the solid-state imaging element to be improved.

[0213] The image generation method according to the embodiment includes a light intensity change step (steps S101, S201, and S203), a light reception step (steps S102, S202, and S204), and a generation step (steps S106 and S209). In the light intensity change step (steps S101, S201, and S203), the light intensity of light within a plurality of wavelength bands is changed. In the light reception step (steps S102, S202, and S204), light from an object T irradiated with light within a plurality of wavelength bands is received. In the generation step, an image having components of light within a plurality of wavelength bands is generated based on the timing of changing the light intensity of each of the lights within the plurality of wavelength bands in the light intensity change step and the detection result during the light reception step when the light intensity is changed in the light intensity change step.

[0214] This configuration enables the convenience of the solid-state imaging element to be improved.

[0215] In addition, the image generation program according to the embodiment causes the image generation device 1 to execute a light intensity change process, a light reception process, and a generation process. In the light intensity change process, the light intensity of each of the lights within a plurality of wavelength bands is changed. In the light reception process, light from an object T irradiated with light within a plurality of wavelength bands is received. In the generation process, an image having components of light within a plurality of wavelength bands is generated based on the timing of changing the light intensity of each of the lights within the plurality of wavelength bands in the light intensity change process and the detection result during the light reception process when the light intensity is changed in the light intensity change process.

[0216] This configuration enables the convenience of the solid-state imaging element to be improved.

[0217] The embodiments of the present disclosure have been described above. However, the technical scope of the present disclosure is not limited to the above embodiments, and various modifications and changes can be made without departing from the spirit and scope of the present disclosure. In addition, the constituent elements of different embodiments and variations can be appropriately combined with each other.

[0218] In addition, the effects described herein are merely examples, and the present disclosure is not limited to these effects, but may have other effects.

[0219] Note that the present technology may also have the following configurations. (1)

[0221] An image generation device, comprising:

[0222] A light source unit capable of changing the light intensity of each of lights in a plurality of wavelength bands;

[0223] A pixel array unit in which a plurality of pixels each having a photoelectric conversion unit are arranged in a row direction and a column direction, and the pixel array unit receives light from an object irradiated with lights in a plurality of wavelength bands; and

[0224] An image generation unit that generates an image having components of lights in a plurality of wavelength bands based on the timing of the change in the light intensity of each of lights in a plurality of wavelength bands of the light source unit and the detection result of the pixel array unit when the light intensity of the light source unit changes. (2)

[0226] The image generation device according to the above (1), wherein

[0227] the pixel array unit includes any one of an event-based vision sensor (EVS), a black-and-white image sensor, an infrared sensor, an avalanche photodiode (APD) sensor, and a single-photon avalanche diode (SPAD) sensor. (3)

[0229] The image generation device according to the above (1), wherein

[0230] the pixel array unit is included in a sensor in which a plurality of sensors selected from an EVS, a black-and-white image sensor, an image sensor having a color filter, an infrared sensor, an APD sensor, and a SPAD sensor are combined and arranged. (4)

[0232] The image generation device according to the above (1) or (2), wherein

[0233] the pixel array unit includes any one of an EVS, an APD sensor, and a SPAD sensor, and

[0234] The image generation unit generates an image having components of light in a plurality of wavelength bands based on the timing of the change in the light intensity of each of the lights in the plurality of wavelength bands of the light source unit and the number of accumulated event signals detected by any one of the EVS, the APD sensor, and the SPAD sensor when the light intensity of the light source unit changes. (5)

[0236] The image generation device according to the above (1) or (2), wherein

[0237] The pixel array unit is included in either the black and white image sensor or the infrared sensor, and

[0238] The image generation unit generates an image having components of light in a plurality of wavelength bands based on the timing of the change in the light intensity of each of the lights in the plurality of wavelength bands of the light source unit and the brightness difference detected by either the black and white image sensor or the infrared sensor when the light intensity of the light source unit changes. (6)

[0240] The image generation device according to any one of the above (1) to (5), wherein

[0241] The light source unit is capable of changing the light intensity of each of the light in the red region, the light in the green region, and the light in the blue region. (7)

[0243] The image generation device according to any one of the above (1) to (5), wherein

[0244] The light source unit is capable of changing the light intensity of each of the light in the red region and the light in the near-infrared region. (8)

[0246] An image generation device, comprising:

[0247] A light source unit capable of changing the light intensity of light in one wavelength band;

[0248] An EVS in which a plurality of pixels each detecting a change in the brightness of incident light and outputting an event signal are arranged in a row direction and a column direction, and the EVS receives light from an object irradiated with light in one wavelength band; and

[0249] An image generation unit that generates an image based on the timing of the change in the light intensity of the light in one wavelength band of the light source unit and the number of accumulated event signals detected by the EVS when the light intensity of the light source unit changes. (9)

[0251] The image generation device according to the above (8), wherein

[0252] The light source unit can change the light intensity of light in the ultraviolet region. (10)

[0254] An image generation device, comprising:

[0255] A light source unit that can change the light intensity of light in a wavelength band;

[0256] A pixel array unit in which a plurality of pixels each having a photoelectric conversion unit are arranged in a row direction and a column direction, and the pixel array unit receives light from an object irradiated with light in a wavelength band; and

[0257] An image generation unit that generates an image based on the timing of the change in the light intensity of light in a wavelength band of the light source unit and the detection result of the pixel array unit when the light intensity of the light source unit changes. (11)

[0259] An image generation method, comprising:

[0260] A light intensity change step of changing the light intensity of each of the lights in a plurality of wavelength bands;

[0261] A light reception step of receiving light from an object irradiated with light in a plurality of wavelength bands; and

[0262] A generation step of generating an image having components of light in a plurality of wavelength bands based on the timing of changing the light intensity of each of the lights in a plurality of wavelength bands in the light intensity change step and the detection result performed during the light reception step when the light intensity changes in the light intensity change step. (12)

[0264] According to the image generation method of the above (11), wherein,

[0265] The light reception step is performed by any one of an event-based vision sensor (EVS), a black-and-white image sensor, an infrared sensor, an avalanche photodiode (APD) sensor, and a single-photon avalanche diode (SPAD) sensor. (13)

[0267] According to the image generation method of the above (11), wherein,

[0268] The light reception step is performed by a sensor in which a plurality of sensors selected from an EVS, a black-and-white image sensor, an image sensor with a color filter, an infrared sensor, an APD sensor, and a SPAD sensor are combined and arranged. (14)

[0270] According to the image generation method of the above (11) or (12), wherein,

[0271] Performing a light reception step by any one of an EVS sensor, an APD sensor, and a SPAD sensor, and

[0272] In the generation step, based on the timing of changing the light intensity of each of the lights in a plurality of wavelength bands in the light intensity change step and the cumulative number of event signals detected by any one of the EVS sensor, the APD sensor, and the SPAD sensor when the light intensity changes in the light intensity change step, an image having components of lights in a plurality of wavelength bands is generated. (15)

[0274] According to the image generation method of the above (11) or (12), wherein,

[0275] Performing a light reception step by any one of a black-and-white image sensor and an infrared sensor, and

[0276] In the generation step, based on the timing of changing the light intensity of each of the lights in a plurality of wavelength bands in the light intensity change step and the brightness difference detected by any one of the black-and-white image sensor and the infrared sensor when the light intensity changes in the light intensity change step, an image having components of lights in a plurality of wavelength bands is generated. (16)

[0278] According to the image generation method of any one of the above (11) to (15), wherein,

[0279] In the light intensity change step, changing the light intensity of each of the light in the red region, the light in the green region, and the light in the blue region. (17)

[0281] According to the image generation method of any one of the above (11) to (15), wherein,

[0282] In the light intensity change step, changing the light intensity of each of the light in the red region and the light in the near-infrared region. (18)

[0284] An image generation method, comprising:

[0285] A light intensity change step of changing the light intensity of light in a wavelength band;

[0286] A light reception step of receiving light from an object irradiated with light in a wavelength band by an EVS; and

[0287] A generation step of generating an image based on the timing of changing the light intensity within a wavelength band in the light intensity change step and the cumulative number of event signals respectively detected by an EVS when the light intensity changes in the light intensity change step. (19)

[0289] According to the image generation method of the above (18), wherein,

[0290] In the light intensity change step, change the light intensity of light within the ultraviolet region. (20)

[0292] An image generation method, comprising:

[0293] A light intensity change step of changing the light intensity within a wavelength band;

[0294] A light reception step of receiving light from an object irradiated with light within a wavelength band; and

[0295] A generation step of generating an image based on the timing of changing the light intensity within a wavelength band in the light intensity change step and the detection results obtained during the light reception step when the light intensity changes in the light intensity change step. (21)

[0297] An image generation program that causes an image generation device to execute:

[0298] A light intensity change process of changing the light intensity of each of lights within a plurality of wavelength bands;

[0299] A light reception process of receiving light from an object irradiated with lights within a plurality of wavelength bands; and

[0300] A generation process of generating an image having components of lights within a plurality of wavelength bands based on the timing of changing the light intensity of each of lights within a plurality of wavelength bands in the light intensity change process and the detection results obtained during the light reception process when the light intensity changes in the light intensity change process. (22)

[0302] According to the image generation program of the above (21), wherein,

[0303] The light reception process is performed by any one of an event-based vision sensor (EVS), a black-and-white image sensor, an infrared sensor, an avalanche photodiode (APD) sensor, and a single-photon avalanche diode (SPAD) sensor. (23)

[0305] According to the image generation program of the above (21), wherein,

[0306] A light reception process is performed by a sensor in which a plurality of sensors selected from an EVS, a black-and-white image sensor, an image sensor with a color filter, an infrared sensor, an APD sensor, and an SPAD sensor are combined and arranged. (24)

[0308] An image generation program according to the above (21) or (22), wherein

[0309] A light reception process is performed by any one of an EVS sensor, an APD sensor, and an SPAD sensor, and

[0310] In the generation process, an image having components of light in a plurality of wavelength bands is generated based on the timing of changing the light intensity of each of the lights in the plurality of wavelength bands in the light intensity change process and the cumulative number of event signals respectively detected by any one of the EVS sensor, the APD sensor, and the SPAD sensor when the light intensity changes in the light intensity change process. (25)

[0312] An image generation program according to the above (21) or (22), wherein

[0313] A light reception process is performed by any one of a black-and-white image sensor and an infrared sensor, and

[0314] In the generation process, an image having components of light in a plurality of wavelength bands is generated based on the timing of changing the light intensity of each of the lights in the plurality of wavelength bands in the light intensity change process and the brightness difference detected by any one of the black-and-white image sensor and the infrared sensor when the light intensity changes in the light intensity change process. (26)

[0316] An image generation program according to any one of the above (21) to (25), wherein

[0317] In the light intensity change process, the light intensity of each of the light in the red region, the light in the green region, and the light in the blue region is changed. (27)

[0319] An image generation program according to any one of the above (21) to (25), wherein

[0320] In the light intensity change process, the light intensity of each of the light in the red region and the light in the near-infrared region is changed. (28)

[0322] An image generation program, comprising:

[0323] A light intensity change process that changes the light intensity of light in one wavelength band;

[0324] A light reception step of receiving, by an EVS, light from an object irradiated with light within a wavelength band; and

[0325] A generation step of generating an image based on the timing of changing the light intensity of light within a wavelength band in a light intensity change step and the cumulative number of event signals respectively detected by the EVS when the light intensity changes in the light intensity change step. (29)

[0327] According to the image generation program of the above (28), wherein,

[0328] In the light intensity change step, the light intensity of light within the ultraviolet region is changed. (30)

[0330] An image generation program, comprising:

[0331] A light intensity change step of changing the light intensity of light within a wavelength band;

[0332] A light reception step of receiving light from an object irradiated with light within a wavelength band; and

[0333] A generation step of generating an image based on the timing of changing the light intensity of light within a wavelength band in the light intensity change step and the detection results obtained during the light reception step when the light intensity changes in the light intensity change step.

[0334] List of reference numerals

[0335] 1 Image generation device

[0336] 10 Light source unit

[0337] 11 Red LED light source

[0338] 12 Green LED light source

[0339] 13 Blue LED light source

[0340] 15 Near-infrared LED light source

[0341] 16 Ultraviolet LED light source

[0342] 40 EVS

[0343] 41 Pixel array unit

[0344] 41a Event pixel (an example of a pixel)

[0345] 50 System control unit (an example of an image generation unit)

[0346] 140 Black and white image sensor

[0347] 141 Pixel array unit

[0348] S event signal

[0349] T object.

Claims

1. An image generation device, comprising: A light source unit capable of changing the light intensity of each of lights in a plurality of wavelength bands; A pixel array unit in which a plurality of pixels each having a photoelectric conversion unit are arranged in a row direction and a column direction, and the pixel array unit receives light from an object irradiated with the lights in the plurality of wavelength bands; And An image generation unit that generates an image having components of lights in the plurality of wavelength bands based on the timing of the change in the light intensity of each of the lights in the plurality of wavelength bands of the light source unit and the detection result of the pixel array unit when the light intensity of the light source unit changes.

2. The image generation device according to claim 1, wherein The pixel array unit includes any one of an EVS (event-based vision sensor), a black-and-white image sensor, an infrared sensor, an APD (avalanche photodiode) sensor, and a SPAD (single-photon avalanche diode) sensor.

3. The image generation device according to claim 1, wherein The pixel array unit includes a sensor in which a plurality of sensors selected from an EVS, a black-and-white image sensor, an image sensor with a color filter, an infrared sensor, an APD sensor, and a SPAD sensor are combined and arranged.

4. The image generation device according to claim 1, wherein The pixel array unit includes any one of an EVS, an APD sensor, and a SPAD sensor, and The image generation unit generates the image having components of lights in the plurality of wavelength bands based on the timing of the change in the light intensity of each of the lights in the plurality of wavelength bands of the light source unit and the cumulative number of event signals respectively detected by any one of the EVS, the APD sensor, and the SPAD sensor when the light intensity of the light source unit changes.

5. The image generation device according to claim 1, wherein The pixel array unit includes any one of a black-and-white image sensor and an infrared sensor, and The image generation unit generates the image having components of lights in the plurality of wavelength bands based on the timing of the change in the light intensity of each of the lights in the plurality of wavelength bands of the light source unit and the brightness difference detected by any one of the black-and-white image sensor and the infrared sensor when the light intensity of the light source unit changes.

6. The image generation device according to claim 1, wherein The light source unit is capable of changing the light intensity of each of the light in the red region, the light in the green region, and the light in the blue region.

7. The image generation device according to claim 1, wherein The light source unit is capable of changing the light intensity of each of the light in the red region and the light in the near-infrared region.

8. An image generation device, comprising: A light source unit capable of changing the light intensity of the light in one wavelength band; An EVS in which a plurality of pixels each detecting a change in the brightness of incident light and outputting an event signal are arranged in row and column directions, and the EVS receives light from an object irradiated with light within the one wavelength band; And An image generation unit that generates an image based on the timing of the change in the light intensity of the light within the one wavelength band of the light source unit and the cumulative number of event signals each detected by the EVS when the light intensity of the light source unit changes.

9. The image generation device according to claim 8, wherein The light source unit is capable of changing the light intensity of light in the ultraviolet region.

10. An image generation device, comprising: A light source unit that is capable of changing the light intensity of light within one wavelength band; A pixel array unit in which a plurality of pixels each having a photoelectric conversion unit are arranged in row and column directions, and the pixel array unit receives light from an object irradiated with light within the one wavelength band; And An image generation unit that generates an image based on the timing of the change in the light intensity of the light within the one wavelength band of the light source unit and the detection result of the pixel array unit when the light intensity of the light source unit changes.

11. An image generation method, comprising: A light intensity change step of changing the light intensity of each of lights in a plurality of wavelength bands; A light reception step of receiving light from an object irradiated with the lights in the plurality of wavelength bands; And A generation step of generating an image having components of the lights in the plurality of wavelength bands based on the timing of changing the light intensity of each of the lights in the plurality of wavelength bands in the light intensity change step and the detection result during the light reception step when the light intensity changes in the light intensity change step.

12. An image generation program that causes an image generation device to execute: A light intensity change process of changing the light intensity of each of lights in a plurality of wavelength bands; A light reception process of receiving light from an object irradiated with the lights in the plurality of wavelength bands; And A generation process of generating an image having components of the lights in the plurality of wavelength bands based on the timing of changing the light intensity of each of the lights in the plurality of wavelength bands in the light intensity change process and the detection result during the light reception process when the light intensity changes in the light intensity change process.

Images