Imaging device and portable terminal

Abstract

The invention provides an imaging device and a portable terminal, which can obtain a high-resolution color image without losing the sensitivity of a distance image. The imaging device includes: a sensor including a plurality of visible light pixels and a plurality of non-visible light pixels; a non-visible light pixel driving unit that controls a first exposure to the non-visible light pixel, performs a first reading operation for reading a charge generated from the first exposure, and generates a distance image; and a visible light pixel driving unit that controls a second exposure to the visible light pixels, performs a second reading operation for reading electric charges generated from the second exposure, and generates a visible light image, and the invisible light pixel driving unit performs a first exposure to the two or more invisible light pixels such that the electric charges generated from the second exposure are read at the time of the first exposure. At least a part of the first exposure period overlaps the second exposure period, charges generated from the first exposure are added, the added charges are read in a first reading operation, and a distance image is generated based on the added charges.

Description

[0001] This application is a divisional application of international application number PCT / JP2017 / 027324 and application number 201780093058.7, which entered the Chinese national phase on January 10, 2020. Technical Field

[0002] The present invention relates to an imaging device and a portable terminal, and in particular to a technology effective for acquiring a range image. Background Art

[0003] In recent years, it is known that a technology is added to an imaging element for taking pictures to obtain a distance image. The distance image can be obtained, for example, using the TOF (Time-Of-Flight) method. The TOF method is a technology that emits pulsed light from a light source to an object, measures the time until the reflected light from the object reaches the observation point, and thereby measures the distance to the object.

[0004] As such an imaging element for acquiring both a normal image and a distance image, there is, for example, Patent Document 1. Patent Document 1 describes an imaging element including: a light emitting device for emitting light toward a subject; an imaging element having a plurality of pixels arranged two-dimensionally, including a photodiode (PD), a pixel memory connected to the PD, and an FD connected to the pixel memory via a switch; a driving circuit 105 for driving the imaging element, turning on the switch at a predetermined time from the start of the charge storage period during the light emission period of the light emitting device to the light emission period, reading a first signal corresponding to the charge stored in the FD after the switch is turned off, and reading a second signal corresponding to the charge stored in the pixel memory when the charge storage period has passed; and a signal processing circuit 107 for calculating the distance to the subject based on the first signal and the second signal.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2015-201733 Summary of the invention

[0008] Technical issues to be solved by the invention

[0009] In the imaging element of Patent Document 1, the driving method for reading signals from the imaging element when obtaining a normal image and when obtaining a range image is divided into two methods, namely, for the normal image and for the range image. Therefore, the normal image and the range image cannot be obtained simultaneously in the same frame timing.

[0010] Therefore, it is conceivable to switch between the color image and the range image for each frame to acquire them, but in this case, the acquisition frame rates of the color image and the range image are each halved.

[0011] As a result, it is difficult to obtain a high-resolution color image when acquiring a range image. In addition, in the range image, there is a concern that the signal-to-noise ratio will decrease, and the distance measurement accuracy will decrease.

[0012] An object of the present invention is to provide a technology capable of obtaining a high-resolution color image without impairing the sensitivity of a range image.

[0013] The above objects and features as well as other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.

[0014] Technical solutions to solve problems

[0015] The following is a brief description of the outline of representative inventions among the inventions disclosed in the present invention.

[0016] That is, a typical imaging element includes a sensor unit, a visible light pixel driving unit, and a non-visible light pixel driving unit. The sensor unit includes a plurality of visible light pixels having sensitivity to visible light and a plurality of non-visible light pixels having sensitivity to non-visible light.

[0017] The visible light pixel driving unit controls exposure of the visible light pixels and reading of charges obtained by photoelectric conversion of the visible light pixels by exposure. The non-visible light pixel driving unit controls exposure of the non-visible light pixels and reading of charges obtained by photoelectric conversion of the non-visible light pixels by exposure.

[0018] The non-visible light pixel driving unit exposes two or more pre-set non-visible light pixels during exposure and reading of the non-visible light pixels, adds charges obtained by photoelectric conversion of the two or more non-visible light pixels through exposure, and generates a distance image based on the added charges.

[0019] In particular, the non-visible light pixel driving unit performs control so that at least a portion of a first exposure period for exposing two or more non-visible light pixels overlaps with a second exposure period for exposing visible light pixels.

[0020] Effects of the Invention

[0021] The following briefly describes the effects that can be obtained by the representative inventions among the inventions disclosed in the present invention.

[0022] It can obtain high-resolution color images and high-SN ratio range images. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is an explanatory diagram showing an example of the structure of the image pickup element according to the first embodiment.

[0024] Figure 2 Yes means Figure 1 A block diagram showing an example of the structure of an imaging element.

[0025] Figure 3 Yes means Figure 2 An explanatory diagram of an example of an equivalent circuit diagram of a visible light pixel driving unit and a non-visible light pixel driving unit of an imaging element.

[0026] Figure 4 Yes means Figure 3 A timing diagram showing an example of the reading timing of the visible light pixel driving unit and the non-visible light pixel driving unit.

[0027] Figure 5 Yes means Figure 3 An illustration of another example of an equivalent circuit diagram of a non-visible light pixel driving unit.

[0028] Figure 6 Yes means Figure 4 A timing diagram showing another example of the read timing.

[0029] Figure 7 Yes means Figure 1 An explanatory diagram of an example of the light bandpass characteristics of the color filter included in the visible light pixel of the imaging element.

[0030] Figure 8 The following is a configuration example of a filter used in an image pickup element for capturing a color image that does not capture a distance image, which has been studied by the inventors of the present invention.

[0031] Fig. 9 Yes means Figure 1 An illustration of another example of the pixel arrangement of an imaging element.

[0032] Fig.10 Yes means Fig. 9 An illustration of another example of the pixel arrangement of an imaging element.

[0033] Fig.11 Yes means Fig.10 An illustration of another example of the pixel arrangement of an imaging element.

[0034] Fig.12 Yes means Figure 1 An illustration of another example of the pixel arrangement of an imaging element.

[0035] Fig.13 It means use Figure 2 An explanatory diagram of an example of an imaging device having an imaging element.

[0036] Fig.14 It means use Fig.13 An illustration of an example of a portable terminal having an imaging device.

[0037] Fig.15 Yes means Fig.14 An illustration of an application example of augmented reality in a portable terminal.

[0038] Fig.16 It means that Fig.15 An illustration of an example of a virtual object being displayed on a display unit of a portable terminal using augmented reality.

[0039] Fig.17 This is an explanatory diagram showing an application example of a portable terminal according to the second embodiment.

[0040] Fig.18 This is an explanatory diagram showing an application example of a portable terminal according to the third embodiment.

[0041] Fig.19 This is an explanatory diagram showing an example of the structure of a portable terminal according to the fourth embodiment.

[0042] Fig. 20 This is an explanatory diagram showing an application example of a portable terminal according to the fifth embodiment.

[0043] Fig.21 Yes means Fig. 20 A block diagram of an example of the structure of a portable terminal.

[0044] Fig. 22 It means the user is wearing Fig. 20 An illustration of an example of a menu display when using a portable terminal.

[0045] Fig.23 Yes means Fig. 20 An illustration of an application example of a portable terminal.

[0046] Fig.24 Yes means Fig.23 An illustration of other application examples of the portable terminal.

[0047] Fig.25 This is an explanatory diagram showing an application example of the imaging device according to the sixth embodiment.

[0048] Fig.26 Yes means Fig.25 An illustration of other application examples of the imaging device. DETAILED DESCRIPTION

[0049] In all the drawings for explaining the embodiment, the same components are denoted by the same reference numerals in principle, and repeated description thereof is omitted.

[0050] The following describes the implementation in detail.

[0051] (Implementation Method 1)

[0052] <Pixel Structure Example of Image Pickup Element>

[0053] Figure 1 This is an explanatory diagram showing an example of the structure of the image pickup element according to the first embodiment.

[0054] The image sensor 10 includes visible light pixels 11 and pixels 14. The visible light pixels 11 and pixels 14 constitute a sensor unit.

[0055] The visible light pixel 11 has a color filter that is sensitive in the visible light region of red (R), green (G), and blue (B). The non-visible light pixel 14 is a pixel that receives near-infrared light close to the non-visible light region. In addition, the pixels recorded as "R", "G", and "B" in the figure are visible light pixels 11, and the pixels recorded as "IR" are non-visible light pixels 14. The same description is also made for the following figure representing the pixels of the imaging element.

[0056] The visible light pixels 11 and the non-visible light pixels 14 are arranged in a grid pattern, which is the so-called Bayer arrangement. The Bayer arrangement is a grid arrangement in which R pixels, G pixels, G pixels, and B pixels are arranged as one group. Figure 1 In the case of the imaging element 10 shown in FIG. 1 , a non-visible light pixel 14 is arranged in place of one G pixel among the two G pixels constituting the Bayer array. Figure 1 R pixels, G pixels, B pixels, and non-visible light pixels 14 are arranged clockwise from the upper left corner of .

[0057] Thus, a group is formed by the visible light pixel 11 of R pixel, G pixel, and B pixel and one non-visible light pixel 14. The visible light pixel 11 of R pixel, G pixel, and B pixel in one group is a pixel for capturing a color image. The non-visible light pixel 14 is a pixel for capturing a distance image using the principle of a TOF (Time-Of-Flight) sensor.

[0058] The visible light pixels 11 having sensitivity in the visible light region read out the charges obtained by the photoelectric conversion independently. On the other hand, the non-visible light pixels 14 are configured to read out the charges of a predetermined number of, for example, four, non-visible light pixels 14 mixed together.

[0059] In order to capture visible light of natural and indoor lighting, color images are generally produced with sufficient sensitivity. On the other hand, in order to obtain a distance image, the TOF sensor irradiates infrared light required for distance measurement, and performs photoelectric conversion on the reflected light to obtain a distance measurement image. Therefore, the amount of light is limited compared to visible light.

[0060] Thus, it is sufficient to improve the sensitivity of non-visible light pixels. However, in an imaging element in which pixels for photoelectric conversion are uniformly formed on the imaging element, it is difficult to improve the sensitivity of only pixels for IR, that is, non-visible light pixels.

[0061] Therefore, as mentioned above Figure 1 As shown in the structure of FIG. 1 , if only the non-visible light pixel 14 is mixed with multiple pixels and read, the amount of light received by the mixed non-visible light pixel is about 4 times, and as a result, the signal amount for near-infrared light (IR) can be increased. As a result, the signal-to-noise ratio (SN ratio) for the distance image can be improved, so a good distance image can be obtained.

[0062] In addition, since the charges of the R pixels, G pixels, and B pixels are read independently, a high-resolution color image can be obtained. Furthermore, the color image and the distance image can be acquired at the same frame time, that is, the frame rate.

[0063] in addition, Figure 1 , an example of mixing and reading four non-visible light pixels 14 is described as above, but the number of mixed non-visible light pixels 14 is not limited to this. For example, when the number of pixels of the imaging element 10 is 16 megapixels, the number of mixed non-visible light pixels 14 can also be about 16 pixels.

[0064] <Configuration Example of Image Pickup Element>

[0065] Next, the image sensor 10 will be described in detail.

[0066] Figure 2 Yes means Figure 1 A block diagram showing an example of the structure of an imaging element.

[0067] The imaging element 10 is Figure 2 The device shown in FIG. 1 includes a visible light pixel 11, a non-visible light pixel 14, a visible light pixel driving unit 210, a non-visible light pixel driving unit 220, and an output interface 230. The visible light pixel 11 is Figure 1 The R pixel, G pixel, and B pixel shown are pixels that are sensitive to red (R), green (G), and blue (B), respectively. Figure 1 The pixels shown accept near infrared light which is close to the non-visible light region.

[0068] The visible light pixel driving unit 210 drives the visible light pixels 11 of R, G, and B pixels of the imaging element 10. The non-visible light pixel driving unit 220 drives the non-visible light pixels 14 of the imaging element 10. The non-visible light pixel driving unit 220 outputs an IR light emission control signal for capturing a distance image as a TOF sensor.

[0069] The output interface 230 is an interface for outputting the outputs of the visible light pixel driving unit 210 and the non-visible light pixel driving unit 220 as image signals of the imaging element 10 .

[0070] <Circuit Example of Visible Light Pixel Driving Unit and Non-Visible Light Pixel Driving Unit>

[0071] Next, the driving and reading operations of the visible light pixel 11 will be described.

[0072] Figure 3 Yes means Figure 2 An explanatory diagram of an example of an equivalent circuit diagram of a visible light pixel driving unit and a non-visible light pixel driving unit of an imaging element.

[0073] Among them, Figure 3 In the figure, for simplicity, the visible light pixel driving unit 210 only shows the structure of the equivalent circuit corresponding to the three visible light pixels 11 , and the non-visible light pixel driving unit 220 only shows the structure of the equivalent circuit corresponding to the four non-visible light pixels 14 .

[0074] The visible light pixel driving section 210 includes switches 321 to 323 , switches 341 to 343 , capacitors 331 to 333 , an amplifier 351 , an image data generating section 353 , and a visible light timing generating section 352 .

[0075] In the image sensor 10 , the visible light pixel 11 is formed of, for example, a photodiode. The cathode of the photodiode is connected to one connection portion of each of the switches 321 to 323 .

[0076] One connection portion of the capacitors 331 to 333 and one connection portion of the switches 341 to 343 are connected to the other connection portions of the switches 321 to 323. The other connection portions of the capacitors 331 to 333 are connected to a reference potential.

[0077] The other connection parts of the switches 341 to 343 are commonly connected to the input part of the amplifier 351, and the output part of the amplifier 351 is connected to the input part of the image data generating part 353. The signal output from the output part of the image data generating part 353 becomes image data.

[0078] The switches 321 to 323 are switches for acquiring charges obtained by photoelectric conversion in each of the R pixel, the G pixel, and the B pixel visible light pixels 11. The capacitors 331 to 333 convert the acquired charges into voltages.

[0079] The switches 341 to 343 sequentially read out the charges respectively stored in the capacitors 331 to 333. The amplifier 351 is a reading amplifier that amplifies the voltage converted by the capacitors 331 to 333. The image data generating unit 353 converts the pixel output read out by the amplifier 351 into, for example, digital data and outputs the digital data.

[0080] The visible light timing generation unit 352 generates control signals for controlling on / off driving of the switches 321 to 323 , the switches 341 to 343 , etc. Thus, reading of each visible light pixel 11 is performed at a timing described later.

[0081] Furthermore, the visible light timing generation unit 352 is connected so as to input setting parameters as control commands input from the outside, and can set timing parameters such as exposure time based on the setting parameters.

[0082] The timing parameters set by the setting parameters are stored in a register (not shown) or the like included in the visible light timing generation unit 352. The setting parameters are output from, for example, a microcomputer or the like that controls the imaging element 10.

[0083] The non-visible light pixel driving unit 220 includes switches 361 to 364, capacitors 371 to 374, amplifiers 381 to 384, a distance data calculation generating unit 386, and a non-visible light timing generating unit 382. The switches 361 to 364 constitute a first switch unit. The capacitors 371 to 374 constitute a first capacitor unit. The amplifiers 381 to 384 constitute an amplifying unit.

[0084] The non-visible light pixel 14 is also constituted by a photodiode similarly to the visible light pixel 11. The cathodes of the four photodiodes serving as the non-visible light pixel 14 are connected in common and connected to one connection portion of the switches 361 to 364 which are also connected in common.

[0085] The other connection parts of the switches 361 to 364 are connected to the connection parts of one side of the capacitors 371 to 374 and the input parts of the amplifiers 381 to 384. The output parts of the amplifiers 381 to 384 are connected to the input part of the distance data calculation generating unit 386, and the distance image including the distance data is output from the output part of the distance data calculation generating unit 386.

[0086] Here, an example Figure 1 The case where the charges of four non-visible light pixels 14 are mixed and read is described in the above, but the non-visible light pixels 14 that are mixed and read are not limited to 4. For example, when the charges of eight non-visible light pixels 14 are mixed and read, the cathodes of eight photodiodes serving as the non-visible light pixels 14 and eight switches that are connected together become a structure that is connected together.

[0087] The remaining electrostatic capacitors and amplifiers are also configured to have the same number as the number of visible light pixels 11 to be mixed and read. Thus, eight electrostatic capacitors and eight amplifiers are provided, and the outputs of the eight amplifiers are connected to the inputs of the distance data calculation and generation unit 386, respectively.

[0088] The switches 361 to 364 are switches for acquiring charges obtained by photoelectric conversion in the commonly connected non-visible light pixels 14. The capacitors 371 to 374 convert the acquired charges into voltages.

[0089] in addition, Figure 5 In the figure, the switches 361 to 364 are provided in the non-visible light pixel driving unit 220 , but the switches 361 to 364 may be provided in the imaging element 10 .

[0090] The amplifiers 381 to 384 amplify the voltage converted by the capacitor elements 371 to 374. The distance data calculation generating unit 386 as a distance image calculation unit converts the pixel output read from the amplifiers 381 to 384 into digital data, outputs the digital data, performs TOF calculation, and outputs a distance image obtained by calculating the pixel data of the distance image.

[0091] The non-visible light timing generation unit 385, which serves as a timing control unit, generates a control signal for turning on / off the switches 361 to 364, etc., which perform reading of the non-visible light pixels 14. The non-visible light timing generation unit 385 is connected so that the frame time output by the visible light timing generation unit 352 and the setting parameters are respectively input. The frame time becomes the frame rate of the color image read from the imaging element 10.

[0092] <Reading operation example>

[0093] Figure 4 Yes means Figure 3 A timing diagram showing an example of the timing for reading charges of the visible light pixel driving unit and the non-visible light pixel driving unit.

[0094] from Figure 4 From top to bottom, the signal timings of frame time, R pixel exposure, B pixel exposure, B pixel exposure, R pixel reading, B pixel reading, B pixel reading, non-visible light pixel exposure and non-visible light pixel mixed reading are respectively represented.

[0095] The frame time is a signal indicating the timing of exposure acquisition of a color image. The R pixel exposure is the exposure timing of the visible light pixel 11 as the R pixel, indicating the period during which the visible light pixel 11 acquires light. The G pixel exposure is the exposure timing of the visible light pixel 11 as the G pixel. The B pixel exposure is the exposure timing of the visible light pixel 11 as the B pixel. Figure 4The Hi signal period of R pixel exposure, G pixel exposure, and B pixel exposure becomes the second exposure period.

[0096] The R pixel readout is a readout timing of the visible light pixel 11 as the R pixel. The G pixel readout is a readout timing of the visible light pixel 11 as the G pixel. The B pixel readout is a readout timing of the visible light pixel 11 as the B pixel.

[0097] The non-visible light pixel exposure is a timing of exposure of the four non-visible light pixels 14, and the non-visible light pixel mixed reading is a timing of reading the charges of the four non-visible light pixels 14. The repetition period of the Hi signal and the Lo signal in the non-visible light pixel exposure is the first exposure period.

[0098] First, the reading process performed by the visible light pixel driving unit 210 will be described.

[0099] Each visible light pixel 11 of R, G, and B is exposed and read based on the frame time. Each visible light pixel 11 of R, G, and B is exposed at the timing of the Hi signal of R, G, and B pixel exposure.

[0100] When the signals indicating the exposure of the R pixel, the G pixel, and the B pixel are changed from the Hi signal to the Lo signal, the control signals for turning on the switches 321 to 323 are output from the visible light timing generator 352, and the switches 321 to 323 are turned on. As a result, the charges of the respective visible light pixels 11 are transferred to the capacitors 331 to 333.

[0101] During the Hi signal period of the R pixel reading, the switch 341 is turned on, and the charge of the capacitor element 331 is output to the amplifier 351. During the Hi signal period of the G pixel reading, the switch 342 is turned on, and the charge of the capacitor element 332 is output to the amplifier 351. During the Hi signal period of the B pixel reading, the switch 343 is turned on, and the charge of the capacitor element 333 is output to the amplifier 351.

[0102] A control signal for turning on the switches 341 to 343 is output from the visible light timing generator 352. As a result, the charges of the capacitors 331 to 333 are sequentially output to the amplifier 351, and the charge reading process is completed.

[0103] Next, the reading process of the non-visible light pixel driving unit 220 will be described.

[0104] 4 non-visible light pixels 14 such as Figure 4 As shown in the non-visible light pixel exposure, the non-visible light pixel exposure process is repeatedly performed during the Hi signal of the frame time.

[0105] Figure 4The signal timing shown below the mixed reading of the non-visible light pixels represents the timing of the exposure processing of the above-mentioned non-visible light pixels. From top to bottom, it represents the signal timing of near-infrared light luminescence, reflected light, exposure a, exposure b, exposure c and exposure d respectively.

[0106] Near infrared light emission is described later Fig.13 The timing of the emission of near-infrared light emitted by the IR light source unit 1103 of the imaging device. The reflected light is the timing of the light reflected from the subject. Exposure a, exposure b, exposure c and exposure d are the exposure timings of the four non-visible light pixels 14 that are mixed and read.

[0107] First, the invisible light timing generator 385 generates a light emission timing signal and outputs it to the IR light source 1103. The light emission timing signal is a signal for the IR light source 1103 to emit near infrared light. Based on the light emission timing signal such as the Hi signal, the IR light source 1103 emits near infrared light.

[0108] When the IR light source unit 1103 as the light source unit emits near infrared light, exposure is performed sequentially from exposure a to exposure d as shown in the figure. Therefore, the invisible light timing generation unit 385 outputs a control signal to turn on switches 361 to 364 sequentially at the end of each exposure a to exposure d.

[0109] For example, in exposure a, when the exposure of the non-visible light pixel 14 corresponding to the exposure a is completed, the switch 361 is turned on. Next, in exposure b, when the exposure of the non-visible light pixel 14 corresponding to the exposure b is completed, the switch 362 is turned on.

[0110] The reflected near infrared light emitted by the IR light source unit 1103 reaches the image sensor 10 with a delay of time t1 according to the distance to the subject. The near infrared light arriving with a delay of time t1 is charged in the capacitors 371 to 374 at the exposure timing of the Hi signal from exposure a to exposure d.

[0111] exist Figure 4 In the case shown, in the exposure sequence of exposure a, the reflected light during the time t2 indicated by the hatched line is photoelectrically converted, and the charge Qa after the photoelectric conversion is stored in the capacitor element 371. In the exposure sequence of exposure b, the reflected light during the time t3 indicated by the hatched line is photoelectrically converted, and the charge Qa after the photoelectric conversion is stored in the capacitor element 372.

[0112] Therefore, the reflected light delay time t1 of the near-infrared light corresponds to the distance from the subject, and the amount of charge stored by photoelectric conversion during the Hi signal period of exposure a and the Hi signal period of exposure b changes. Therefore, the distance to the subject can be calculated by the distance data calculation generation unit 386 based on the charge amounts of charges Qa and Qb.

[0113] Alternatively, the distance can be obtained from the ratio of charge Qa to charge Qb. When the distance is obtained from the ratio of charge Qa to charge Qb, the influence of the difference in reflectivity of the subject and the near-infrared light from interference other than the near-infrared light source can be eliminated.

[0114] When the distance from the subject is longer for exposure c and exposure d, in other words, when the reflection time of near-infrared light is longer, distance measurement can also be performed based on the output of the non-visible light pixel 14, thereby ensuring distance measurement accuracy and extending the distance measurement range.

[0115] In the first embodiment, an example of four phases of exposure a to exposure d is shown, but the number of detection phases is not limited thereto.

[0116] The mixed reading of the charges of the non-visible light pixels 14 repeatedly accumulated during the non-visible light pixel exposure between frames is performed at the end of the frame time at the timing of the Hi signal of the non-visible light mixed pixel reading. The non-visible light mixed pixel reading is a process of calculating the distance to the subject calculated by the distance data calculation and generation unit 386 and outputting a distance image.

[0117] In addition, if the timing of reading the IR mixed pixel is not set to one frame but several frames are read once, the amount of light stored in the non-visible light pixel 14 can be further increased, and the SN ratio of the range image can be further improved.

[0118] As described above, by mixing and reading the charges of the four non-visible light pixels 14 , the amount of light received by the mixed non-visible light pixels 14 becomes approximately a multiple of the number of mixed pixels, resulting in an increase in the amount of near-infrared light signals.

[0119] Thus, since the signal-to-noise ratio (SN ratio) for the distance image can be improved, a good distance image can be obtained. In addition, the charges of each visible light pixel 11 of the R pixel, the G pixel, and the B pixel can be read separately, so a high-resolution color image can be obtained. Furthermore, the color image and the distance image can be obtained in the same frame time.

[0120] When the pixel pitch of the imaging element is made smaller to increase the resolution, the number of non-visible light pixels 14 can be increased. As a result, the number of non-visible light pixels 14 that read charges in a mixed manner can be increased, thereby increasing the signal amount for near-infrared light.

[0121] As described above, it is possible to provide the image sensor 10 capable of acquiring a high-resolution color image and a high-SN ratio range image.

[0122] <Circuit Example of Non-Visible Light Pixel Driving Section>

[0123] Figure 5 Yes means Figure 3 An illustration of another example of an equivalent circuit diagram of a non-visible light pixel driving unit.

[0124] Figure 5 The non-visible light pixel driving unit 220 and Figure 3 The non-visible light pixel driving unit 220 is different in that switches 391 to 394 are further provided as second switch units. These switches 391 to 394 are controlled to be turned on / off based on the control signal output from the non-visible light timing generating unit 385. The switches 391 to 394 are read switches for reading the charge of the non-visible light pixel 14 of the mixed charge.

[0125] The cathodes of the photodiodes constituting the non-visible light pixels 14 are connected to the connection parts of the switches 391 to 394. The other connection parts of the switches 391 to 394 are connected to the connection parts of the switches 361 to 364. Figure 3 Therefore, the description is omitted.

[0126] When the charges of the four non-visible light pixels 14 shown in the figure are mixed and read, for example, the non-visible light timing generation unit 385 outputs a control signal so that all the switches 361 to 364 are turned on.

[0127] In addition, the non-visible light timing generation unit 385 can change the conduction timing of the switches 391 to 394 connected to the photodiodes as the non-visible light pixels 14 based on the setting parameters as setting information input from the outside. Since the number of switches to be turned on can be arbitrarily changed in this way, the number of non-visible light pixels 14 for mixing charges can be easily changed.

[0128] This allows the number of mixed pixels to be easily changed in a short time according to the signal amount of the non-visible light pixels 14 required for generating a range image, and thus allows the generation of a range image with a good SN ratio.

[0129] <Reading sequence example>

[0130] Figure 6 Yes means Figure 4 A timing diagram showing another example of the read timing.

[0131] Figure 4 An example of reading during the frame time of one frame is shown in FIG. Figure 6 An example of reading during a frame time period of three frames is shown.

[0132] from Figure 6 From top to bottom, the signal timings of frame time, R pixel exposure, G pixel exposure, B pixel exposure, R pixel reading, B pixel reading, B pixel reading, non-visible light pixel exposure and non-visible light pixel mixed reading are respectively represented.

[0133] exist Figure 4 In the example shown, the non-visible light pixel exposure process in the near-infrared light exposure is repeated during one frame, but Figure 6 In the example shown, the non-visible light pixel exposure process is repeated over a period of three frames. Figure 6 The charge is read during the Hi signal of the non-visible light pixel mixed reading.

[0134] Thus, the mixed charge is read after the non-visible light pixel exposure process is continuously performed during the three-frame period of the frame time of the visible light pixel 11. By making the charge reading of the non-visible light pixel 14 1 / 3 of the frame rate, a visible light image and a distance image corresponding to the photographic environment can be obtained.

[0135] In this Figure 6 In the example, an example is shown in which the charge of the non-visible light pixel 14 is read at 1 / 3 of the frame rate, but by setting it to 1 / N (N is an integer) of the frame rate, a good visible light image and distance image corresponding to the photographic environment can be obtained.

[0136] The setting of 1 / N of the read timing is set by, for example, a setting parameter input from the outside. The non-visible light timing generator 385 controls the on / off timing of the switches 361 to 364 based on the input setting parameter so that the read timing is 1 / N of the frame rate.

[0137] As described above, the mixing amount of the charge of the non-visible light pixel 14 can be easily made variable, and the reading timing can be easily made, for example, 1 / N (N is an integer) with respect to the frame rate of the visible light pixel 11 .

[0138] Therefore, when it is desired to maintain the resolution of the distance image and improve the SN ratio, it is desired to make the reading timing of the non-visible light pixel 14 1 / N of the frame rate and make the reading timing the same as the frame time of the visible light pixel 11. In this way, the number of mixed non-visible light pixels 14 can be increased to improve the SN ratio.

[0139] <About Color Filters>

[0140] Figure 7 Yes means Figure 1 An explanatory diagram of an example of the light bandpass characteristics of the color filter included in the visible light pixel of the imaging element.

[0141] exist Figure 7In the figure, from top to bottom, the characteristics of the blue band color filter used in the visible light pixel of the B pixel, the green band color filter used in the visible light pixel of the G pixel, the red band color filter used in the visible light pixel of the R pixel, and the near-infrared band color filter used in the non-visible light pixel 14 are respectively represented.

[0142] also, Figure 8 The following is a structural example of a color filter used in an image pickup element for capturing a color image other than a distance image, which was studied by the inventors of the present invention.

[0143] In the imaging element 603 used for capturing color images that do not capture distance images, an unillustrated color filter using a frequency band of green and near-infrared light (B+IR) is used in the B pixel, an unillustrated color filter using a frequency band of green and near-infrared light (G+IR) is used in the G pixel, and an unillustrated color filter using a frequency band of red and near-infrared light (B+IR) is used in the R pixel.

[0144] In addition, since the near-infrared light component is unnecessary, an IR cut filter 602 that allows only the visible light band to pass and cuts off the near-infrared light component is formed on the optical axis between the microlens 601 and the imaging element 603, so as to remove the near-infrared light that is unnecessary in the color signal generation. The microlens is formed on each pixel of the imaging element 603 and is a lens that captures light at each pixel.

[0145] on the other hand, Figure 1 In the case of the imaging element 10 shown in FIG. Figure 7 For example, in the case of a visible light pixel 11 as a B pixel, a filter having a bandpass characteristic is used. Figure 7 The color filter shown above captures the blue band.

[0146] Similarly, in the case of the visible light pixel 11 as a G pixel, the acquisition Figure 7 In the case of the visible light pixel 11 as the R pixel, the green band color filter is used to obtain Figure 7 This effectively removes near-infrared light components that are not necessary for color image generation.

[0147] In the case of non-visible light pixels 14, a color filter that captures near-infrared wavelengths is used to capture near-infrared light components required for range images. By using such a color filter, a good color image and range image can be captured substantially simultaneously.

[0148] <Pixel configuration example>

[0149] Fig. 9 Yes means Figure 1 An illustration of another example of the pixel arrangement of an imaging element.

[0150] Fig. 9 The pixel arrangement of the imaging element 10 is for Figure 1 The imaging element 10 shown in the figure has an arrangement in which G pixels and IR pixels are replaced. Fig. 9 Starting from the upper left corner of , the visible light pixel 11 of R pixel, the non-visible light pixel 14, the visible light pixel 11 of B pixel, and the visible light pixel 11 of G pixel are arranged in sequence clockwise.

[0151] Fig.10 Yes means Fig. 9 An illustration of another example of the pixel arrangement of an imaging element.

[0152] Fig. 9 In FIG. 1 , among the pixels arranged in a grid pattern, an example of arranging non-visible light pixels 14 every other pixel is shown, but in Fig.10 (a) shows an example in which the non-visible light pixels 14 are arranged in a grid pattern and are arranged every three pixels in both the row direction and the column direction.

[0153] also, Fig.10 (a) means in Figure 1 In the illustrated pixel arrangement, the non-visible light pixels 14 are arranged in a grid pattern, and are arranged every three pixels in both the row direction and the column direction.

[0154] according to Fig.10 (a) and Fig.10 The pixel arrangement shown in (b) can increase the number of visible light pixels 11, thereby improving the resolution of color images and visible light sensitivity.

[0155] Fig.11 Yes means Fig.10 An illustration of another example of the pixel arrangement of an imaging element.

[0156] Fig.11 (a)~ Fig.11 (d) shows an example in which two non-visible light pixels 14 are arranged adjacent to each other in the row direction and two non-visible light pixels are arranged adjacent to each other in the column direction. By arranging the non-visible light pixels 14 adjacent to each other in the row direction and the column direction in this way, the wiring of the non-visible light pixels 14 and the visible light pixels 11 of the imaging element 10 can be easily formed.

[0157] Fig.12 Yes means Figure 1 An illustration of another example of the pixel arrangement of an imaging element.

[0158] In this Fig.12The arrangement of the imaging element 10 shown in the figure shows an example in which the non-visible light pixels 14 are arranged every other pixel in the oblique direction.

[0159] In the case of an image sensor that only captures color images, the photodiode has the highest sensitivity in the green wavelength band. In the case of an image sensor that has a sensitivity peak on the longer wavelength side, by using Fig.12 The pixel configuration shown is similar to that of Figure 1 , Fig. 9 Compared with the arrangement of the non-visible light pixels 14 shown in the figure, the density of the R pixels, the G pixels, and the B pixels can be increased. As a result, the visible light sensitivity can be improved.

[0160] Above, in Figure 1 , Figures 9 to 12 2 shows an example of arrangement of various pixels, but the arrangement of the visible light pixels 11 and the non-visible light pixels 14 is not limited to this.

[0161] <Configuration Example of Imaging Device>

[0162] Fig.13 It means use Figure 2 An explanatory diagram of an example of an imaging device having an imaging element.

[0163] The imaging device 1100 includes an optical lens unit 1101, an imaging element 10, an IR light source unit 1103, and a camera control unit 1104. The optical lens unit 1101 forms an image of a subject or the like.

[0164] The imaging element 10 and Figure 2 Similarly, a color image and a distance image are obtained. The IR light source unit 1103 irradiates near infrared light. The camera control unit 1104 controls the exposure time of the imaging element 10 and the white balance of the color image. Figure 4 The near-infrared light generated in the imaging element 10 shown is emitted by controlling the emission timing.

[0165] like Fig.13 As shown, by using Figure 2 The imaging device 1100 is constituted by the imaging element 10, and can obtain a range image with a good S / N ratio and a color image with a high resolution. Furthermore, it is possible to provide the imaging device 1100 that can obtain a color image and a range image in the same frame time.

[0166] in addition, Fig.13 In the embodiment, the camera control unit 1104 controls the exposure and white balance separately, but a structure in which the exposure and white balance are controlled outside the imaging device 1100, for example, based on an image signal output by the imaging element 10 may be adopted.

[0167] As a method for simultaneously acquiring a range image and a color image, a widely used method is to separately mount a color image sensor and a range image sensor. However, when acquiring a range image and a color image simultaneously using different image sensors, the viewing angles (angles of view) of the color image sensor and the range image sensor must be matched, which increases the adjustment cost.

[0168] In this regard, Fig.13 In the imaging device 1100 having the structure shown, only one imaging element 10 is used, so that cost advantages due to the number of imaging elements and reduction in adjustment costs can be achieved.

[0169] <Configuration Example of Portable Terminal>

[0170] Fig.14 It means use Fig.13 An illustration of an example of a portable terminal having an imaging device.

[0171] The portable terminal 1200 includes, for example, a tablet computer, a smart phone, etc. Fig.14 The device is shown to have a communication interface 1213, a control unit 1214, a display unit 1217, a button 1215, a microphone 1261, a speaker 1262, a signal separation unit 1221, a modulation / demodulation unit 1222, a storage 1225, a mobile communication interface 1231, a memory 1232, an acceleration sensor unit 1233, a geomagnetic sensor unit 1234, a GPS receiving unit 1235, a gyro sensor unit 1236, a camera 1100, a camera 1100a, a switch input unit 1250, and a sound input / output unit 1260. Among them, each functional unit except the button 1215, the microphone 1261, and the speaker 1262 is connected via a bus 1201.

[0172] The storage 1225 stores application programs, and the control unit 1214 expands the application programs from the storage 1225 into the memory 1232 . The control unit 1214 executes the expanded application programs, thereby realizing various functions.

[0173] In the following description, for simplicity of description, various functions realized by the control unit 1214 executing each application program will be described as being realized mainly by various program function units.

[0174] In addition, the application can be pre-stored in the storage 1225 before the portable terminal 1200 leaves the factory, or it can be stored in an optical medium such as a CD (Compact Disk), DVD (Digital Versatile Disk), a semiconductor memory, or other medium, and installed in the portable terminal 1200 via a medium connection unit not shown.

[0175] Alternatively, the software may be downloaded and installed from an external network (not shown) via the communication interface 1213 and a wireless router (not shown), or downloaded and installed from a distribution source via a base station (not shown) using the mobile communication interface 1231.

[0176] Furthermore, the application program can be connected to a personal computer that has acquired the application program via a network via an external device connection interface (not shown), and can be moved or copied from the personal computer to the portable terminal 1200 and installed.

[0177] Furthermore, the application program can also be realized by hardware as a processing unit having the same function. When realized by hardware, each processing unit mainly realizes each function.

[0178] The communication interface 1213 is connected to a wireless router (not shown) using a wireless LAN (Local Area Network) etc. The communication interface 1213 is also connected to an external network via a wireless router to transmit and receive information with a server etc. on the external network.

[0179] In addition to or instead of the communication function with a wireless router, it is possible to directly communicate with a server without going through a wireless router by using a technology such as a wireless LAN such as Wi-Fi (registered trademark).

[0180] The communication interface 1213 may be equipped with chips for different communication methods. Alternatively, a single chip for processing multiple communication methods may be installed. The communication method called BLUETOOTH (registered trademark) for short-distance communication may also be used to communicate with other devices.

[0181] The mobile communication interface 1231 can utilize the third-generation mobile communication system, or LTE (Long Term Evolution), the next-generation 5G communication and other mobile communication networks, connect to the communication network via a base station, send and receive information with the server on the communication network, and can also share each other's locations between terminals.

[0182] The third generation mobile communication system (hereinafter referred to as "3G") includes GSM (registered trademark) (Global System for Mobile Communications) method, W-CDMA (Wideband Code Division Multiple Access) method, CDMA2000 method, UMTS (Universal Mobile Telecommunications System) method, etc.

[0183] Furthermore, it is possible to give priority to connection to an external network via the communication interface 1213 over connection to a communication network via the mobile communication interface 1231 .

[0184] The control unit 1214 receives a user operation request from the button 1215 via the switch input unit 1250 or from the microphone 1261 via the sound input and output unit 1260 , and controls the signal separation unit 1221 , the display unit 1217 , the communication interface 1213 , and various program function units.

[0185] Furthermore, the control unit 1214 also has a function of acquiring various information from an external network via the communication interface 1213 and a wireless router, or from a server on an external network via the mobile communication interface 1231 and a base station, and transmitting the information to various program function units.

[0186] The storage 1225 can be controlled by an instruction of the control unit 1214 to store the application program and store various information generated by the application program.

[0187] The content such as video and audio streams may be stored based on the signal received from the modulation / demodulation unit 1222, the communication I / F 1213 or the mobile communication I / F 1231. The storage 1225 may be built in the portable terminal 1200 or may be a removable storage device that can be attached to and detached from the portable terminal 1200.

[0188] The memory 1232 is controlled by instructions from the control unit 1214. The functional unit of the application program stored in the storage 1225 is expanded in the memory 1232 by the control unit 1214.

[0189] The display unit 1217 displays images and videos stored in the storage 1225 , broadcast / distributed videos, a UI for performing various operations, and the like. The images and videos to be displayed may be images generated by an application.

[0190] Alternatively, the images or videos may be content received via the modulation / demodulation unit 1222 , or may be images or videos received from a server on an external network via the communication interface 1213 .

[0191] Alternatively, the images or videos may be received from a television receiver via the communication interface 1213, or may be distributed from a server on a communication network via the mobile communication interface 1231. The display unit 1217 may be integrally formed with a touch panel or the like described later, for example.

[0192] The button 1251 is an input unit that receives an operation on the portable terminal 1200 from the user and inputs control information related to the input operation, and can use a touch panel or the like, for example.

[0193] Below, in Fig.14 In the example shown, an example using a touch panel is described, but a structure using physical buttons for various operations may also be adopted. By using a touch panel, an object or the like can be freely moved by a drag operation or a quick drag operation.

[0194] The drag operation is an operation of moving an arbitrary object such as an icon on the touch panel while touching it with a finger. The quick drag operation is an operation of moving the finger on the screen in a flicking manner.

[0195] When a click operation or a double-click operation is performed, an object such as an icon can be activated or another screen can be switched. A click operation is an operation of tapping an object once with a finger. A double-click operation is an operation of tapping an object twice with a finger. Here, each operation of the touch panel is referred to as a drag operation, a quick drag operation, and a click operation.

[0196] The modulation / demodulation unit 1222 and the signal separation unit 1221 perform modulation / demodulation and signal separation of the television receiver, respectively. The acceleration sensor unit 1233 measures acceleration applied to the portable terminal 1200 .

[0197] The control unit 1214 can know which part of the portable terminal 1200 is at the top by measuring the gravitational acceleration using the acceleration sensor unit 1233, for example, and displays the top of the screen displayed on the display unit 1217 in a manner that matches the top measured by the acceleration sensor unit 1233. In this way, a screen that matches the way the user holds the portable terminal 1200 can be displayed.

[0198] The geomagnetic sensor unit 1234 measures geomagnetism using a plurality of magnetic sensors, etc. The GPS receiver unit 1235 receives signals transmitted from a plurality of satellites using GPS (Global Positioning System). The control unit 1214 can calculate the position information of the portable terminal 1200 based on the signal received by the GPS receiver unit 1235 .

[0199] The gyro sensor unit 1236 measures the angular velocity of the portable terminal 1200 generated when the user moves the portable terminal 1200. The camera 1100 and the camera 1100a both include Fig.13 The camera device 1100 is shown.

[0200] For example, the camera 1100 is used as a so-called external camera for photographing frontal scenery or people, and the camera 1100a is used as an internal camera mainly for video calls and self-portraits.

[0201] The image captured by the camera device 1100 is recorded in the memory 1225. The recording of the image is controlled by the control unit 1214 according to the camera control program recorded in the memory 1225 by the control unit 1214.

[0202] Furthermore, the control unit 1214 controls the display unit 1217 to display an image acquired by the camera device 1100 or the like, and three-dimensional data of a virtual object stored in the storage 1225 or the like in a superimposed manner.

[0203] The audio input / output unit 1260 is used to input and output audio input signals from a microphone 1261 provided in the portable terminal 1200 and audio output signals to a speaker 1262 , and the volume of the audio input / output is controlled by the control unit 1214 .

[0204] The switch input unit 1250 obtains switch information according to the operation of the physical button 1251, obtains the switch information to the control unit 1214 through the bus 1201, and uses it in the control of various application programs as needed.

[0205] As an example, the volume of the sound output is adjusted by pressing two buttons of the button 1251, that is, it is used to control the increase and decrease of the volume. In addition, the number of the button 1251 may be one or more.

[0206] Next, the use of the above Fig.14 Application example of portable terminal.

[0207] <Application Examples>

[0208] Fig.15 Yes means Fig.14 An illustration of an application example of augmented reality performed on a portable terminal.

[0209] In recent years, augmented reality (hereinafter referred to as AR), which uses color images and distance images to overlay and display virtual objects that do not actually exist on a captured screen, has attracted attention as an application example of portable terminals.

[0210] Fig.15 The user 1301 holding the portable terminal 1200 is shown taking a picture of the table 1304. At this time, the picture is taken by the camera 1100 installed as an external camera provided on the back side facing the display unit 1217.

[0211] Fig.15 The virtual object 1303 indicated by the dotted line is a virtual object whose size and distance to the portable terminal 1200 are defined in the portable terminal 1200 and does not actually exist. The user 1301 indicates a state where a palm 1305 is placed in front of the virtual object 1303 .

[0212] <Display example>

[0213] Fig.16 It means that Fig.15 An illustration of an example of a virtual object being displayed on a display unit of a portable terminal using augmented reality.

[0214] Fig.16 (a) and Fig.16 (b) both show an example of image synthesis display as an augmented reality (AR) application. Fig.16 In (a), on the display unit 1217 of the portable terminal 1200, in addition to a table 1304 obtained as a color image and a hand 1305 extended by the user, a virtual object 1303 at the same distance from the table is displayed.

[0215] Fig.16 (b) shows a display example when the palm 1305 is located in the back direction of the screen. As shown in the figure, when the distance between the virtual object 1303 and the portable terminal 1200 is closer, the palm 1305 is displayed in a manner of being located in the back of the virtual object 1303.

[0216] A distance image is obtained in which, for example, a location with a large pixel value indicates a close distance and a location with a small pixel value indicates a far distance in each pixel position obtained from the distance image. Therefore, the virtual object is displayed in an overlapping manner such that the palm 1303 is located at a pixel position farther than the defined distance of the virtual object.

[0217] Furthermore, a color image can be displayed at a pixel position where the palm 1303 is located closer than the defined distance of the virtual object, and thus an application that the user 1301 likes can be provided.

[0218] In such applications, color images and distance images can be captured simultaneously for each frame, so when the palm 1305 moves forward and backward, the virtual object 1303 can be smoothly displayed in an overlapping manner. Furthermore, color images can be displayed with a large number of pixels and high resolution.

[0219] In addition, when three-dimensional image capture data is generated based on the color image using the distance image, and when the edges of the color image are synthesized with the distance image to generate high-precision 3D (Dimensions) image data, an image with no time deviation between the distance image and the color image can be used.

[0220] When other factors are controlled by gesture input, there is no time difference between the distance image and the color image, so the frame rate can be increased. As a result, the responsiveness of gesture input can be improved.

[0221] (Implementation Method 2)

[0222] <Application Example 2>

[0223] Fig.17 It is an explanatory diagram showing an application example of the portable terminal according to the second embodiment.

[0224] Should Fig.17 In, it means Fig.14 An application example of augmented reality of the portable terminal 1200.

[0225] Fig.17 (a) and Fig.17 (b) shows that the user 1401 uses the built-in camera of the portable terminal 1200 Fig.14 The camera device 1100a is used to take a selfie, thereby using Fig.14 The camera device 1100a is used for conducting a video call, etc.

[0226] exist Fig.17 In (a), the color image captured by the user 1401 using the camera 1100a is directly displayed on the display unit 1217. As a result, the background of the room, such as the wall, is also displayed on the display unit 1217.

[0227] Here, the camera 1100a also captures a distance image, so it is used, such as Fig.17 As shown in (b), the background farther from the user 1401 is displayed on the display unit 1217 overlaid with the preset background image. The background image is, for example, stored in Fig.14 The background image of the storage device 1255, etc.

[0228] Thereby, the functions preferred by the user 1401 can be provided.

[0229] (Implementation 3)

[0230] <Application Example 3>

[0231] Fig.18 It is an explanatory diagram showing an application example of the portable terminal according to the third embodiment.

[0232] Should Fig.18 Indicated in Fig.14 An example of an augmented reality application on a portable terminal. Fig.18 , it shows a state where, for example, the web is browsed using the portable terminal 1200 and the contents on the web are displayed on the display unit 1217 .

[0233] At this time, by using the built-in camera of the portable terminal 1200 Fig.14 The imaging device 1100a is operated to obtain the distance from the face of the user using the portable terminal 1200 based on the distance image. Then, the font size of the display content 1601 and the display content 1602 is enlarged or reduced based on the obtained distance.

[0234] For example, when the distance from the user's face is far, the font size of the displayed contents 1601 and 1602 is enlarged. When the distance from the user's face is close, the font size of the displayed contents 1601 and 1602 is reduced.

[0235] This makes it possible to easily identify the display contents 1601 and 1602 , and to reduce the burden on the user's eyes.

[0236] (Implementation 4)

[0237] <Application Example 4>

[0238] Fig.19 This is an explanatory diagram showing an example of the structure of a portable terminal according to the fourth embodiment.

[0239] Fig.19 The portable terminal 1200 shown has a configuration in which an imaging device 1100 a is provided above a display portion 1217 included in the portable terminal 1200 .

[0240] The camera device 1100a is as follows Figure 1 As shown, pixels for color images and non-visible light pixels 14 as IR pixels having sensitivity to near infrared light are provided. The camera having sensitivity to near infrared light is used for finger vein authentication and the like.

[0241] The non-visible light pixels 14 of the camera 1100a can be used, for example, to perform finger vein authentication. The user's palm 1701 can be brought close to the camera 1100a, and the fingerprint on the surface of the finger can be captured using a color image. In addition, the non-visible light pixels can be used to capture finger veins instead of capturing a distance image.

[0242] In this way, the fingerprint and the finger vein are photographed simultaneously for authentication, thereby having the effect of preventing masquerading using only the fingerprint image.

[0243] Next, other application examples will be described.

[0244] In recent years, glasses-type portable terminals called MR (Mixed Reality) devices have been studied and commercialized. Network information and the like are displayed in a three-dimensional overlapping manner on the field of vision of the wearer wearing glasses, and operation menus and the like are also displayed in an overlapping manner on the field of vision. If the operation is performed in a manner similar to touching it with the hand, the menu will respond and various operations can be performed. These devices can also be applied to the camera device 1100.

[0245] (Implementation 5)

[0246] <Application Example 5>

[0247] Fig. 20It is an explanatory diagram showing an application example of the portable terminal according to the fifth embodiment.

[0248] Should Fig. 20 In the embodiment, the portable terminal 1801 is configured as a glasses type, for example. The portable terminal 1801 includes a right transparent member 1804 , a left transparent member 1805 , a right display unit 1802 , a left display unit 1803 , and the camera 1100 .

[0249] The right transparent member 1804 and the left transparent member 1805 are provided at the position corresponding to the lens of the glasses. The right display unit 1802 and the left display unit 1803 are provided on the upper part of the frame 1806 of the glasses. The camera device 1100 is provided on one side support part of the frame 1806.

[0250] The right transparent member 1804 and the left transparent member 1805 are transparent members coated with a coating that allows light from the outside to pass through and reflects light from the side on which the user wears and observes. The right display unit 1802 and the left display unit 1803 project, for example, a menu for operating the portable terminal 1801 onto the right transparent member 1804 and the left transparent member 1805, respectively.

[0251] In addition, the camera 1100 is a camera capable of simultaneously capturing a color image and a distance image. Fig.13 Since the structure is the same, the description is omitted.

[0252] The user's action of placing his hand on the menu projected by the right display unit 1802 and the left display unit 1803 and operating is captured by the camera device 1100. The operation of the portable terminal 1801 is determined by analyzing the hand movement based on the obtained color image and distance image, and the operation result is displayed on the right display unit 1802 and the left display unit 1803.

[0253] Furthermore, the portable terminal 1801 can exchange information with another portable terminal 1200. The information is exchanged by, for example, communicating via BLUETOOTH (registered trademark, Bluetooth).

[0254] <Configuration Example of Portable Terminal>

[0255] Fig.21 Yes means Fig. 20 A block diagram of an example of the structure of a portable terminal.

[0256] Fig.21 The functional structure of the portable terminal 1801 is Fig.14 The portable terminal 1200 shown is different in that a right display unit 1802 and a left display unit 1803 are newly provided, and the modulation / demodulation unit 1222, the signal separation unit 1221, the display unit 1217, the switch input unit 1250, and the sound input / output unit 1260 are removed.

[0257] The right display unit 1802 and the left display unit 1803 are arranged on Fig. 20 As described above, the operation menus are displayed on the right transparent part 1804 and the left transparent part 1805 respectively at the upper part of the frame 1806 of the glasses-type portable terminal 1801.

[0258] In addition, the camera device 1100 is Fig.13 The same structure can be used to obtain the user's palm movements in three dimensions and perform the menu operation described below. Fig.14 Therefore, the description is omitted.

[0259] <Menu Display Example>

[0260] Fig. 22 It means the user is wearing Fig. 20 An illustration of an example of a menu display when using a portable terminal.

[0261] In the right transparent part 1804 and the left transparent part 1805 , the menu 1902 is displayed in a floating manner in the virtual space.

[0262] The user operates the virtual menu 1902 with the user's palm 1901. Which item is selected from the items in the menu 1901 is displayed by a color image, and the execution command of the selected item is displayed in a manner of judging the movement of the inner side direction of the palm 1901 by using a distance image, thereby providing a portable terminal 1801 that is easy to use.

[0263] Therefore, even in a glasses-type portable terminal where installation space is limited, the imaging device 1100 that can capture a color image and a distance image with a single imaging element can be used, thereby achieving miniaturization. Thus, the wearing feeling of the glasses-type portable terminal 1801 can be improved.

[0264] <Application example of portable terminal>

[0265] Fig.23 Yes means Fig. 20 An illustration of an application example of a portable terminal.

[0266] Fig.23 Representation and Fig. 22 Similarly, when a user wears a glasses-type portable terminal 1801, the user's eyes see through the portable terminal 1801.

[0267] In this Fig.23 The portable terminal 2002 displayed in FIG. 1 is a virtual portable terminal of a smartphone type projected on the glasses-type portable terminal 1801 . The user operates the operation menu projected on the virtual portable terminal 2002 with the palm 2001 .

[0268] This allows the movement of the palm 2001 to be performed within the range of the virtual portable terminal 2002, thereby reducing the possibility of collision with other people's hands during operation in a place where there are people around.

[0269] Furthermore, it would be more convenient if the user could turn on / off a mode for operating by selecting a menu screen of the portable terminal 1801 or a virtual terminal mode for operating by displaying the virtual portable terminal 2002 .

[0270] Or when there are many people nearby during terminal communication, if the display mode of the virtual terminal is controlled to be on, it is possible to prevent the user's hands from colliding with other people's hands while operating in places where there are people around, which is convenient for the user and the people around him.

[0271] Fig.24 Yes means Fig.23 An illustration of another application example of a portable terminal.

[0272] In this Fig.24 In Chinese, it also means Fig.23 Similarly, when a user wears a glasses-type portable terminal 1801, the user's eyes see through the portable terminal 1801.

[0273] For the user, as shown in the figure, operation menu 2102, operation menu 2103, and operation menu 2104 are displayed from left to right. In addition, the portable terminal 1200 in front of the operation menu 2103 is an actual portable terminal, which adopts the same Fig.14 Same structure.

[0274] The glasses-type portable terminal 1801 and the portable terminal 1200 are equipped with Fig.13 The imaging device 1100 shown in FIG. 1 is thus able to grasp the spatial position information of the surroundings of the portable terminal 1801 and the portable terminal 1200 using the color image and the distance image, and thus grasp the relative position of the portable terminal 1801 and the portable terminal 1200.

[0275] As shown in the figure, when the portable terminal 1200 is in the operation menu 2103 displayed on the portable terminal 1801, the portable terminal 1200 becomes the operation terminal of the menu 2103. Similarly, when the portable terminal 1200 is in the operation menu 2102, the selection operation of the menu 2102 can be performed by the portable terminal 1200. In addition, when the portable terminal 1200 is in the operation menu 2104, the selection operation of the menu 2104 can be performed by the portable terminal 1200.

[0276] Thus, the menu selection operation itself can be operated by clicking the screen of the actual portable terminal 1200. As a result, since the actual component can be operated, the user can feel more secure.

[0277] In addition, by utilizing the relative positional relationship between the menus 2102 to 2104 displayed on the portable terminal 1801 and the portable terminal 1200 used in the operation, when the portable terminal 1200 is in front of each of the virtually displayed menus 2102 to 2104, a color image of the portable terminal is displayed covering the menu located on the inner side, thereby having the advantage of allowing the user to easily operate the portable terminal 1200.

[0278] (Implementation 6)

[0279] <Application Examples>

[0280] In the sixth embodiment, an application example in the field of three-dimensional measurement will be described.

[0281] Fig.25 This is an explanatory diagram showing an application example of the imaging device according to the sixth embodiment.

[0282] Fig.25 , for example, a technique for determining the shape of a product in a factory, etc. A conveyor belt 2101 is provided in the factory, etc., and the conveyor belt 2101 transports a produced product 2202.

[0283] An imaging device 1100 is provided above the conveyor belt 2101, and a judgment device 2304 including, for example, a personal computer is connected to the imaging device. Fig.13 same.

[0284] The imaging device 1100 acquires a color image and a distance image of the product 2202. In addition, the color image may be a black-and-white image as required.

[0285] The determination device 2304 determines whether the product 2202 conveyed by the conveyor 2101 is a good product or a bad product. In the determination of good products or bad products, the three-dimensional shape of the product 2202 is determined based on the color image and the distance image captured by the imaging device 1100.

[0286] At this time, the conveying speed of the conveyor belt 2201 can be high.

[0287] As described above, the imaging device 1100 can acquire a color image and a distance image in the same frame time, thereby increasing the frame rate of imaging. As a result, even if the conveying speed of the conveyor belt 2201 is high, the quality of the product 2202 can be stably determined.

[0288] In this way, the transport speed of the conveyor belt 2201 can be increased, thereby improving the production efficiency of the product 2202.

[0289] Furthermore, the imaging device 1100 has an advantage not only in terms of frame rate but also in terms of cost compared to an imaging device that uses an imaging element for a range image and an imaging element for a color image, respectively.

[0290] Fig.26 Yes means Fig.25 An illustration of another application example of the imaging device.

[0291] Should Fig.26 The following is an example in which the imaging device 1100 is applied to an autonomous mobile robot 2301. The imaging device 1100 is provided as the eyes of the robot 2301, and acquires a color image and a distance image.

[0292] The robot 2301 has a control device (not shown) that performs 3D imaging and recognizes obstacles and people ahead based on the data. At this time, the frame rate of photography can be increased by using an imaging element that alternately acquires color images and distance images.

[0293] Furthermore, compared with an imaging device that uses an imaging element for a distance image and an imaging element for a color image, respectively, this is advantageous in terms of cost.

[0294] In addition, by Figure 2 The imaging element 10 is used in a laparoscope or the like which requires a miniaturized imaging unit, and can provide a laparoscope which can capture a distance image and a color image with one imaging element and can confirm the size of an affected part or the like.

[0295] In addition, the above-mentioned embodiments are not limited to a single form. Fig.24 In the example shown, Fig.14 When the portable terminal 1200 is placed at a position farther than the virtual position of the operation menu 2103, the portable terminal 1200 is displayed in a manner such that it is located at a position deeper than the menu 2103.

[0296] Furthermore, when the portable terminal 1200 is placed closer to the virtual position of the operation menu 2103, the portable terminal 1200 is displayed in front of the menu 2103, thereby providing a portable terminal that is more convenient to use. The various embodiments can be freely combined.

[0297] In addition, a part of the structure of a certain embodiment can be converted into the structure of another embodiment, and the structure of another embodiment can be added to the structure of a certain embodiment. In addition, for a part of the structure of each embodiment, other structures can be added, deleted, or replaced.

[0298] In addition, even if part or all of the above structures are constituted by hardware, they can also be constituted in a manner realized by a processor executing a program. In addition, the control lines and information lines represent the parts required for explanation, and do not represent all the control lines and information lines on the product. In fact, almost all structures are connected to each other.

[0299] <Notes>

[0300] In addition, part of the contents described in the embodiment will be described below.

[0301] (1) A portable terminal having an imaging device capable of capturing images, the imaging device comprising: an optical lens unit for acquiring an optical image; an imaging element for acquiring image data from the optical image acquired by the optical lens unit; a light source unit for emitting near-infrared light; and a control unit for controlling the light emission of the light source unit and the operation of the imaging element, the imaging element comprising: a sensor unit including a plurality of visible light pixels sensitive to visible light and a plurality of non-visible light pixels sensitive to non-visible light; a visible light pixel driving unit for controlling exposure of the visible light pixels and reading of charges obtained by photoelectric conversion of the visible light pixels through exposure; and a non-visible light pixel driving unit for controlling exposure of the non-visible light pixels and reading of charges obtained by photoelectric conversion of the non-visible light pixels through exposure, the non-visible light pixel driving unit exposing two or more non-visible light pixels at a predetermined time when exposing and reading the non-visible light pixels, adding up the charges obtained by photoelectric conversion of the two or more non-visible light pixels through exposure and reading them, and the control unit controls so that the light source unit emits near-infrared light when exposing two or more non-visible light pixels.

[0302] (2) In the portable terminal of (1), the non-visible light pixel driving unit performs control so that at least a portion of a first exposure period for exposing two or more non-visible light pixels overlaps with a second exposure period for exposing visible light pixels.

[0303] (3) In the portable device of (2), the non-visible light pixel driving unit includes: a first switch unit for acquiring charges obtained by photoelectric conversion of more than two non-visible light pixels; a first capacitor unit for storing the charges acquired by the switch unit in more than two non-visible light pixels and converting them into voltages; an amplifier unit for amplifying the voltages converted by the first capacitor unit; and a timing control unit for controlling the exposure timing of more than two non-visible light pixels and the operation of the first switch unit, the timing control unit controls the first switch unit so that the charges obtained by photoelectric conversion in the first capacitor unit are stored separately for each non-visible light pixel, and when more than two non-visible light pixels are exposed, the more than two non-visible light pixels are exposed sequentially in a manner such that the exposure periods do not overlap.

[0304] (4) In the portable terminal of (2), the non-visible light pixel driving unit has a second switch unit for reading the charges obtained by photoelectric conversion of multiple non-visible light pixels possessed by the sensor unit, and the timing control unit controls the second switch unit for selecting two or more non-visible light pixels to add the charges during exposure and reading.

[0305] (5) In the portable terminal of (4), the timing control unit controls the second switch unit according to a first control command input from the outside so that the number of non-visible light pixels to which the charges are added changes.

[0306] (6) In the portable terminal of (3), the non-visible light pixel driving unit has a distance image calculation unit that calculates the voltage output from the amplifier to generate and output a distance image, and the timing control unit controls so that the calculation timing of the voltage calculated by the distance image calculation unit is 1 / N of the frame rate.

[0307] (7) In the portable terminal of (6), the timing control unit controls the calculation timing of the range image calculation unit according to the second control command input from the outside.

[0308] Description of Reference Numerals

[0309] 10. Camera Components

[0310] 11 Visible light pixels

[0311] 14 Non-visible light pixels

[0312] 210 Visible light pixel driving unit

[0313] 214 Control Department

[0314] 220 Non-visible light pixel driving unit

[0315] 230 Output interface

[0316] 321~323 switch

[0317] 331~333 capacitor element

[0318] 341~343 switch

[0319] 351 Amplifier

[0320] 352 Visible light timing generation unit

[0321] 352 Amplifier

[0322] 353 Image data generation unit

[0323] 361 Switch

[0324] 362 Switch

[0325] 371 Capacitor Components

[0326] 372 Capacitor Components

[0327] 381~384 amplifier

[0328] 382 Non-visible light timing generation unit

[0329] 385 Invisible Light Timing Generation Unit

[0330] 386 distance data calculation and generation unit

[0331] 391~394 switch

[0332] 1100 Camera Device

[0333] 1101 Optical lens department

[0334] 1103 IR light source

[0335] 1104 Camera Control Unit

[0336] 1200 Portable Terminal

[0337] 1213 Communication Interface

[0338] 1214 Control Department

[0339] 1215 Button

[0340] 217 Display

[0341] 1221 Signal Separation Unit

[0342] 1222 Modulation / Demodulation Unit

[0343] 1225 Storage

[0344] 1231 Mobile Communication Interface

[0345] 1232 Memory

[0346] 1233 Acceleration sensor unit

[0347] 1234 Geomagnetic sensor department

[0348] 1235 GPS receiving unit

[0349] 1236 Gyro Sensor Department

[0350] 1250 switch input

[0351] 1260 Sound input and output unit

[0352] 1261 Microphone

[0353] 1262 Speakers

[0354] 1802 Right display unit

[0355] 1803 Left display unit.

Claims

1. A camera device, characterized in that: include: a sensor comprising a plurality of visible light pixels having sensitivity to visible light and a plurality of non-visible light pixels having sensitivity to non-visible light; a non-visible light pixel driving unit that controls a first exposure of the non-visible light pixel, performs a first reading operation of reading a charge generated by photoelectric conversion of the non-visible light pixel resulting from the first exposure, and generates a distance image; and a visible light pixel driving unit that controls a second exposure of the visible light pixel, performs a second reading operation of reading the charge generated by the photoelectric conversion of the visible light pixel resulting from the second exposure, and generates a visible light image, The non-visible light pixel driving unit, performing the first exposure on two or more of the non-visible light pixels, controlling so that, during the first exposure, at least a portion of a first exposure period during which the first exposure of the two or more non-visible light pixels is performed overlaps with a second exposure period during which the second exposure of the visible light pixels is performed, adding the charges generated by the photoelectric conversion of the two or more non-visible light pixels resulting from the first exposure, reading the added charge in the first reading operation, A range image is generated based on the added charges.

2. The camera device according to claim 1, wherein: The non-visible light pixel driving unit includes a first timing generating unit that generates a first control signal for controlling the timing of the first reading operation. The visible light pixel driving unit includes a second timing generating unit for generating a second control signal for controlling the timing of the second reading operation. The first timing generating unit receives the second control signal generated by the second timing generating unit and generates the first control signal. In the first reading operation, the added charge is read at a timing corresponding to the first control signal generated by the first timing generating section.

3. The camera device according to claim 2, wherein: The non-visible light pixel driving unit comprises: a first switch unit for acquiring charges obtained by photoelectric conversion of the two or more non-visible light pixels; a first capacitor that stores the charges acquired by the first switch unit in the two or more non-visible light pixels and converts the charges into voltages; and an amplifier for amplifying the voltage converted by the first capacitor, The first switch section is controlled based on the signal generated by the first timing generator so that the charges obtained by photoelectric conversion are stored in the first capacitor section for each of the non-visible light pixels.

4. The camera device according to claim 3, wherein: The non-visible light pixel driving unit includes a second switch unit for reading charges obtained by photoelectric conversion of the plurality of non-visible light pixels included in the sensor unit. The second switch unit is controlled to select the two or more non-visible light pixels based on the signal generated by the first timing generator.

5. The camera device according to claim 4, wherein: The non-visible light pixel driving unit controls the second switch unit according to a first control command input from the outside, so that the number of the non-visible light pixels to which charges are added changes.

6. The camera device according to claim 3, wherein: The non-visible light pixel driving unit includes a distance image calculation circuit that calculates the voltage output from the amplifier to generate a distance image and outputs the distance image. The distance image operation circuit is controlled so that the operation timing of the voltage is 1 / N of the frame rate.

7. The camera device according to claim 6, wherein: The non-visible light pixel driving unit controls the calculation timing according to a second control command input from the outside.

8. The camera device according to claim 1, wherein: During the first exposure, the two or more non-visible light pixels are sequentially exposed so that exposure periods of the non-visible light pixels do not overlap with each other.

9. A portable terminal, characterized in that: include: Input section; a sensor comprising a plurality of non-visible light pixels having sensitivity to non-visible light and a plurality of visible light pixels having sensitivity to visible light; A camera unit having the sensor; A display unit for displaying a virtual object; and Control Department, The control unit, controlling a first exposure of the non-visible light pixel of the sensor and a first reading of charges obtained by photoelectric conversion through the first exposure, controlling a second exposure of the visible light pixel of the sensor and a second reading of the charge obtained by photoelectric conversion of the visible light pixel by the second exposure, Control is performed so that in the same frame, image data of a real object captured by the camera unit based on the visible light pixels of the sensor and distance data corresponding to the image data based on the non-visible light pixels of the sensor are generated, controlling to display the real object and the virtual object on the display unit based on the position of the virtual object and the distance data, Control is performed to change the number of the non-visible light pixels to which charge addition is performed based on an input to the input section.

10. The portable terminal according to claim 9, characterized in that: The control unit performs control, The first exposure is performed for each of two or more non-visible light pixels. Charges obtained by photoelectric conversion of the two or more non-visible light pixels through the first exposure are added and the added charges are read.

11. The portable terminal according to claim 9, characterized in that: The imaging unit includes a non-visible light pixel driving unit and a visible light pixel driving unit. The non-visible light pixel driving unit performs the first exposure and the first reading, The visible light pixel driving unit performs the second exposure and the second reading, The control unit controls the non-visible light pixel driving unit and the visible light pixel driving unit.

12. The portable terminal according to claim 9, characterized in that: The control unit performs control, to generate display data of the virtual object based on the distance data of the real object and the position of the virtual object, The display data and the image data are synthesized and the synthesized data is displayed on a display unit.

13. The portable terminal according to claim 9, characterized in that: The portable device has a light source unit capable of emitting near-infrared light. The control unit controls the light source unit to emit the near-infrared light when performing the first exposure.

14. The portable terminal according to claim 9, characterized in that: The control unit controls so that at least a portion of a period during which the first exposure is performed on the two or more non-visible light pixels overlaps with a period during which the second exposure is performed on the visible light pixels.

Images