Solid-state imaging device, electronic apparatus, and program

Abstract

[Problem] To improve authentication accuracy. [Solution] A solid-state imaging device includes a light source, a first light-receiving region, and a second light-receiving region. A light source is disposed on a side of the display opposite a display surface of the display, the light source emitting light in an IR region via the display. The first light receiving region is provided on a side of the display opposite to the display surface, and is provided with pixels for receiving light in a visible light region and pixels for receiving at least light in an IR region emitted from the light source. The second light receiving area is provided on a side of the display opposite to the display surface, and is provided with pixels for receiving at least light in the IR area emitted from the light source.

Description

Technical Field

[0001] The present disclosure relates to a solid-state imaging device, an electronic device, and a program. Background Art

[0002] The implementation of a face authentication function has become a widely used technology. When implementing a process requiring various types of security in a mobile terminal, it is desirable to improve the accuracy of the face authentication function in the mobile terminal. As a face authentication method, there are a method of acquiring both a range image and a two-dimensional image through a depth camera, a method of acquiring a range image through a depth camera and acquiring a two-dimensional image through an RGB camera, and a method of acquiring a two-dimensional image and a parallax by using two RGB cameras and acquiring a range image from the parallax.

[0003] In the method using the RGB camera, there are problems in that it is difficult to acquire a two-dimensional image in a dark place, and the number of cameras increases, which causes design restrictions in the mobile terminal.

[0004] On the other hand, generally, there is a difference between the distance information required in the face authentication technology and the resolution of the two-dimensional image, and the two-dimensional image requires a high-resolution image. At this time, in the case of a format in which two images are acquired by a depth camera, there is a problem that a depth with a higher resolution than the required resolution needs to be acquired in order to maintain the resolution of the two-dimensional image. In addition, in a mobile terminal, it is only necessary to make a hole for the camera on the display side for face authentication, which is also undesirable.

[0005] Reference List

[0006] Patent Literature

[0007] Patent document 1: US2021 / 0406350 A Summary of the invention

[0008] Problems to be solved by the present invention

[0009] Therefore, the present disclosure provides a solid-state imaging device that improves authentication accuracy.

[0010] Solution to the problem

[0011] According to an embodiment, a solid-state imaging device includes a light source, a first light receiving area, and a second light receiving area. The light source is arranged on the opposite side of the display surface of the display, and emits light in the infrared light band via the display. The first light receiving area is arranged on the opposite side of the display surface of the display, and includes pixels that receive light in the visible light band and pixels that receive at least light in the infrared light band emitted from the light source. The second light receiving area is arranged on the opposite side of the display surface of the display, and includes pixels that receive at least light in the infrared light band emitted from the light source.

[0012] The solid-state imaging device may further include: a first processing circuit that generates an infrared light image based on the intensity of infrared light received in the first light receiving area; and a second processing circuit that generates a depth image based on the intensity of infrared light received in the second light receiving area.

[0013] The solid-state imaging device may further include a third processing circuit that determines whether the object is an authentication target based on the depth image, and performs authentication processing using at least the infrared light image if it is determined that the object is an authentication target.

[0014] The first light receiving area may include a dual band pass filter having a transmission characteristic in two bands of a visible light band and an infrared light band between the first light receiving area and a display surface of the display.

[0015] The second light receiving area may include a band pass filter having a transmission characteristic in an infrared light band between the second light receiving area and a display surface of the display.

[0016] The light source may include a surface light source.

[0017] The light source may include a surface light source receiving light in the first light receiving area and a point light source receiving light in the second light receiving area.

[0018] The solid-state imaging device may further include a light shielding wall between the first light receiving region and the second light receiving region on the opposite side of the light source to the display surface of the display.

[0019] The solid-state imaging device may further include a control circuit that performs drive control of the first light receiving area, drive control of the second light receiving area, and control of the light source. The control circuit may issue an identifier, perform light emission control of the light source, associate the identifier with the acquired infrared light image and the acquired depth image, respectively, perform extinction control of the light source, and determine that the infrared light image and the depth image are images acquired at the same time when the identifiers match each other.

[0020] The solid-state imaging device may further include a control circuit that performs drive control of the first light receiving area, drive control of the second light receiving area, and control of the light source. The control circuit may issue an identifier, perform light emission control of a surface light source, associate the identifier with the acquired infrared light image, perform extinction control of the surface light source, perform light emission control of a point light source, associate the identifier with the acquired depth image, perform extinction control of the point light source, and determine that the infrared light image and the depth image are images acquired at the same time when the identifier associated with the infrared light image and the identifier associated with the depth image match each other.

[0021] The first processing circuit may previously acquire an interference infrared light image by emitting light from a light source onto a plane arranged at a distance from the object, and may correct the acquired infrared light image using the interference infrared light image.

[0022] The second processing circuit may acquire an interference depth image in advance by emitting light from a light source onto a plane arranged at a distance of the object, and may correct the acquired depth image using the interference depth image.

[0023] The pixels belonging to the first light receiving area and the pixels belonging to the second light receiving area may be arranged in the same pixel array.

[0024] According to an embodiment, an electronic device includes a display, a light source, a first light receiving area, a second light receiving area, and a processing circuit. The display displays information. The light source is arranged on the opposite side of the display surface of the display, and emits light in the infrared light band via the display. The first light receiving area is arranged on the opposite side of the display surface of the display, and includes pixels that receive light in the visible light band and pixels that receive at least light in the infrared light band emitted from the light source. The second light receiving area is arranged on the opposite side of the display surface of the display, and includes pixels that receive at least light in the infrared light band emitted from the light source. The processing circuit performs authentication processing using an infrared light image acquired by pixels belonging to the first light receiving area and a depth image acquired by pixels belonging to the second light receiving area.

[0025] The processing circuit may determine whether the object is an authentication target based on the depth image, and may perform authentication processing using at least the infrared light image if it is determined that the object is an authentication target.

[0026] According to an embodiment, the program causes a processor to execute the process of the control circuit according to any one of the above.

[0027] According to an embodiment, the program causes a processor to execute the processing of the processing circuit according to any one of the above.

[0028] The electronic device may include at least one of a mobile terminal, a smart phone, a tablet terminal, a vehicle-mounted camera with a display, an authentication device with a display, or a monitoring camera with a display. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a diagram schematically illustrating an electronic device according to an embodiment.

[0030] Figure 2 is a diagram showing an example of arrangement of pixels according to an embodiment.

[0031] Figure 3 is a block diagram schematically showing a solid-state imaging device according to an embodiment.

[0032] Figure 4 is a block diagram schematically showing a solid-state imaging device according to an embodiment.

[0033] Figure 5 is a flowchart showing processing in the solid-state imaging device according to the embodiment.

[0034] Figure 6 is a diagram schematically illustrating an electronic device according to an embodiment.

[0035] Figure 7 is a flowchart showing processing in the solid-state imaging device according to the embodiment.

[0036] Figure 8 is a diagram schematically illustrating an electronic device according to an embodiment.

[0037] Fig. 9 is a diagram schematically illustrating an electronic device according to an embodiment.

[0038] Fig.10 is a diagram illustrating an example of interference fringes according to an embodiment.

[0039] Fig.11 is a diagram illustrating an example of interference fringes according to an embodiment.

[0040] Fig.12 is a diagram schematically illustrating an electronic device according to an embodiment.

[0041] Fig.13 is an external view of an electronic device according to an embodiment.

[0042] Fig.14 is an external view of an electronic device according to an embodiment.

[0043] Fig.15 is a block diagram showing an example of a schematic configuration of a vehicle control system.

[0044] Fig.16 It is a diagram showing an example of the installation positions of the vehicle exterior information detection section and the imaging section. DETAILED DESCRIPTION

[0045] Embodiments of the present disclosure are described below with reference to the accompanying drawings. The accompanying drawings are for explanation, and the shape and size of each configuration in an actual device, the ratio of the size to other configurations, etc. are not necessarily as shown in the drawings. In addition, because the drawings are shown in a simplified manner, configurations necessary for implementation other than those shown in the drawings are appropriately provided.

[0046] (First Embodiment)

[0047] Figure 1 1 is a block diagram schematically showing an electronic device including a solid-state imaging device according to an embodiment. The electronic device 1 includes a solid-state imaging device 10 and a display 100. The electronic device 1 displays information on the display, captures an image through the solid-state imaging device 10 provided below the display, and performs authentication based on the captured image, etc.

[0048] For example, the electronic device 1 may be at least one of a mobile terminal, a smart phone, a tablet terminal, a vehicle-mounted camera with a display, an authentication device with a display, or a surveillance camera with a display. As an example, the electronic device 1 is configured to be able to perform authentication processing on the display surface side of these devices.

[0049] The display 100 is an output interface for displaying information of the electronic device 1. The display can be implemented by a method such as liquid crystal or organic EL. In addition, the display 100 can also be used as an input interface including a touch panel or the like for inputting information to the electronic device 1. The display 100 includes a display portion 102, and also includes a display surface 100A and a back surface 100B as the display surface 100A.

[0050] The display unit 102 is an area for displaying information. According to the specifications of the first light receiving area described later, the display unit 102 can be provided with an opening 104. For example, the opening 104 can be set to an area where no display pixel is provided. As another example, the opening 104 can be implemented by thinning out the pixels in the area, or can be implemented by using pixels with transparency to a predetermined wavelength. In this case, for the transmitted pixel, the pixel circuit may have a configuration different from the configuration of the pixel in another display unit 102.

[0051] The display surface 100A is a surface where a user views displayed information from the outside, and may be coated with, for example, glass, a filter, etc., or may be provided with a polarizing plate, a polarizing filter, etc. so that light output from pixels can be appropriately viewed from the outside.

[0052] The back surface 100B is a surface opposite to the display surface 100A of the display 100, and faces the inside of the electronic device 1. In the present disclosure, the area on the back surface 100B side in the electronic device 1 may be referred to as an area below the display.

[0053] The solid-state imaging device 10 is a device included in the electronic device 1 and is provided below the display 100. The solid-state imaging device 10 includes, for example, a light source 11, a first light receiving area 12, a second light receiving area 13, a dual bandpass filter 14, and a bandpass filter 15.

[0054] The light source 11 is a light source used in the authentication process, and is, for example, a light source that emits light in the infrared light band, and is disposed on the back surface 100B side of the display 100, that is, below the display 100. The light source 11 includes, for example, a surface light source. A plane wave (which may not be a strict plane wave) emitted from the light source 11 illuminates an object via the display 100, is reflected by the object, and is incident on a light receiving element in the solid-state imaging device 10. It is to be noted that the light source 11 does not have to be disposed in the solid-state imaging device 10, and may be disposed in the electronic device 1 as a module different from the solid-state imaging device 10.

[0055] The wavefront of the light emitted from the light source 11 propagates as shown by the dotted line, and the light reflected by the object of the irradiated object is incident on the first light receiving area 12 and the second light receiving area 13, and information for generating an image is acquired. Note that although the distance between the object and the electronic device 1 is very short due to the space shown in the drawings, in practice, the distance can be long enough. Therefore, the light emitted from the light source 11 is irradiated roughly in front of the object, and the reflected light can be received by the pixels belonging to each light receiving area.

[0056] The first light receiving area 12 is an area disposed below the display 100 and includes pixels that receive light via the opening 104 of the display 100 and output signals based on the intensity of the received light. The first light receiving area 12 includes pixels that receive light in the visible light band and pixels that receive infrared light. For example, the pixels that receive light in the visible light band may be pixels that receive each of the three primary colors of red (R), green (G), and blue (B), or may be pixels that receive complementary colors or white. The pixels that receive light in the infrared light band are mixed and arranged in the same area (pixel array) as the pixels that receive light in the visible light band.

[0057] Figure 2 is a diagram showing an example of light receiving bands of pixels according to an embodiment. R represents a pixel receiving a red wavelength band, G represents a pixel receiving a green wavelength band, B represents a pixel receiving a blue wavelength band, and IR represents a pixel receiving an infrared wavelength band. As shown in the figure, IR pixels can be mixed with R, G, and B pixels and arranged in the same pixel array.

[0058] By providing Figure 2 The IR pixels shown can acquire an infrared light image with a high resolution similar to that of an image in the visible light band. However, the arrangement of the pixels is not limited thereto, and may be an arrangement including complementary colors and white as described above, an arrangement corresponding to a multi-spectrum, or a configuration including pixels that can be used for various purposes such as a plasma filter and pixels for acquiring an image plane phase difference.

[0059] As described above, the first light receiving area 12 is arranged by mixing pixels for acquiring visible light images and pixels for acquiring infrared light images, and information for generating infrared light images with high resolution can be acquired. For example, the electronic device 1 can perform authentication by using infrared light images. The biometric authentication can be, for example, biometric authentication based on information that can be acquired by infrared light, such as facial authentication, iris authentication, or vein authentication of a person. In addition, since the authentication uses infrared light, authentication of a predetermined object can be achieved even in a dark place.

[0060] return Figure 1 A dual band pass filter 14 having transmission characteristics in the visible light band and the infrared light band may be disposed on the light receiving side of the first light receiving region 12. The dual band pass filter 14 may allow light within an appropriate band to enter the first light receiving region 12.

[0061] The second light receiving area 13 is, for example, disposed below the display 100, and is an area in which pixels for receiving light dedicated to the infrared region are arranged. The second light receiving area 13 is, for example, an area including pixels for measuring the time of flight (ToF). For example, the second light receiving area 13 measures the time from when the light source 11 emits light to when the reflected light from the object returns to each pixel, and measures the distance to each position of the object corresponding to the pixel. A depth image (distance image) can be generated based on the distance.

[0062] A bandpass filter 15 having a transmission characteristic in the infrared light band may be provided between the second light receiving area 13 and the display 100. When light via the bandpass filter 15 enters the second light receiving area 13, information related to ToF measurement may be appropriately acquired in the second light receiving area 13.

[0063] According to the characteristics of the display 100 , in the case where the display 100 has a form capable of appropriately transmitting infrared light, for example, the light source 11 and the second light receiving area 13 may emit light or receive light without passing through the opening 104 .

[0064] The dual-bandpass filter 14 and the bandpass filter 15 do not need to be provided separately, and may be provided integrally with a lens for converging light from a subject to each light receiving area, for example. That is, in these filters, for example, by appropriately selecting a substance applied to the lens surface or the material of the lens itself, the lens itself may have the function of a filter.

[0065] Next, a non-limiting example of an internal configuration for processing the information acquired as described above will be described.

[0066] Figure 31 is a block diagram schematically showing an example of a solid-state imaging device 10 according to an embodiment. The solid-state imaging device 10 includes a control circuit 20, a storage circuit 21, a first processing circuit 22, a second processing circuit 23, and a third processing circuit 24 in addition to a light source 11, a first light receiving area 12, and a second light receiving area 13. Not shown Figure 1 Filters in, etc. Furthermore, although not shown, other configurations necessary for the operation of the solid-state imaging device 10 are appropriately provided although not shown.

[0067] The control circuit 20 controls the solid-state imaging device 10. For example, the control circuit 20 controls the driving of the pixel at the light emission timing of the light source 11 and the light reception timing of the first light receiving area 12 and the second light receiving area 13. In addition, the control circuit 20 can control the first processing circuit 22, the second processing circuit 23, and the third processing circuit 24 at appropriate timings. That is, the control circuit 20 can control imaging control and operations related to authentication in the solid-state imaging device 10.

[0068] The storage circuit 21 is a circuit that stores data necessary for the operation of the solid-state imaging device 10 and data to be acquired. The storage circuit 21 may include at least one of a volatile memory and a nonvolatile memory, a memory, or the like.

[0069] Part or all of the processing of the control circuit or the processing circuit may be implemented by information processing of software. In this case, some or all of the operations may be implemented by software, and information processing by software may be implemented specifically by using a circuit such as a processor, and a program, an execution file, etc. for executing the software may be stored in the storage circuit 21. Not only in this embodiment, but also in the embodiments described later, some or all of the processes may be described in a program, and information processing by software may be implemented using hardware resources.

[0070] The first processing circuit 22 generates at least an infrared light image from a signal acquired based on the intensity of light in the first light receiving area 12. The infrared light image is an image generated based on a signal output from infrared light receiving pixels belonging to the first light receiving area 12. It should be noted that it is not excluded that the first processing circuit 22 generates a visible light image together with the infrared light image.

[0071] The second processing circuit 23 generates a depth image based on a signal acquired based on the light reception time in the second light receiving area 13. The depth image is, for example, a distance image of an area including an object generated by a distance based on the light reception time in the ToF pixel. Using the distance measurement pixels belonging to the second light receiving area 13, infrared light reflected by the object is received for each pixel, and the second processing circuit 23 generates a depth image based on the distance acquired for each pixel.

[0072] The third processing circuit 24 performs authentication processing based on the image data acquired by the first processing circuit 22 and the second processing circuit 23. For example, the third processing circuit 24 uses the depth image generated by the second processing circuit 23 to determine whether the object is valid as an authentication target, and performs authentication using the infrared light image generated by the first processing circuit 22 based on the result.

[0073] The third processing circuit 24 determines whether the object is a person's face using, for example, a depth image. The third processing circuit 24 determines whether the acquired object information has a three-dimensional structure with appropriate concavities and convexities as a person's face using a depth image, and in the case where the acquired object information is recognized as a person's face, performs authentication processing using an infrared light image by an authentication method using any infrared light, such as feature amount extraction of a face, an iris image, and a vein image.

[0074] The third processing circuit 24 can determine whether the object includes authenticated symmetry by applying rule-based processing or processing using a learned model to the depth image. In the case of performing authentication processing, the third processing circuit 24 can perform authentication processing on the object to be authenticated by applying rule-based processing or processing using a learned model to the infrared light image.

[0075] In addition, the third processing circuit 24 may obtain information about the area where the authentication target exists in the depth image. In this case, the third processing circuit 24 may further extract the area where the object to be authenticated exists from the infrared light image and perform authentication processing on the image area.

[0076] In the identification process, the third processing circuit 24 may use information stored in the storage circuit 21 or information registered in a database or the like outside the electronic device 1, if necessary. Such information may be appropriately encrypted and stored, or may be stored as irreversible data that cannot be returned to personal information (such as feature amounts) or is difficult.

[0077] As described above, according to the present embodiment, in an electronic device including a display, in the case of performing authentication processing using a light source and an imaging element provided under the display, more accurate processing can be achieved using a depth image and an infrared light image. In this processing, since the emission of light within the infrared band and the image of light within the infrared band are used, appropriate authentication can be performed even in a dark place. Using this electronic device, face authentication processing in a smartphone or the like can be achieved, and a monitoring system using a tablet terminal or the like can be achieved.

[0078] Note that the authentication target may be not only a person's face but also other parts of a person, such as the veins of a hand, or may be a non-person target, such as a pet or a vehicle number in front of or behind the vehicle-mounted device. In addition, authentication may be a process such as acquiring the feeling, state, etc. of a person facing the display. For example, the electronic device may acquire information such as the drowsiness and emotions of a person facing the display in the vehicle-mounted device, and appropriately issue an alarm, etc.

[0079] Furthermore, the electronic device may be a display or the like of a conference system using the web, and in this case, authentication processing such as whether the target is appropriately present in the conference or whether the appropriate person is checking via the web can be implemented.

[0080] Figure 4 is a diagram showing another example of the solid-state imaging device 10. As shown in the figure, the third processing circuit 24 may be provided outside the solid-state imaging device 10. That is, the solid-state imaging device 10 may be configured to perform processing up to the image generation processing without performing the authentication processing, and output the processing to the third processing circuit 24 that performs the authentication processing included in the electronic device 1.

[0081] Figure 5 : is a flowchart showing the processing of the solid-state imaging device 10 according to the embodiment.

[0082] When performing the authentication process, the control circuit 20 issues an identifier (S100). This identifier is an identifier for synchronizing the captured images. The identifier only needs to be able to identify whether the image acquisition time is the same time or different time, and can be information based on the time when the identifier is issued, for example.

[0083] After issuing the identifier, the control circuit 20 transmits a lighting signal for performing lighting control of the light source 11 to the light source 11 ( S101 ).

[0084] The light source 11 receiving the light emission signal emits light in the infrared band to the outside of the electronic device 1 via the display 100 (S202). The infrared light emitted from the light source 11 outside the electronic device 1 is used to illuminate the object.

[0085] The infrared light reflected by the object enters the first light receiving area 12 and the second light receiving area 13, and is converted into an analog signal based on the intensity information in each light receiving area. The first processing circuit 22 generates an infrared light image based on the signal output from the first light receiving area 12 (S403), and the second processing circuit 23 generates a depth image based on the signal output from the second light receiving area 13 (S303).

[0086] Note that the second processing circuit 23 can obtain the light emission signal from the control circuit 20, and can generate the second image using the light emission time and the light receiving time. However, it is also possible to obtain a relative distance image within the object without obtaining information related to the light emission time. For example, it is also possible to obtain the time when the reflected light is obtained in the second light receiving area 13 for each pixel, and use the image in the depth direction based on the reference point as the depth image based on the earliest time.

[0087] The above-described process does not exclude the generation of a visible light image based on the signal received in the first light receiving area 12 .

[0088] After acquiring the infrared light image, the first processing circuit 22 associates the identifier issued in S100 with the infrared light image (S404). Similarly, after acquiring the depth image, the second processing circuit 23 associates the identifier issued in S100 with the depth image (S304). Each of the first processing circuit 22 and the second processing circuit 23 sends the acquired image and identifier information to the control circuit 20. For example, the control circuit 20 may send the identifier to the first processing circuit 22 and the second processing circuit 23.

[0089] The light source 11 emits light for a predetermined time and then turns off the light (S205). The predetermined time can be determined, for example, based on the moment when the reflected light fully reaches the first light receiving area 12 and the second light receiving area 13, when the object exists at a specified distance, for example, based on the moment when the action of one frame in each light receiving area ends. As another example, the first processing circuit 22 and the second processing circuit 23 can be triggered by generating an image for performing extinction. In addition, the light source 11 can be extinguished by obtaining a trigger through the control circuit 20 and the control circuit 20 sending an extinction signal to the light source 11.

[0090] The control circuit 20 that acquires the image associated with the identifier determines whether the identifier matches ( S106 ). By matching the identifiers, it can be determined that the received depth image and infrared light image are images acquired at the same light emission time of the light source 11 .

[0091] Note that the timestamp may also be used as an identifier. The first processing circuit 22 and the second processing circuit 23 may associate the timestamp when the image is acquired with the image as an identifier at the moment when the identifier is associated. In this case, the control circuit 20 may also determine whether the identifier matches by determining whether the timestamp acquired by the control circuit 20 in S100 is consistent with the timestamp when the first processing circuit 22 and the second processing circuit 23 acquire the image. For example, the control circuit 20 may determine whether the identifier matches by determining whether the timestamp associated with the infrared light image and the depth image is a timestamp acquired later than the timestamp acquired in S100 and a timestamp acquired within a predetermined time.

[0092] In the case where the identifiers do not match ( S107 : NO), the control circuit 20 may repeatedly execute the processing from the imaging control (processing from S100 ), or may end the processing on the assumption that identification cannot be performed.

[0093] If the identifiers match (S107: YES), the control circuit 20 causes the third processing circuit 24 to perform authentication processing (S108). As described above, the third processing circuit 24 determines whether the object is an authentication target based on the depth image, and performs authentication using the infrared light image if the object is an authentication target.

[0094] Note that the third processing circuit 24 may also check whether the identifier matches. That is, the first processing circuit 22 and the second processing circuit 23 may send the infrared light image and the depth image together with the identifier to the third processing circuit 24. Then, after determining the identifier, the third processing circuit 24 may perform the recognition process. As another example, the control circuit 20 may perform the operation of the third processing circuit 24. That is, the control circuit 20 and the third processing circuit 24 may be configured by the same processing circuit.

[0095] By processing the identifier in this manner, after the object is set as an authentication target in the depth image, the solid-state imaging device 10 can avoid erroneous authentication in the infrared light image by replacing the object with a photograph or the like.

[0096] (Second Embodiment)

[0097] In the above-mentioned embodiment, the case where there is one light source has been described. The light source 11 is, for example, a light source that emits a plane wave, and the infrared light image in the first light receiving area 12 and the depth image in the second light receiving area 13 are acquired using the light source. Because the depth image can be operated at a lower resolution than the infrared light image, it is also effective to use a point light source with higher intensity and lower resolution than the plane wave. In the present embodiment, a light source for acquiring a depth image is provided separately.

[0098] Figure 6 Schematic diagram of an electronic device 1 according to an embodiment. The electronic device 1 includes a first light source 110 and a second light source 112 as the light source 11.

[0099] The first light source 110 is a light source that emits light suitable for receiving reflected light from an object in the IR pixels included in the first light receiving area 12. The first light source 110 includes, for example, a surface light source, and emits a plane wave to the object. As shown by the dotted line in the figure, the object is illuminated with the light emitted from the first light source 110, and the reflected light is received by the first light receiving area 12.

[0100] The second light source 112 is a light source that emits light suitable for receiving reflected light from an object in a ToF pixel or the like included in the second light receiving area 13. For example, the second light source 112 includes a point light source having a higher intensity from one point than the intensity of the first light source 110. The light emitted from the second light source 112 is emitted to the object as shown by the dotted line in the figure, and the reflected light is received by the ToF pixel or the like provided in the second light receiving area 13, and a signal for forming a depth image according to the light reception timing can be acquired.

[0101] Figure 7 is a flowchart showing the processing of the solid-state imaging device 10 according to the embodiment. Figure 5 The processes denoted by the same reference symbols are basically the same processes, and thus their detailed descriptions will be omitted.

[0102] After issuing the identifier ( S100 ), the control circuit transmits a light emitting signal of the second light source 112 to the second light source 112 ( S110 ).

[0103] For example, the second light source 112 having received the light emission signal emits light of a point light source (S211). The light of the point light source has an intensity required to acquire ToF information from a shape without an opening.

[0104] The second processing circuit 23 generates a depth image based on information of the timing of receiving light in the second light receiving area 13 ( S303 ), and associates the depth image with the identifier ( S304 ).

[0105] The second light source 112 is turned off at a predetermined time (S212). The timing of the light turning off may be the time according to the above-mentioned embodiment.

[0106] Next, the control circuit 20 transmits a lighting signal of the first light source 110 (S113). For example, the lighting signal may be transmitted at a predetermined time after the lighting signal of the second light source 112 is transmitted, or after the second light source 112 is confirmed to be turned off.

[0107] The first light source 110 that receives the light emission signal emits, for example, a plane wave ( S214 ). The plane wave has an intensity level required to acquire an infrared light image through an opening.

[0108] The first processing circuit 22 generates an infrared light image based on the intensity information received in the first light receiving area 12 ( S404 ), and associates the infrared light image with the identifier ( S404 ).

[0109] The first light source 110 is turned off at a predetermined time (S215). The time of turning off the light may also be the time according to the above embodiment.

[0110] The subsequent processing is similar to that of the aforementioned embodiment. By performing such processing, even in the case where a light source of the light receiving area is set, the authentication processing can be appropriately implemented.

[0111] (Third Embodiment)

[0112] Figure 8 is a diagram showing an example of the arrangement of light receiving pixels according to an embodiment. The first light receiving area 12 and the second light receiving area 13 can be implemented in the same area. That is, the visible light receiving pixels that obtain information in the visible light band, the infrared light receiving pixels that obtain information in the infrared light band, and the ToF pixels that obtain ToF information from the information in the infrared light band can be arranged in the same pixel array.

[0113] With this arrangement, the circuit area in which the photoelectric conversion element is provided as the light receiving element can be reduced. In addition, the coordinate deviation between the depth image and the infrared light image can be minimized, and the accuracy of the process of determining the authentication target in the depth image and the authentication process in the infrared light image can be further improved.

[0114] (Fourth Embodiment)

[0115] Light emitted from the light source 11 is emitted to the outside via the display 100, and therefore may interfere in the configuration of the display 100. In the present embodiment, a solid-state imaging device 10 that suppresses the influence of interference fringes that may be generated by this interference will be described.

[0116] Fig. 9 1 is a diagram schematically showing an example of an electronic device 1 according to an embodiment. As a previous step of performing authentication processing, the electronic device 1 illuminates a flat surface 3 with light from a light source 11. The electronic device 1 previously acquires an infrared light image generated based on reflected light on a flat surface, and generates a pattern of interference fringes at the position of a pixel.

[0117] Fig.10 is a diagram showing a non-limiting example of interference fringes on a plane. The solid-state imaging device 10 may acquire in advance such as Fig.10 The image of the interference fringes shown in is a stage before the authentication process is performed.

[0118] For example, the solid-state imaging device 10 projects interference fringes by irradiating a plane with the light source 11, and generates an infrared light image in which interference fringes exist from the image received in the first light receiving area 12. The infrared light image is an infrared light image indicating the interference state of the display 100. The solid-state imaging device 10 can correct the image at the stage of the authentication process using the acquired infrared light image with interference fringes.

[0119] Fig.11is a diagram showing an example of interference fringes in an object area in an infrared light image according to an embodiment. The first processing circuit 22 can perform subtraction from the image so as to suppress the influence of interference fringes from the area used for authentication processing at the time of acquiring the infrared light image. In the case of performing subtraction, the first processing circuit 22 can control so that the accuracy of the information of the image acquired in the first light receiving area 12 is not reduced by multiplying the image of the interference fringes by the gain and performing subtraction.

[0120] As another example, at the moment of acquiring the infrared light image, the first processing circuit 22 may perform mask processing based on the interference fringes on the area used for authentication processing. Similarly, mask processing may be performed not by deleting information but by acquiring a coefficient of the intensity of light based on the interference fringes and multiplying each pixel by the coefficient to suppress the influence of the interference fringes.

[0121] For example, after acquiring the region of the object on which authentication is to be performed from the depth image acquired by the second processing circuit 23 , the solid-state imaging device 10 may perform interference fringe suppression processing in the region of the object in the first processing circuit 22 .

[0122] As described above, according to the present embodiment, the influence of interference or the like caused when the display 100 generates an image in the light source and the light receiving area provided below the display 100 can be reduced.

[0123] Removal of interference fringes is not limited to the above-described method, and a method capable of appropriately removing interference fringes on a plane acquired in advance may be used.

[0124] In addition, information about interference fringes in a three-dimensional image that is closer to the authentication target than a plane can be acquired in advance. For example, in the case of performing authentication processing using a person's face, the solid-state imaging device 10 can acquire information about interference fringes in advance using a spherical body, a plaster image with less unevenness, etc. Then, the authentication processing can be implemented using the information of interference fringes acquired in advance.

[0125] It should be noted that in the above description, the interference pattern in the infrared light image is acquired, but the present disclosure is not limited to this. For example, the solid-state imaging device 10 may acquire a pattern of interference fringes corresponding to the depth image in advance. The pattern may include information taking into account the arrival time delay of the reflected light from the plane or another solid body on the display 100 to each pixel (such as ToF) arranged below the display 100. Therefore, similarly, in the depth image, the occurrence of errors caused by the display 100 at the time of authentication can be suppressed.

[0126] (Fifth Embodiment)

[0127] Fig.121 is a diagram schematically showing an electronic device 1 according to an embodiment. The solid-state imaging device 10 includes a light shielding wall 16 between a light source 11 and a light receiving area.

[0128] The light shielding wall 16 is arranged so that the light emitted from the light source 11 does not directly reach each light receiving area in the electronic device 1. The light shielding wall 16 may include a material that does not transmit at least the infrared light band emitted by the light source 11.

[0129] By providing the light shielding wall 16 , it is possible to realize an authentication process in which the influence of reflection or the like in the electronic device 1 and the solid-state imaging device 10 is suppressed.

[0130] Note that, in each embodiment, the first light receiving area 12 and the second light receiving area 13 are desirably arranged at close positions to reduce parallax. Figure 6 In the figure, the first light source 110 and the second light source 112 are adjacent to each other, and the light receiving area is arranged on the outside thereof, but the present disclosure is not limited thereto. For example, the first light source 110 and the second light source 112 may be arranged outside the corresponding light receiving area. That is, the first light source 110, the first light receiving area 12, the second light receiving area 13, and the second light source 112 may be arranged in sequence from the left side in the figure.

[0131] In addition, also in Figure 6 In the case of, a light shielding wall may be provided in each. For example, the first light source 110, the first light shielding wall, the first light receiving area 12, the second light receiving area 13, the second light shielding wall, and the second light source 112 may be provided in sequence from the left side of the figure. In this case, the light shielding wall may be further arranged between the first light receiving area 12 and the second light receiving area 13.

[0132] Fig.13 1 is an external view of the electronic device 1 according to the embodiment. The right figure shows a cross-sectional view taken along the arrow in the left figure.

[0133] The electronic device 1 includes a solid-state imaging device 10 below a display 100. The display 100 is provided with an opening 104, and light required for image generation in the visible light band and the infrared light band enters a light receiving area through the opening 104. The opening 104 may be provided on a display surface of a display portion 102 of the display 100, or may be provided in a frame portion 106 at an edge of the display 100 present in the electronic device 1.

[0134] For example, the opening 104 may have a pattern in which a light emitting pixel is not provided in a region and the region is filled with a substance that transmits light in a visible light band and light in an infrared light band to perform recognition processing.

[0135] Fig.14is an external view of the electronic device 1 according to the embodiment. Fig.13 , the right figure shows a cross-sectional view taken along the arrow in the left figure.

[0136] As described in the above embodiment, the electronic device 1 may include the solid-state imaging device 10 without including the opening 104. In this case, it is possible to have a configuration in which the light-emitting pixels included in the display 100 are thinned or appropriate light is transmitted together with the light-emitting pixels or a configuration around the light-emitting pixels as a configuration different from other regions. The position is not limited to the upper part of the drawing of the electronic device 1, and the solid-state imaging device 10 may be provided at any position in the display 100.

[0137] As described above, in the electronic device 1 , the solid-state imaging device 10 can be appropriately provided below the display 100 .

[0138] The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure can also be implemented as a device installed on any kind of mobile body (such as, a car, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, a robot, an engineering machine, or an agricultural machine (tractor)).

[0139] Fig.15 7000 is a block diagram showing an example of a schematic configuration of a vehicle control system 7000, which is an example of a mobile body control system to which the technology according to the embodiment of the present disclosure can be applied. The vehicle control system 7000 includes a plurality of electronic control units connected to each other via a communication network 7010. Fig.15 In the example shown, the vehicle control system 7000 includes a drive system control unit 7100, a body system control unit 7200, a battery control unit 7300, an external information detection unit 7400, an internal information detection unit 7500, and an integrated control unit 7600. The communication network 7010 that connects the plurality of control units to each other may be an in-vehicle communication network conforming to any standard, such as a controller area network (CAN), a local interconnect network (LIN), a local area network (LAN), FlexRay (registered trademark), and the like.

[0140] Each control unit includes: a microcomputer that performs calculation processing according to various programs; a storage unit that stores programs executed by the microcomputer, parameters for various operations, etc.; and a drive circuit that drives various control target devices. Each control unit also includes: a network interface (I / F) for performing communication with other control units via the communication network 7010; and a communication I / F for performing communication with devices, sensors, etc. inside and outside the vehicle through wired communication or radio communication. Fig.15The functional configuration of the integrated control unit 7600 shown includes a microcomputer 7610, a general communication I / F 7620, a dedicated communication I / F 7630, a positioning unit 7640, a beacon receiving unit 7650, an in-vehicle device I / F 7660, a sound / image output unit 7670, an in-vehicle network I / F 7680, and a storage unit 7690. Other control units also similarly include a microcomputer, a communication I / F, a storage unit, and the like.

[0141] The drive system control unit 7100 controls the operation of the devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 7100 is used as a control device to control: a device for generating a driving force for the vehicle, such as an internal combustion engine, a drive motor, etc.; a driving force transmission mechanism for transmitting the driving force to the wheels; a steering mechanism for adjusting the steering angle of the vehicle; a braking device for generating a braking force for the vehicle, etc. The drive system control unit 7100 may have the function of a control device for an anti-lock braking system (ABS), an electronic stability control (ESC), etc.

[0142] The drive system control unit 7100 is connected to a vehicle state detection unit 7110. The vehicle state detection unit 7110 includes, for example, at least one of the following items: a gyro sensor for detecting the angular velocity of the axial rotational motion of the vehicle body, an acceleration sensor for detecting the acceleration of the vehicle, and a sensor for detecting the operation amount of the accelerator pedal, the operation amount of the brake pedal, the steering angle of the steering wheel, the engine speed or the rotation speed of the wheel, etc. The drive system control unit 7100 performs calculation processing using the signal input from the vehicle state detection unit 7110 to control the internal combustion engine, the drive motor, the electric power steering device, the brake device, etc.

[0143] The body system control unit 7200 controls the operation of various devices installed on the body according to various programs. For example, the body system control unit 7200 is used as a control device to control: a keyless entry system, a smart key system, a power window device, or various lights such as headlights, reverse lights, brake lights, turn signals, fog lights, etc. In this case, the body system control unit 12020 can receive radio waves transmitted from a mobile device that replaces the key or signals from various switches as input. The body system control unit 7200 receives these input radio waves or signals to control the vehicle's door lock devices, power window devices, lights, etc.

[0144] The battery control unit 7300 controls the secondary battery 7310 used as a power source for driving the motor according to various programs. For example, the battery control unit 7300 receives information about battery temperature, battery output voltage, remaining battery power, etc. from a battery device including the secondary battery 7310. The battery control unit 7300 uses these signals to perform calculation processing, perform temperature adjustment control of the secondary battery 7310, or control a cooling device of the battery device, etc.

[0145] The vehicle exterior information detection unit 7400 detects information outside the vehicle including the vehicle control system 7000. For example, the vehicle exterior information detection unit 7400 is connected to at least one of the imaging unit 7410 and the vehicle exterior information detection unit 7420. The imaging unit 7410 includes at least one of a time-of-flight (ToF) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. The vehicle exterior information detection unit 7420 may include at least one of the following items: an environmental sensor for detecting current atmospheric conditions or weather conditions, and a peripheral information detection sensor for detecting other vehicles, obstacles, pedestrians, etc. around the vehicle including the vehicle control system 7000.

[0146] The environmental sensor may be, for example, at least one of the following: a raindrop sensor for detecting rain, a fog sensor for detecting fog, a sunshine sensor for detecting the degree of sunshine, and a snow sensor for detecting snowfall. The peripheral information detection sensor may be at least one of the following: an ultrasonic sensor, a radar device, and a LIDAR device (a light detection and ranging device, or a laser imaging detection and ranging device). Each of the imaging unit 7410 and the vehicle exterior information detection unit 7420 may be configured as an independent sensor or device, or may be configured as a device in which a plurality of sensors or devices are integrated.

[0147] Fig.16 An example of the installation position of the imaging unit 7410 and the vehicle exterior information detection unit 7420 is shown. The imaging units 7910, 7912, 7914, 7916 and 7918 can be arranged at the positions of the front nose, side mirrors, rear bumper, rear door and the upper part of the windshield inside the vehicle 7900. The imaging unit 7910 arranged at the front nose and the imaging unit 7918 arranged at the upper part of the windshield inside the vehicle mainly obtain the image in front of the vehicle 7900. The imaging units 7912 and 7914 arranged at the side mirrors mainly obtain the image of the side of the vehicle 7900. The imaging unit 7916 arranged at the rear bumper or rear door mainly obtains the image of the rear of the vehicle 7900. The imaging unit 7918 arranged at the upper part of the windshield inside the vehicle is mainly used to detect the front vehicle, pedestrians, obstacles, signals, traffic signs, lanes, etc.

[0148] By the way, Fig.16An example of the shooting range of each imaging unit 7910, 7912, 7914 and 7916 is shown. Imaging range a represents the imaging range of the imaging unit 7910 arranged at the front nose. Imaging ranges b and c respectively represent the imaging ranges of the imaging units 7912 and 7914 arranged at the side mirrors. Imaging range d represents the imaging range of the imaging unit 7916 arranged at the rear bumper or the rear door. For example, a bird's-eye view image of the vehicle 7900 observed from above can be obtained by superimposing image data imaged by the imaging units 7910, 7912, 7914 and 7916.

[0149] The vehicle exterior information detection units 7920, 7922, 7924, 7926, 7928, and 7930 arranged at the front, rear, side, and corner of the vehicle 7900 and the upper portion of the windshield inside the vehicle may be ultrasonic sensors or radar devices. The vehicle exterior information detection units 7920, 7926, and 7930 arranged at the front nose of the vehicle 7900, the rear bumper of the vehicle 7900, the rear door, and the upper portion of the windshield inside the vehicle may be LIDAR devices. These vehicle exterior information detection units 7920-7930 are mainly used to detect vehicles ahead, pedestrians, obstacles, and the like.

[0150] Back to Fig.15 , continue to describe. The vehicle exterior information detection unit 7400 causes the imaging unit 7410 to image the image outside the vehicle and receive the imaged image data. In addition, the vehicle exterior information detection unit 7400 receives detection information from the vehicle exterior information detection unit 7420 connected to the vehicle exterior information detection unit 7400. When the vehicle exterior information detection unit 7420 is an ultrasonic sensor, a radar device, or a LIDAR device, the vehicle exterior information detection unit 7400 causes ultrasonic waves, electromagnetic waves, etc. to be transmitted, and receives information about the received reflected waves. Based on the received information, the vehicle exterior information detection unit 7400 may perform processing of detecting objects (such as people, vehicles, obstacles, signs, symbols, etc. on the road surface), or perform processing of detecting the distance of the object. The vehicle exterior information detection unit 7400 may perform environmental recognition processing based on the received information to identify rainfall, fog, road conditions, etc. The vehicle exterior information detection unit 7400 may calculate the distance to the object outside the vehicle based on the received information.

[0151] In addition, based on the received image data, the vehicle exterior information detection unit 7400 may perform image recognition processing for identifying an object (such as a person, vehicle, obstacle, sign, symbol, etc. on the road), or perform processing for detecting the distance of the object. The vehicle exterior information detection unit 7400 may perform processing such as distortion correction, alignment, etc. on the received image data, and generate a bird's-eye view image or a panoramic image by combining image data imaged by a plurality of different imaging units 7410. The vehicle exterior information detection unit 7400 may perform viewpoint conversion processing using image data imaged by different imaging units 7410.

[0152] The in-vehicle information detection unit 7500 detects information inside the vehicle. The in-vehicle information detection unit 7500 may be connected to a driver state detection unit 7510 that detects the driver's state. The driver state detection unit 7510 may include a camera for photographing the driver, a biosensor for detecting the driver's biometric information, a microphone for collecting sounds inside the vehicle, and the like. The biosensor may be arranged on a seat surface, a steering wheel, and the like, and detects biometric information of a passenger sitting in a seat or a driver holding the steering wheel. Based on the detection information input from the driver state detection unit 7510, the in-vehicle information detection unit 7500 may calculate the driver's fatigue level or the driver's concentration level, or may distinguish whether the driver is dozing off. The in-vehicle information detection unit 7500 may perform processing such as noise elimination processing on the audio signal obtained by sound collection.

[0153] The integrated control unit 7600 controls the overall operation within the vehicle control system 7000 according to various programs. The integrated control unit 7600 is connected to the input unit 7800. The input unit 7800 is a device that can be operated by the passenger, such as a touch panel, a button, a microphone, a switch, a joystick, etc. The integrated control unit 7600 can receive data obtained by voice recognition of the voice input via the microphone. The input unit 7800 can be a remote control device using infrared or other radio waves, or can be an external connection device such as a mobile phone, a personal digital assistant (PDA), etc. that supports the operation of the vehicle control system 7000. The input unit 7800 can be a camera. In this case, the passenger can input information through gestures. Alternatively, data obtained by detecting the movement of a wearable device worn by the passenger can be input. In addition, the input unit 7800 may include an input control circuit, etc., which generates an input signal based on the information input by the passenger, etc. using the above-mentioned input unit 7800, and outputs the generated input signal to the integrated control unit 7600. Passengers and the like can input various data into the vehicle control system 7000 through the operation input unit 7800 to process operation instructions.

[0154] The storage unit 7690 may include a read-only memory (ROM) storing various programs executed by the microcomputer and a random access memory (RAM) storing various parameters, operation results, sensor values, etc. In addition, the storage unit 7690 may be a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, etc.

[0155] The general communication I / F 7620 is a widely used communication I / F that mediates communication with various devices existing in the external environment 7750. The general communication I / F 7620 can implement: cellular communication protocols such as Global System for Mobile Communications (GSM (registered trademark)), Worldwide Interoperability for Microwave Access (WiMAX (registered trademark)), Long Term Evolution (LTE (registered trademark)), LTE Advanced (LTE-A), etc., or other wireless communication protocols such as wireless LAN (also known as Wireless Fidelity (Wi-Fi (registered trademark)), Bluetooth (registered trademark), etc. The general communication I / F 7620 can be connected to a device (e.g., an application server or a control server) existing on an external network (e.g., the Internet, a cloud network, or a company-specific network) via a base station or an access point. In addition, the general communication I / F 7620 can be connected to a terminal (e.g., a terminal of a driver, a pedestrian, or a store, or a machine type communication (MTC) terminal) existing near the vehicle using a peer-to-peer (P2P) technology.

[0156] The dedicated communication I / F 7630 is a communication I / F that supports a communication protocol developed for vehicle use. The dedicated communication I / F 7630 can implement: a standard protocol, for example, Wireless Access in Vehicular Environments (WAVE) (which is a combination of the Institute of Electrical and Electronics Engineers (IEEE) 802.11p as a lower layer and IEEE 1609 as an upper layer), a dedicated short-range communication (DSRC), or a cellular communication protocol. The dedicated communication I / F 7630 generally performs V2X communication including one or more of the following concepts: communication between vehicles and vehicles (vehicle-to-vehicle), communication between roads and vehicles (vehicle-to-infrastructure), communication between vehicles and homes (vehicle-to-home), and communication between pedestrians and vehicles (vehicle-to-pedestrian).

[0157] The positioning unit 7640 can perform positioning by receiving a global navigation satellite system (GNSS) signal from a GNSS satellite (e.g., a GPS signal from a global positioning system (GPS) satellite) to generate position information including the latitude, longitude, and altitude of the vehicle. Incidentally, the positioning unit 7640 can identify the current position by exchanging signals with a wireless access point, or can obtain position information from a terminal such as a mobile phone, a personal handyphone system (PHS), or a smart phone with a positioning function.

[0158] The beacon receiving unit 7650 can receive radio waves or electromagnetic waves transmitted from a radio station installed on a road or the like, thereby obtaining information on the current position, congestion, road closure, required time, etc. Incidentally, the function of the beacon receiving unit 7650 may be included in the above-mentioned dedicated communication I / F 7630.

[0159] The in-vehicle device I / F 7660 is a communication interface for mediating the connection between the microcomputer 7610 and various in-vehicle devices 7760 present in the vehicle. The in-vehicle device I / F 7660 can establish a wireless connection using wireless communication protocols such as wireless LAN, Bluetooth (registered trademark), near field communication (NFC) or wireless universal serial bus (WUSB). In addition, the in-vehicle device I / F 7660 can establish a wired connection through a universal serial bus (USB), a high-definition multimedia interface (HDMI (registered trademark)), a mobile high-definition link (MHL), etc. via a connection terminal (and a cable, if necessary) not shown in the figure. The in-vehicle device 7760 may include at least one of the following items: a mobile device and a wearable device owned by a passenger and an information device loaded into or attached to the vehicle. The in-vehicle device 7760 may also include a navigation device for searching a path to an arbitrary destination. The in-vehicle device I / F 7660 exchanges control signals or data signals with these in-vehicle devices 7760.

[0160] The in-vehicle network I / F 7680 is an interface that mediates communication between the microcomputer 7610 and the communication network 7010. The in-vehicle network I / F 7680 transmits and receives signals and the like in accordance with a predetermined protocol supported by the communication network 7010.

[0161] The microcomputer 7610 of the integrated control unit 7600 controls the vehicle control system 7000 according to various programs based on information obtained via at least one of the general communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the vehicle network I / F 7680. For example, the microcomputer 7610 may calculate a control target value for a driving force generating device, a steering mechanism, or a braking device based on the obtained vehicle interior or vehicle exterior related information, and output a control command to the drive system control unit 7100. For example, the microcomputer 7610 may perform cooperative control aimed at realizing functions of an advanced driver assistance system (ADAS), including collision avoidance or impact buffering for a vehicle, following driving based on a vehicle-to-vehicle distance, vehicle speed keeping driving, a vehicle collision warning, a vehicle lane departure warning, and the like. In addition, the microcomputer 7610 can control the drive force generation device, the steering mechanism, the braking device based on the information obtained about the vehicle's surroundings, thereby performing cooperative control intended for automatic driving, etc. that does not depend on the driver's operation.

[0162] The microcomputer 7610 may generate three-dimensional distance information between the vehicle and objects such as surrounding structures and people based on information obtained via at least one of the general communication I / F 7620, the dedicated communication I / F 7630, the positioning unit 7640, the beacon receiving unit 7650, the in-vehicle device I / F 7660, and the vehicle network I / F 7680, and generate local map information including information about the surrounding environment in which the vehicle is currently located. In addition, the microcomputer 7610 may predict dangers such as collision of the vehicle, approach of pedestrians, etc., entering a closed road, etc. based on the obtained information, and generate an alarm signal. The alarm signal may be a signal for generating a warning sound or lighting an alarm lamp.

[0163] The sound / image output unit 7670 transmits an output signal of at least one of sound and image to an output device, which can visually or auditorily notify information to the passengers of the vehicle or the outside of the vehicle. Fig.15 In the example, the audio speaker 7710, the display unit 7720 and the instrument panel 7730 are shown as output devices. The display unit 7720 may include at least one of an on-board display and a head-up display. The display unit 7720 may have an augmented reality (AR) display function. The output device may be other devices other than these devices, such as headphones, wearable devices such as eyeglass displays worn by passengers, projectors, lights, etc. In the case where the output device is a display device, the display device visually displays the results obtained by various processes performed by the microcomputer 7610, or displays information received from other control units in various forms (such as text, images, tables, graphs, etc.). In addition, in the case where the output device is an audio output device, the audio output device converts an audio signal composed of played audio data or sound data into an analog signal and outputs the analog signal in an auditory manner.

[0164] By the way, in Fig.15 In the example shown, at least two control units connected to each other via the communication network 7010 may be integrated into one control unit. Alternatively, each individual control unit may include multiple control units. In addition, the vehicle control system 7000 may include other control units not shown in the figure. In addition, part or all of the functions performed by one of the control units in the above description may be assigned to another control unit. That is, a predetermined operation process may be performed by any control unit as long as the information is transmitted and received via the communication network 7010. Similarly, a sensor or device connected to one of the control units may be connected to another control unit, and multiple control units may transmit and receive detection information to each other via the communication network 7010.

[0165] It should be noted that the Figures 1 to 14The computer program for each function of the electronic device 1 or the solid-state imaging device 10 of the present embodiment described can be installed on any control unit or the like. In addition, a computer-readable recording medium in which such a computer program is stored can be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, etc. In addition, the above-mentioned computer program can be distributed via, for example, a network without using a recording medium.

[0166] In the above vehicle control system 7000, according to the reference Figures 1 to 14 The electronic device 1 or the solid-state imaging device 10 of the present embodiment described can be applied to a device that is accompanied by Fig.15 The application examples of these units shown in FIG. 7 are the vehicle exterior information detection unit 7400 , the vehicle interior information detection unit 7500 , and the imaging section 7410 , the vehicle exterior information detection section 7420 , or the driver state detection section 7510 .

[0167] In addition, reference Figures 1 to 12 At least some of the configuration elements of the electronic device 1 or the solid-state imaging device 10 described above may be used in Fig.15 The integrated control unit 7600 shown in FIG. 7 may be implemented in a module (eg, an integrated circuit module including a bare chip) or may be implemented by Fig.15 The vehicle control system 7000 shown in FIG. 7 is implemented by multiple control units.

[0168] The above-mentioned embodiments may have the following forms. (1)

[0170] A solid-state imaging device comprising:

[0171] a light source disposed on an opposite side of the display surface of the display and emitting light in an infrared light band via the display;

[0172] a first light receiving area disposed on an opposite side of a display surface of the display and including pixels receiving light in a visible light band and pixels receiving light in at least an infrared light band emitted from a light source; and

[0173] The second light receiving area is provided on the opposite side of the display surface of the display and includes pixels that receive at least light in the infrared light band emitted from the light source. (2)

[0175] The solid-state imaging device according to (1), further comprising:

[0176] a first processing circuit that generates an infrared light image based on the intensity of the infrared light received in the first light receiving area; and

[0177] The second processing circuit generates a depth image based on the intensity of the infrared light received in the second light receiving area. (3)

[0179] The solid-state imaging device according to (2), further comprising:

[0180] The third processing circuit determines whether the object is an authentication target based on the depth image, and performs authentication processing using at least the infrared light image when it is determined that the object is the authentication target. (4)

[0182] The solid-state imaging device according to any one of (2) to (3), wherein

[0183] The first light receiving area includes a dual band pass filter having a transmission characteristic in two bands of a visible light band and an infrared light band between the first light receiving area and a display surface of the display. (5)

[0185] The solid-state imaging device according to any one of (2) to (4), wherein

[0186] The second light receiving area includes a band pass filter having a transmission characteristic in an infrared light band between the second light receiving area and a display surface of the display. (6)

[0188] The solid-state imaging device according to any one of (2) to (5), wherein

[0189] The light source includes a surface light source. (7)

[0191] The solid-state imaging device according to any one of (2) to (5), wherein

[0192] Light sources include:

[0193] a surface light source that receives light in the first light receiving area; and

[0194] The point light source receives light in the second light receiving area. (8)

[0196] The solid-state imaging device according to any one of (2) to (7), further comprising

[0197] A light shielding wall is located between the light source and the first light receiving area and the second light receiving area on opposite sides of the display surface of the display. (9)

[0199] The solid-state imaging device according to (6) or (8) dependent on (6), further comprising

[0200] a control circuit for executing drive control of the first light receiving area, drive control of the second light receiving area, and control of the light source,

[0201] Among them, the control circuit issues an identifier,

[0202] Execute light emission control of the light source,

[0203] Associating the identifiers with the acquired infrared light image and the acquired depth image, respectively, performing extinction control of the light source, and

[0204] It is determined that the infrared light image and the depth image are images acquired at the same time when the identifiers match each other. (10)

[0206] The solid-state imaging device according to (7) or (8) dependent on (7), further comprising

[0207] a control circuit for executing drive control of the first light receiving area, drive control of the second light receiving area, and control of the light source,

[0208] Among them, the control circuit issues an identifier,

[0209] Perform light emission control of surface light sources,

[0210] Associating an identifier with the acquired infrared image,

[0211] Perform extinction control of surface light sources,

[0212] Execute the emission control of point light sources,

[0213] associating an identifier with the acquired depth image,

[0214] Perform extinction control of point light sources, and

[0215] In a case where the identifier associated with the infrared light image and the identifier associated with the depth image match each other, it is determined that the infrared light image and the depth image are images acquired at the same time. (11)

[0217] The solid-state imaging device according to any one of (2) to (10), wherein

[0218] The first processing circuit acquires an interference infrared light image in advance by emitting light from a light source onto a plane arranged at a distance from the object, and

[0219] The acquired infrared light image is corrected using the interferometric infrared light image. (12)

[0221] The solid-state imaging device according to any one of (2) to (11), wherein the second processing circuit acquires an interference depth image in advance by emitting light from a light source onto a plane set at a distance of the object, and

[0222] The acquired depth image is corrected using the interferometric depth image. (13)

[0224] The solid-state imaging device according to any one of (1) to (12), wherein

[0225] The pixels belonging to the first light receiving area and the pixels belonging to the second light receiving area are arranged in the same pixel array. (14)

[0227] An electronic device, comprising:

[0228] Display, which displays information;

[0229] a light source disposed on an opposite side of the display surface of the display and emitting light in an infrared light band via the display;

[0230] a first light receiving area disposed on an opposite side of a display surface of the display and including pixels receiving light in a visible light band and pixels receiving light in at least an infrared light band emitted from a light source;

[0231] a second light receiving area disposed on an opposite side of a display surface of the display and including pixels that receive at least light in an infrared light band emitted from the light source;

[0232] A processing circuit performs authentication processing using the infrared light image acquired by the pixels belonging to the first light receiving area and the depth image acquired by the pixels belonging to the second light receiving area. (15)

[0234] The electronic device according to (14), wherein

[0235] The processing circuit determines whether the object is an authentication target based on the depth image, and

[0236] In a case where it is determined that the object is an authentication target, authentication processing is performed using at least the infrared light image. (16)

[0238] A program causes a processor to execute the processing of the control circuit according to (9). (17)

[0240] A program causes a processor to execute the processing of the control circuit according to (10). (18)

[0242] A program causes a processor to execute the processing of the processing circuit according to (14) or (15). (19)

[0244] The electronic device according to (14) or (15), wherein

[0245] The electronic device includes at least one of a mobile terminal, a smart phone, a tablet terminal, a vehicle-mounted camera with a display, an authentication device with a display, or a monitoring camera with a display.

[0246] The aspects of the present disclosure are not limited to the above-mentioned embodiments, and include various conceivable variations. The effects of the present disclosure are not limited to the above-mentioned contents. The configuration elements in each embodiment may be appropriately combined and applied. That is, various additions, modifications, and partial deletions may be made without departing from the conceptual concept and gist of the present disclosure obtained from the contents defined in the claims and their equivalents, etc.

[0247] Reference Symbols List

[0248] 1 Electronic equipment

[0249] 10 Solid-state imaging device

[0250] 11 Light Source

[0251] 110 First Light Source

[0252] 112 Second Light Source

[0253] 12. First light receiving area

[0254] 13. Second light receiving area

[0255] 14 Dual Bandpass Filter

[0256] 15 Bandpass filter

[0257] 100 Display

[0258] 100A display surface

[0259] 100B Back

[0260] 102 Display unit

[0261] 104 Opening

[0262] 106 Border section

[0263] 20 Control Circuit

[0264] 21 Storage Circuit

[0265] 22. First processing circuit

[0266] 23. Second processing circuit

[0267] 24 Third processing circuit

Claims

1. A solid-state imaging device, comprising: a light source disposed on an opposite side of a display surface of the display and emitting light in an infrared light band via the display; a first light receiving area disposed on an opposite side of a display surface of the display and including pixels receiving light in a visible light band and pixels receiving light in at least an infrared light band emitted from the light source; as well as A second light receiving area is provided on the opposite side of the display surface of the display and includes pixels that receive at least light in the infrared light band emitted from the light source.

2. The solid-state imaging device according to claim 1, further comprising: a first processing circuit that generates an infrared light image based on the intensity of the infrared light received in the first light receiving area; as well as The second processing circuit generates a depth image based on the intensity of the infrared light received in the second light receiving area.

3. The solid-state imaging device according to claim 2, further comprising: The third processing circuit determines whether the object is an authentication target based on the depth image, and performs authentication processing using at least the infrared light image when it is determined that the object is the authentication target.

4. The solid-state imaging device according to claim 2, wherein: The first light receiving area includes a dual band pass filter having a transmission characteristic in two bands of a visible light band and an infrared light band between the first light receiving area and a display surface of the display.

5. The solid-state imaging device according to claim 2, wherein: The second light receiving area includes a band pass filter having a transmission characteristic in an infrared light band between the second light receiving area and a display surface of the display.

6. The solid-state imaging device according to claim 2, wherein: The light source includes a surface light source.

7. The solid-state imaging device according to claim 2, wherein: The light source comprises: a surface light source that receives light in the first light receiving area; and A point light source receives light in the second light receiving area.

8. The solid-state imaging device according to claim 2, further comprising: A light shielding wall is provided between the light source and the first light receiving area and the second light receiving area on the opposite side of the display surface of the display.

9. The solid-state imaging device according to claim 6, further comprising: a control circuit for executing drive control of the first light receiving area, drive control of the second light receiving area, and control of the light source, wherein the control circuit issues an identifier, executing light emission control of the light source, Associating the identifier with the acquired infrared image and the acquired depth image respectively, performing extinction control of the light source, and It is determined that the infrared light image and the depth image are images acquired at the same time when the identifiers match each other.

10. The solid-state imaging device according to claim 7, further comprising: a control circuit for executing drive control of the first light receiving area, drive control of the second light receiving area, and control of the light source, wherein the control circuit issues an identifier, executing light emission control of the surface light source, associating the identifier with the acquired infrared light image, performing extinction control of the surface light source, executing light emission control of the point light source, associating the identifier with the acquired depth image, performing extinction control of the point light source, and In a case where the identifier associated with the infrared light image and the identifier associated with the depth image match each other, it is determined that the infrared light image and the depth image are images acquired at the same time.

11. The solid-state imaging device according to claim 2, wherein The first processing circuit pre-acquires an interference infrared light image by emitting light from the light source onto a plane arranged at a distance from the object, and The acquired infrared light image is corrected using the interference infrared light image.

12. The solid-state imaging device according to claim 2, wherein: The second processing circuit pre-acquires an interference depth image by emitting light from the light source onto a plane arranged at a distance from the object, and The acquired depth image is corrected using the interferometric depth image.

13. The solid-state imaging device according to claim 1, wherein Pixels belonging to the first light receiving area and pixels belonging to the second light receiving area are arranged in the same pixel array.

14. An electronic device comprising: Display, which displays information; a light source disposed on an opposite side of a display surface of the display and emitting light in an infrared light band via the display; a first light receiving area disposed on an opposite side of a display surface of the display and including pixels receiving light in a visible light band and pixels receiving light in at least an infrared light band emitted from the light source; a second light receiving area disposed on an opposite side of a display surface of the display and including pixels that receive at least light in an infrared light band emitted from the light source; A processing circuit performs authentication processing using the infrared light image acquired by the pixels belonging to the first light receiving area and the depth image acquired by the pixels belonging to the second light receiving area.

15. The electronic device according to claim 14, wherein: The processing circuit determines whether the object is an authentication target based on the depth image, and In a case where it is determined that the object is the authentication target, authentication processing is performed using at least the infrared light image.

16. A program for causing a processor to execute the process of the control circuit according to claim 9.

17. A program for causing a processor to execute the process of the control circuit according to claim 10.

18. A program for causing a processor to execute the processing of the processing circuit according to claim 14.

19. The electronic device according to claim 14, wherein: The electronic device includes at least one of a mobile terminal, a smart phone, a tablet terminal, a vehicle-mounted camera with a display, an authentication device with a display, or a monitoring camera with a display.

Images