Imaging device, display device, and imaging system

Abstract

This technology relates to an imaging element, display device, and imaging system that increase the degree of freedom when installing an imaging device for an electronic side-view mirror. The imaging element includes multiple pixels that receive incident light from a subject without passing through an imaging lens or a pinhole, and outputs a detection signal indicating an output pixel value modulated by the incident angle of the incident light. The light-receiving surface is attached to face the side of the vehicle, and the average value of the center of gravity of the incident angle directionality, which indicates the directionality of the multiple pixels with respect to the incident angle of the incident light, is offset in one direction from the pixel center. This technology can be applied to, for example, electronic side-view mirrors.

Description

Technical Field

[0001] The present technology relates to an imaging device, a display device, and an imaging system, and more particularly, to an imaging device, a display device, and an imaging system suitable for forming an electronic side-view mirror of a vehicle. Background Art

[0002] Recently, vehicles equipped with electronic side-view mirrors instead of conventional side-view mirrors have become common. An electronic side-view mirror is a system in which a camera captures an image of a range similar to that reflected by a conventional side-view mirror and displays the obtained image on a monitor installed in the vehicle (for example, see Patent Documents 1 and 2).

[0003] Citation List

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2012-113605

[0006] Patent Document 2: JP 2016-523204 W Summary of the Invention

[0007] Problems to be solved by the present invention

[0008] In Patent Documents 1 and 2, the camera is mounted at a position similar to that of conventional door mirrors, and the camera protrudes to the right and left from the exterior of the vehicle body. Therefore, like conventional door mirrors, the camera may become an obstacle when passing through narrow places or may be damaged by contact, etc.

[0009] To address this issue, cameras installed inside the vehicle must be installed diagonally relative to the vehicle's side windows to capture images covering a range similar to that of the side mirrors. This creates a gap between the camera lens and the side windows, potentially causing reflections on the side windows.

[0010] On the other hand, if the camera lens is brought close to the side window to prevent reflection, the imaging direction is directed to the side of the vehicle, which makes it difficult to capture the view diagonally behind the vehicle.

[0011] The present technology is proposed in response to this situation and aims to increase the installation freedom of the imaging device used in the electronic side-view mirror.

[0012] Solutions to Problems

[0013] An imaging device according to a first aspect of the present technology includes: a plurality of pixels that receive incident light entering from an object after passing through neither an imaging lens nor a pinhole, and each pixel outputs a detection signal indicating an output pixel value modulated according to the incident angle of the incident light. The imaging device is attached to a vehicle so that a light-receiving surface faces one side of the vehicle, and an average value of a center of gravity of incident angle directivities indicating the directionality of the plurality of pixels with respect to the incident angle of the incident light deviates from the center of the pixel in one direction.

[0014] A display device according to a second aspect of the present technology is arranged in a vehicle at an oblique left side or a oblique right side in front of a driver, and includes: a display surface on which a plurality of display elements are arranged; and a plurality of pixels, which are arranged on the display surface, receive incident light entering from an object after passing through neither an imaging lens nor a pinhole, and each pixel outputs a detection signal indicating an output pixel value modulated according to the incident angle of the incident light.

[0015] An imaging system according to a third aspect of the present technology includes: an imaging device, the imaging device including a plurality of pixels, the plurality of pixels receiving incident light entering from an object after passing through neither an imaging lens nor a pinhole, and each pixel outputting a detection signal indicating an output pixel value modulated according to the incident angle of the incident light, the imaging device being attached to a vehicle so that a light receiving surface faces one side of the vehicle, wherein an average value of the center of gravity of the incident angle directivity indicating the directionality of the plurality of pixels with respect to the incident angle of the incident light deviates from the center of the pixel in one direction; and a display unit, the display unit displaying a restored image restored from the detection image based on the detection signals of the plurality of pixels.

[0016] In the first aspect of the present technology, imaging is performed in a direction deviating from the side of the vehicle.

[0017] In a second aspect of the present technology, an image is displayed by a plurality of display elements provided on a display surface, and a plurality of pixels provided on the display surface receive incident light that enters from a subject without passing through an imaging lens or a pinhole. Each pixel outputs a detection signal indicating an output pixel value modulated according to the incident angle of the incident light.

[0018] In the third aspect of the present technology, imaging is performed in an offset direction from the side of the vehicle, and a restored image restored from a detection image obtained as a result of the imaging is displayed. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a block diagram illustrating an example configuration of an in-vehicle system according to the present technology.

[0020] Figure 2 It shows Figure 1Block diagram showing an example configuration of an imaging unit of an in-vehicle system.

[0021] Figure 3 Is used to illustrate Figure 2 A diagram illustrating the imaging principle in an imaging device is shown.

[0022] Figure 4 It shows Figure 2 FIG. 1 is a diagram showing an example configuration of a pixel array unit of an imaging device.

[0023] Figure 5 Is used to illustrate Figure 2 A diagram of a first example configuration of an imaging device is shown.

[0024] Figure 6 Is used to illustrate Figure 2 A diagram of a second example configuration of an imaging device is shown.

[0025] Figure 7 This is a diagram for explaining the principle of generation of incident angle directivity.

[0026] Figure 8 This is a diagram for explaining changes in the directivity of the incident angle when using an on-chip lens.

[0027] Figure 9 A diagram for explaining the relationship between narrow-field-of-view pixels and wide-field-of-view pixels.

[0028] Figure 10 A diagram for explaining the relationship between narrow-field-of-view pixels and wide-field-of-view pixels.

[0029] Figure 11 A diagram for explaining the relationship between narrow-field-of-view pixels and wide-field-of-view pixels.

[0030] Figure 12 A diagram illustrating the difference in image quality between narrow-field-of-view pixels and wide-field-of-view pixels.

[0031] Figure 13 A diagram illustrating the difference in image quality between narrow-field-of-view pixels and wide-field-of-view pixels.

[0032] Figure 14 is a diagram for explaining an example combination of pixels having multiple fields of view.

[0033] Figure 15 It shows Figure 1 FIG. 1 is a block diagram of an example configuration of a driver identification unit of an in-vehicle system.

[0034] Figure 16 is a diagram illustrating an example hardware configuration of a camera module.

[0035] Figure 17is a diagram illustrating an installation example of a camera module, a display unit, and an imaging unit.

[0036] Figure 18 is a diagram illustrating an example method for attaching a camera module.

[0037] Figure 19 It shows Figure 2 FIG. 1 is a diagram showing a first embodiment of a pixel array unit of an imaging device.

[0038] Figure 20 It shows Figure 19 A diagram showing an example of a light shielding pattern of a pixel.

[0039] Figure 21 This is a flowchart for explaining the first embodiment of the electronic side mirror display control process.

[0040] Figure 22 This is a flowchart for explaining the second embodiment of the electronic side mirror display control process.

[0041] Figure 23 A diagram for explaining a method for changing the display range of an electronic side mirror.

[0042] Figure 24 A diagram for explaining a method for changing the display range of an electronic side mirror.

[0043] Figure 25 A diagram for explaining a method for changing the display range of an electronic side mirror.

[0044] Figure 26 A diagram for explaining a method for changing the display range of an electronic side mirror.

[0045] Figure 27 A diagram for explaining a method for changing the display range of an electronic side mirror.

[0046] Figure 28 is a diagram showing an example of a warning display.

[0047] Figure 29 It shows Figure 2 FIG. 1 is a diagram showing a second embodiment of a pixel array unit of an imaging device.

[0048] Figure 30 It shows Figure 29 A diagram showing an example of a light shielding pattern of a pixel.

[0049] Figure 31 It is a diagram for explaining a method for changing the imaging range.

[0050] Figure 32 It is a diagram for explaining a method for changing the imaging range.

[0051] Figure 33 This is a flowchart for explaining the second embodiment of the electronic side mirror display control process.

[0052] Figure 34 is a diagram showing an example installation of a camera module.

[0053] Figure 35 is a diagram showing an installation example of a camera module and a display unit.

[0054] Figure 36 is a diagram showing a modification of the display unit.

[0055] Figure 37 is a diagram illustrating an example of providing an imaging device in a micro LED display.

[0056] Figure 38 is a diagram illustrating an example of providing an imaging device in a micro LED display.

[0057] Figure 39 is a diagram showing a modification of the imaging apparatus.

[0058] Figure 40 is a diagram showing a modification of the imaging apparatus.

[0059] Figure 41 is a diagram showing a modification of the imaging apparatus.

[0060] Figure 42 is a diagram showing a modification of the imaging apparatus.

[0061] Figure 43 is a diagram showing a modification of the imaging apparatus. DETAILED DESCRIPTION

[0062] The following is a detailed description of preferred embodiments of the present technology with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and explanation thereof will not be repeated.

[0063] Furthermore, explanation will be given in the following order.

[0064] 1. First embodiment

[0065] 2. Second embodiment

[0066] 3. Modification

[0067] 4. Other aspects

[0068] <<1. First embodiment>>

[0069] refer to Figures 1 to 28 , first, a first embodiment of the present technology is described.

[0070] <Example Configuration of In-Vehicle System 11>

[0071] Figure 1 is a block diagram showing an example configuration of the in-vehicle system 11 according to the present technology.

[0072] The in-vehicle system 11 is a system provided in the vehicle and performs electronic side mirror control, etc. The in-vehicle system 11 includes a camera module 21L, a camera module 21R, a communication unit 22, a driver identification unit 23, an alarm control unit 24, a display unit 25L, a display unit 25R, a display control unit 26, and a control unit 27. The camera module 21L, the camera module 21R, the communication unit 22, the driver identification unit 23, the alarm control unit 24, the display control unit 26, and the control unit 27 are connected to each other via a bus B1.

[0073] Note that in the following description, for ease of explanation, reference will not be made to the bus B1 in the case where each component of the in-vehicle system 11 performs data transmission / reception, etc. via the bus B1. For example, the case where the control unit 27 supplies data to the communication unit 22 via the bus B1 will be simply described as the case where the control unit 27 supplies data to the communication unit 22.

[0074] As described below, the camera module 21L performs processing such as imaging and image recognition on the left and rear sides of the vehicle. The camera module 21L includes an imaging unit 41L, a camera ECU 42L, and a micro control unit (MCU) 43L.

[0075] As described below, the imaging unit 41L includes a lensless camera (LLC) that uses neither an imaging lens nor a pinhole. The imaging unit 41L captures images of the left and rear sides of the vehicle and restores a restored image that forms an image of the target object based on the acquired detection images, as described below. The imaging unit 41L provides the restored image (hereinafter referred to as the left image) to the camera ECU 42L as a sensed image obtained by sensing the left and rear sides of the vehicle.

[0076] The camera ECU 42L performs image quality adjustment processing on the left image, such as gain adjustment, white balance adjustment, high dynamic range (HDR) processing, and traffic signal flicker correction processing. Note that the image quality adjustment processing is not necessarily performed by the camera ECU 42L but can be performed within the imaging unit 41L.

[0077] The camera ECU 42L also performs object recognition processing on the left image and detects dangerous objects (e.g., pedestrians, bicycles, motorcycles, or vehicles) on the left and rear sides of the vehicle. The camera ECU 42L provides the left image and data indicating the dangerous object detection result to the MCU 43L.

[0078] The MCU 43L converts data supplied from the camera ECU 42L into data in a communication format and outputs the data to the bus B 1. The MCU 43L also converts data received from the bus B 1 into data in a format for the camera ECU 42L and supplies the data to the camera ECU 42L.

[0079] As described below, the camera module 21R performs processing such as imaging and image recognition on the right and rear sides of the vehicle. The camera module 21R includes an imaging unit 41R, a camera ECU 42R, and a micro control unit (MCU) 43R.

[0080] The imaging unit 41R includes a lensless camera (LLC) that uses neither an imaging lens nor a pinhole, as described below. The imaging unit 41R captures images of the right and rear sides of the vehicle and restores a restored image that forms an image of the target object based on the acquired detection images, as described below. The imaging unit 41R provides the restored image (hereinafter referred to as the right side image) to the camera ECU 42R as a sensed image obtained by sensing the right and rear sides of the vehicle.

[0081] The camera ECU 42R performs image quality adjustment processing on the right image, such as gain adjustment, white balance adjustment, high dynamic range (HDR) processing, and traffic signal flicker correction processing. Note that the image quality adjustment processing is not necessarily performed by the camera ECU 42R but can be performed within the imaging unit 41R.

[0082] The camera ECU 42R also performs object recognition processing on the right image and detects dangerous objects (e.g., pedestrians, bicycles, motorcycles, or vehicles) on the right and rear sides of the vehicle. The camera ECU 42R provides the right image and data indicating the dangerous object detection result to the MCU 43R.

[0083] The MCU 43R converts data supplied from the camera ECU 42R into data in a communication format and outputs the data to the bus B 1. The MCU 43R also converts data received from the bus B 1 into data in a format for the camera ECU 42R and supplies the data to the camera ECU 42R.

[0084] Note that, in the case where no distinction is necessary, the camera module 21L and the camera module 21R will be simply referred to as the camera module 21 hereinafter. In the case where no distinction is necessary, the imaging unit 41L and the imaging unit 41R will be simply referred to as the imaging unit 41 hereinafter. In the case where no distinction is necessary, the camera ECU 42L and the camera ECU 42R will be simply referred to as the camera ECU 42 hereinafter. In the case where no distinction is necessary, the MCU 43L and the MCU 43R will be simply referred to as the MCU 43 hereinafter. In the case where no distinction is necessary, the left image and the right image will be simply referred to as the side image hereinafter.

[0085] In contrast, where it is necessary to distinguish between the components of the camera module 21L and the components of the camera module 21R, the letter “L” is appended to the reference numeral of each component of the camera module 21L, and the letter “R” is appended to the reference numeral of each component of the camera module 21R.

[0086] For example, the communication unit 22 sends / receives information to / from nearby vehicles, portable terminal devices carried by pedestrians, roadside devices, and external servers through various wireless communications such as vehicle-to-vehicle communication, vehicle-to-pedestrian communication, and road-to-vehicle communication.

[0087] The driver identification unit 23 identifies the state of the driver driving the vehicle (or monitors the driver) and outputs data indicating the identification result to the bus B1.

[0088] Based on the results of the camera ECU 42L detecting dangerous objects on the left and rear sides of the vehicle, the warning control unit 24 performs processing to superimpose a warning display urging attention to the dangerous object on the left image. The warning control unit 24 outputs the left image with the superimposed warning display to bus B1. Similarly, based on the results of the camera ECU 42R detecting dangerous objects on the right and rear sides of the vehicle, the warning control unit 24 performs processing to superimpose a warning display urging attention to the dangerous object on the right image. The warning control unit 24 outputs the right image with the superimposed warning display to bus B1.

[0089] Note that, in the case where any dangerous object is not detected, the alarm control unit 24 outputs the left image and the right image to the bus B1 without superimposing any warning display thereon.

[0090] The display unit 25L is formed of, for example, a display such as an organic EL display or a liquid crystal display, and displays the left image.

[0091] The display unit 25R is formed of, for example, a display such as an organic EL display or a liquid crystal display, and displays the right image.

[0092] Note that, in the case where no distinction is necessary, the display unit 25L and the display unit 25R will be simply referred to as the display unit 25 hereinafter.

[0093] The display control unit 26 controls the display processing performed by the display units 25L and 25R. For example, the display control unit 26 controls the display range of the left image displayed by the display unit 25L. Furthermore, for example, the display control unit 26 controls the display of the warning display by controlling the display of the left image displayed by the display unit 25, on which the warning display is superimposed. Similarly, the display control unit 26 controls, for example, the display range of the right image displayed by the display unit 25R. Furthermore, the display control unit 26 controls the display of the warning display by controlling the display of the right image displayed by the display unit 25, on which the warning display is superimposed.

[0094] The control unit 27 includes various processors, for example, to control each component of the in-vehicle system 11 and perform various processes.

[0095] <Example Configuration of Imaging Unit 41>

[0096] Figure 2 is a block diagram illustrating an example configuration of the imaging unit 41 of the camera module 21 .

[0097] The imaging unit 41 includes an imaging device 121, a restoration unit 122, a control unit 123, a storage unit 124, and a communication unit 125. The restoration unit 122, the control unit 123, the storage unit 124, and the communication unit 125 constitute a signal processing control unit 111 that performs signal processing, control of the imaging unit 41, and the like. Note that the imaging unit 41 does not include any imaging lens (no imaging lens).

[0098] Furthermore, the imaging device 121, the restoration unit 122, the control unit 123, the storage unit 124, and the communication unit 125 are connected to one another via the bus B2, and transmit / receive data, etc., via the bus B2. Note that in the following description, for ease of explanation, reference will be made to the bus B2 in the case where each component of the imaging unit 41 performs data transmission / reception, etc., via the bus B2. For example, a case where the communication unit 125 supplies data to the control unit 123 via the bus B2 will be described as a case where the communication unit 125 supplies data to the control unit 123.

[0099] The imaging device 121 is an imaging device in which the detection sensitivity of each pixel has an incident angle directivity, and outputs an image including a detection signal indicating the detection signal level corresponding to the amount of incident light to the restoration unit 122 or bus B2. The detection sensitivity of each pixel having an incident angle directivity means that the light reception sensitivity characteristics corresponding to the incident angle of the incident light entering each pixel vary for each pixel. However, the light reception sensitivity characteristics of all pixels are not necessarily completely different, and some pixels may have the same light reception sensitivity characteristics.

[0100] More specifically, the imaging device 121 may have a basic structure similar to that of a general imaging device such as a complementary metal oxide semiconductor (CMOS) image sensor, for example. However, the configuration of each pixel of the pixel array unit constituting the imaging device 121 is different from that of a general imaging device, and is, for example, a configuration having incident angle directivity, as will be described later. Figures 4 to 6 Furthermore, the imaging device 121 has a light receiving sensitivity that varies (changes) with the incident angle of the incident light in each pixel, and has an incident angle directivity with respect to the incident angle of the incident light in each pixel.

[0101] Here, all objects are a set of point lights, i.e., light is emitted from each point light in all directions. For example, Figure 3 The object surface 102 of the object in the upper left corner is formed by point light sources PA to PC, and the point light sources PA to PC respectively emit multiple light beams with light intensities a to c to the surroundings. In addition, in the following description, the imaging device 121 includes pixels with different incident angle directivities at positions Pa to Pc (hereinafter referred to as pixels Pa to Pc).

[0102] In this case, if Figure 3 As shown in the upper left corner of FIG, light beams having the same light intensity emitted from the same point light source are caused to enter each pixel of the imaging device 121. For example, a light beam having light intensity a emitted from the point light source PA is caused to enter each pixel Pa to Pc of the imaging device 121. However, the light beams emitted from the same point light source are caused to enter each pixel at different incident angles. For example, the light beams from the point light source PA are caused to enter each pixel Pa to Pc at different incident angles.

[0103] On the other hand, because the incident angle directionality of pixels Pa through Pc differs from one another, a beam of light of the same intensity emitted from the same point light source is detected with different sensitivities in each pixel. Consequently, the beam of light of the same intensity is detected with different detection signal levels in each pixel. For example, the detection signal level for a beam of light intensity a emitted from point light source PA has different values ​​in each pixel Pa through Pc.

[0104] Furthermore, the light reception sensitivity level of each pixel with respect to the light beam from each point light source is determined by multiplying the light intensity of the light beam by a coefficient indicating the light reception sensitivity with respect to the incident angle of the light beam (which is the incident angle directivity). For example, the detection signal level of pixel Pa with respect to the light beam from point light source PA is determined by multiplying the light intensity a of the light beam from point light source PA by a coefficient indicating the incident angle directivity of pixel Pa with respect to the incident angle of the light beam entering pixel Pa.

[0105] Therefore, the detection signal levels DA, DB, and DC of the pixels Pc, Pb, and Pa are respectively expressed by equations (1) to (3) shown below.

[0106] DA=α1×a+β1×b+γ1×c...(1)

[0107] DB=α2×a+β2×b+γ2×c...(2)

[0108] DC=α3×a+β3×b+γ3×c...(3)

[0109] Here, coefficient α1 is a coefficient representing the incident angle directivity of pixel Pc with respect to the incident angle of the light beam from point light source PA to pixel Pc, and is set according to the incident angle. In addition, α1×a represents the detection signal level of pixel Pc with respect to the light beam from point light source PA.

[0110] The coefficient β1 is a coefficient representing the incident angle directivity of the pixel Pc with respect to the incident angle of the light beam from the point light source PB to the pixel Pc, and is set according to the incident angle. In addition, β1×b represents the detection signal level of the pixel Pc with respect to the light beam from the point light source PB.

[0111] The coefficient γ1 is a coefficient representing the incident angle directivity of the pixel Pc with respect to the incident angle of the light beam from the point light source Pc to the pixel Pc, and is set according to the incident angle. In addition, γ1×c represents the detection signal level of the pixel Pc with respect to the light beam from the point light source Pc.

[0112] As described above, the detection signal level DA of the pixel Pa is determined by the sum of the products of the respective light intensities a, b and c of the light beams from the point light sources PA, PB and PC in the pixel Pc and the coefficients α1, β1 and γ1 representing the incident angle directivities depending on the respective incident angles.

[0113] Similarly, the detection signal level DB of pixel Pb is determined by the sum of the products of the individual light intensities a, b, and c of the light beams from point light sources PA, PB, and PC in pixel Pb and the coefficients α2, β2, and γ2 representing the incident angle directivity depending on the respective incident angles, as shown in equation (2). In addition, the detection signal level DC of pixel Pc is determined by the sum of the products of the individual light intensities a, b, and c of the light beams from point light sources PA, PB, and PC in pixel Pa and the coefficients α2, β2, and γ2 representing the incident angle directivity depending on the respective incident angles, as shown in equation (3).

[0114] However, the detection signal levels DA, DB, and DC of the pixels Pa, Pb, and Pc are mixed with the light intensities a, b, and c of the light beams emitted from the point light sources PA, PB, and PC, respectively, as shown in equations (1) to (3). Figure 3 As shown in the upper right corner of , the detection signal level in the imaging device 121 is different from the light intensity of each point light source on the object surface 102. Therefore, the image obtained by the imaging device 121 is different from the image in which the image of the object surface 102 is formed.

[0115] At the same time, by creating simultaneous equations formed by equations (1) to (3) and solving the created simultaneous equations, the light intensities a to c of the light beams of the respective point light sources PA to PC are determined. Then, pixels having pixel values ​​corresponding to the obtained light intensities a to c are arranged according to the layout (relative positions) of the point light sources PA to PC so as to restore a restored image in which an image of the object surface 102 is formed, as shown in FIG. Figure 3 shown in the lower right corner.

[0116] In this way, it is possible to obtain the imaging device 121 having incident angle directivity in each pixel without requiring any imaging lens and any pinhole.

[0117] In the following description, a coefficient set (e.g., coefficients α1, β1, and γ1) of each equation forming the simultaneous equations will be referred to as a coefficient set. In the following description, a group formed by a plurality of coefficient sets corresponding to a plurality of equations included in the simultaneous equations (e.g., a coefficient set of α1, β1, and γ1, a coefficient set of α2, β2, and γ2, a coefficient set of α3, β3, and γ3) will be referred to as a coefficient set group.

[0118] Here, if the object distance from the object surface 102 to the light receiving surface of the imaging device 121 changes, the incident angle of the light beam from each point light source on the object surface 102 to the imaging device 121 changes, so a different coefficient set group is required for each object distance.

[0119] Therefore, in the imaging unit 41, coefficient sets for various distances from the imaging device 121 to the object surface (object distance) are prepared in advance, and simultaneous equations are created by switching the coefficient sets for each object distance, and the created simultaneous equations are solved. Therefore, a restored image of the object surface at various object distances can be obtained based on a single detection image. For example, after a detection image is captured and recorded once, the coefficient sets are switched according to the distance to the object surface, and the restored image is restored, so that a restored image of the object surface at the desired object distance can be generated.

[0120] Furthermore, even on the object surface 102 at the same object distance, if the number and layout of the point light sources to be set are different, the incident angles of the light beams from each point light source to the imaging device 121 will also be different. Therefore, in some cases, multiple coefficient sets may be required for the object surface 102 at the same object distance. In addition, the incident angle directionality of each pixel 121a needs to be set so that the independence of the above-mentioned simultaneous equations can be ensured.

[0121] Furthermore, the image to be output by the imaging device 121 is an image formed by the detection signal, in which Figure 3 , an image of the object is not formed as shown in the upper right corner, and therefore, the object cannot be visually recognized. That is, the detection image formed by the detection signal output from the imaging device 121 is a group of pixel signals, but is also an image from which the user cannot visually recognize the object (the object is visually unrecognizable).

[0122] In view of this, the image formed by the detection signal (wherein Figure 3 An image in which no object is formed as shown in the upper right corner of the image) or an image captured by the imaging device 121 will be referred to as a detection image hereinafter.

[0123] Note that all pixels do not need to have different incident angle directionality from each other, but some pixels may have the same incident angle directionality.

[0124] Return Reference Figure 2 For example, the recovery unit 122 obtains the data corresponding to Figure 3 The restoration unit 122 further creates simultaneous equations represented by the above equations (1) to (3) using the detection signal level of each pixel of the detection image output from the imaging device 121 and the acquired coefficient set. Then, the restoration unit 122 solves the created simultaneous equations to obtain the equations that form the image shown in FIG. Figure 3The pixel values ​​of the respective pixels of the image of the object shown in the lower right corner are shown in FIG.

[0125] The image restored from the detection image will be referred to as a restored image. However, in the case where the imaging device 121 has sensitivity only to light outside the visible wavelength band (e.g., ultraviolet rays), the restored image is not an image from which an object can be identified as in a normal image, but is also referred to as a restored image in this case.

[0126] In addition, a restored image, which is an image in which an image of an object is formed and which has not yet undergone color separation or synchronization processing such as demosaicing, will be referred to as a raw (RAW) image hereinafter, and the detection image captured by the imaging device 121 will be distinguished as an image that conforms to the color filter array, rather than a raw image.

[0127] Note that the number of pixels of the imaging device 121 and the number of pixels constituting the restored image are not necessarily the same.

[0128] Furthermore, the restoration unit 122 performs demosaicing, gamma correction, white balance adjustment, conversion to a predetermined compression format, etc. on the restored image as necessary. Then, the restoration unit 122 outputs the restored image to the bus B2.

[0129] The control unit 123 includes, for example, various processors for controlling each component of the imaging unit 41 and performing various processes.

[0130] The storage unit 124 includes one or more storage devices such as a read-only memory (ROM), a random access memory (RAM), and a flash memory, and stores, for example, programs and data to be used in the processing of the imaging unit 41. The storage unit 124 associates, for example, coefficient sets corresponding to the above-mentioned coefficients α1 to α3, β1 to β3, and γ1 to γ3 with various object distances and stores the coefficient sets. More specifically, the storage unit 124 stores, for example, coefficient sets including coefficients for each pixel 121 a of the imaging device 121 with respect to each point light source disposed on the object surface 102 for each object surface 102 at each object distance.

[0131] The communication unit 125 communicates with the camera ECU 42L through a predetermined communication method.

[0132] <First Example Configuration of Imaging Device 121>

[0133] Next, refer to Figure 4 and 5 describe Figure 2 A first example configuration of the imaging device 121 of the imaging unit 41 is shown.

[0134] Figure 4 1 shows a front view of a portion of the pixel array unit of the imaging device 121. Note that Figure 4 An example case where the number of pixels in the pixel array unit is 6×6 is shown. However, the number of pixels in the pixel array unit is not limited thereto. Figure 4 The illustrated example configuration of the pixel array unit is for explaining a first example configuration of the imaging device 121 , and an actual example configuration of the pixel array unit will be described later.

[0135] exist Figure 4 In the imaging device 121 shown, a light shielding film 121b, one of the modulation elements, is provided for each pixel 121a to cover a portion of the light receiving area (light receiving surface) of the photodiode, and optically modulates the incident light entering each pixel 121a according to the angle of incidence. The light shielding film 121b is then provided in different areas of each pixel 121a, so that, for example, the light receiving sensitivity with respect to the angle of incidence of the incident light varies for each pixel 121a, and each pixel 121a has a different incident angle directivity.

[0136] For example, in pixel 121a-1 and pixel 121a-2, the ranges to which the light receiving areas of the photodiodes are shielded by the light shielding films 121b-1 and 121b-2 are different (at least the shielding areas (positions) or shielding areas are different). Specifically, in pixel 121a-1, the light shielding film 121b-1 is provided so as to shield a portion of the left side portion of the light receiving area of ​​the photodiode by a predetermined width. On the other hand, in pixel 121a-2, the light shielding film 121b-2 is provided so as to shield a portion of the right side portion of the light receiving area by a predetermined width. Note that the width to which the light shielding film 121b-1 shields the light receiving area of ​​the photodiode and the width to which the light shielding film 121b-2 shields the light receiving area of ​​the photodiode may be different or may be the same. Similarly, in other pixels 121a, the light shielding films 121b are randomly arranged in the pixel array unit so as to shield different areas in the light receiving area for each pixel.

[0137] Figure 5 The top is a side cross-sectional view of a first example configuration of the imaging device 121, Figure 5 The middle portion of FIG. 1 is a top view of a first example configuration of the imaging device 121 . Figure 5 The side cross-section of the top is also Figure 5 The AB section of the middle part. In addition, Figure 5 An example circuit configuration of the imaging device 121 is shown at the bottom of FIG.

[0138] exist Figure 5In the top portion of the imaging device 121, incident light enters from the upper side of the figure toward the lower side. Adjacent pixels 121a-1 and 121a-2 are so-called back-illuminated pixels, having a wiring layer Z12 set as the lowest layer in the figure and a photoelectric conversion layer Z11 provided thereon.

[0139] Note that in the following description, when it is not necessary to distinguish pixels 121a-1 and 121a-2 from each other, the numbers at the end of each reference numeral will be omitted, and the pixel will be simply referred to as pixel 121a. In the following description, the numbers and letters at the end of the reference numerals may also be omitted for other components in this specification.

[0140] also, Figure 5 The side view and the top view of only two of the pixels constituting the pixel array unit of the imaging device 121 are shown; of course, more pixels 121 a are provided but are not shown in the drawings.

[0141] Pixels 121a-1 and 121a-2 also include photodiodes 121e-1 and 121e-2, respectively, as photoelectric conversion elements in the photoelectric conversion layer Z11. Furthermore, on-chip lenses 121c-1 and 121c-2, and color filters 121d-1 and 121d-2 are stacked in this order from the top on the photodiodes 121e-1 and 121e-2.

[0142] On-chip lenses 121c-1 and 121c-2 focus incident light onto photodiodes 121e-1 and 121e-2.

[0143] Color filters 121d-1 and 121d-2 are filters that transmit light of a specific wavelength (eg, red, green, blue, infrared, or white). Note that in the case of white, color filters 121d-1 and 121d-2 may be transparent filters or may not be provided.

[0144] In the photoelectric conversion layer Z11 of the pixels 121a-1 and 121a-2, for example, light shielding films 121g-1 to 121g-3 are formed at the boundaries between the respective pixels, and they prevent the incident light L from entering adjacent pixels and causing crosstalk, as shown in FIG. Figure 5 shown.

[0145] In addition, if Figure 5As shown in the top and middle portions of the image processing apparatus 121, the light-shielding films 121b-1 and 121b-2 shield a portion of the light-receiving surface S from light when viewed from above. On the light-receiving surface S of the photodiodes 121e-1 and 121e-2 in the pixels 121a-1 and 121a-2, different areas are shielded from light by the light-shielding films 121b-1 and 121b-2, thereby independently setting different incident angle directivities for each pixel. However, the areas to be shielded do not need to be different in all pixels 121a of the imaging device 121, and some pixels 121a may have the same areas shielded from light.

[0146] Note that Figure 5 As shown in the top of FIG, the light shielding film 121b-1 and the light shielding film 121g-1 are connected to each other and arranged in an L shape when viewed from the side. Similarly, the light shielding film 121b-2 and the light shielding film 121g-2 are connected to each other and arranged in an L shape when viewed from the side. In addition, the light shielding film 121b-1, the light shielding film 121b-2, and the light shielding films 121g-1 to 121g-3 are formed of metal, and are formed of, for example, tungsten (W), aluminum (Al), or an alloy of aluminum and copper (Cu). In addition, in the same process as the process of forming wiring in semiconductor processing, the light shielding film 121b-1, the light shielding film 121b-2, and the light shielding films 121g-1 to 121g-3 can be formed simultaneously using the same metal as the wiring.

[0147] Note that the thicknesses of the light-shielding film 121 b - 1 , the light-shielding film 121 b - 2 , and the light-shielding films 121 g - 1 to 121 g - 3 may differ depending on the position.

[0148] In addition, if Figure 5 As shown at the bottom, pixel 121a includes a photodiode 161 (corresponding to photodiode 121e), a transfer transistor 162, a floating diffusion (FD) unit 163, a selection transistor 164, an amplification transistor 165 and a reset transistor 166, and is connected to a current source 168 via a vertical signal line 167.

[0149] An anode electrode of the photodiode 161 is grounded, and a cathode electrode of the photodiode 161 is connected to a gate electrode of the amplification transistor 165 via the transfer transistor 162 .

[0150] The transfer transistor 162 is driven by the transfer signal TG. For example, when the transfer signal TG supplied to the gate electrode of the transfer transistor 162 switches to a high level, the transfer transistor 162 turns on. As a result, the charge accumulated in the photodiode 161 is transferred to the FD unit 163 via the transfer transistor 162.

[0151] The FD unit 163 is a floating diffusion region having a charge capacity C1 and is provided between the transfer transistor 162 and the amplifying transistor 165, and temporarily accumulates the charge transferred from the photodiode 161 via the transfer transistor 162. The FD unit 163 is a charge detection unit that converts the charge into a voltage, and the charge accumulated in the FD unit 163 is converted into a voltage at the amplifying transistor 165.

[0152] The selection transistor 164 is driven according to a selection signal SEL. When the selection signal SEL supplied to the gate electrode of the selection transistor 164 is switched to a high level, the selection transistor 164 is turned on to connect the amplification transistor 165 and the vertical signal line 167.

[0153] The amplifier transistor 165 functions as an input unit of a source follower (which is a readout circuit that reads out a signal obtained by photoelectric conversion performed at the photodiode 161), and outputs a detection signal (pixel signal) of a level corresponding to the charge accumulated in the FD unit 163 to the vertical signal line 167. That is, the drain terminal of the amplifier transistor 165 is connected to the power supply VDD, and its source terminal is connected to the vertical signal line 167 via the selection transistor 164 to form a source follower together with the current source 168 connected to one end of the vertical signal line 167. The value of the detection signal (output pixel value) is modulated according to the incident angle of incident light from the object, and has a characteristic (directivity) that varies with the incident angle (or has incident angle directionality).

[0154] The reset transistor 166 is driven according to the reset signal RST. For example, when the reset signal RST supplied to the gate electrode of the reset transistor 166 is switched to a high level, the charge accumulated in the FD unit 163 is discharged to the power supply VDD, and the FD unit 163 is reset.

[0155] Note that the shape of the light shielding film 121b of each pixel 121a is not limited to Figure 4 The example shown is not intended to be construed as such, but may have any suitable shape. Figure 4 A shape extending in the horizontal direction, an L-shape extending in the vertical and horizontal directions, a shape with a rectangular opening, etc.

[0156] <Second Example Configuration of Imaging Device 121>

[0157] Figure 6 is a diagram illustrating a second example configuration of the imaging device 121 . Figure 6 , which is a side cross-sectional view of a pixel 121 a of an imaging device 121 as a second example configuration, is shown at the top. Figure 6 The middle portion of FIG. 1 shows a top view of the imaging device 121 . Figure 6 The side cross-section of the top is also Figure 6 The AB section of the middle part. In addition, Figure 6 An example circuit configuration of the imaging device 121 is shown at the bottom of FIG.

[0158] Figure 6 The configuration of the imaging device 121 in Figure 5 The configuration of the imaging device 121 in FIG. 1 is different in that four photodiodes 121f-1 to 121f-4 are formed in one pixel 121a, and a light shielding film 121g is formed in a region separating the photodiodes 121f-1 to 121f-4 from each other. Figure 6 In the imaging device 121, the light shielding film 121g is formed into a cross shape when viewed from above. Figure 5 The same components as shown in Figure 5 Components in the same figure are denoted by the same reference numerals and will not be described in detail here.

[0159] exist Figure 6 In the imaging device 121 in FIG. 1 , the photodiodes 121f-1 to 121f-4 are separated by the light shielding film 121g, thereby preventing electrical and optical crosstalk between the photodiodes 121f-1 to 121f-4. Figure 5 Similar to the light shielding film 121g of the imaging device 121, Figure 6 The light shielding film 121g is used to prevent crosstalk, not to provide incident angle directivity.

[0160] In addition, Figure 6 In the imaging device 121 in FIG. 1 , one FD unit 163 is shared among the four photodiodes 121 f - 1 to 121 f - 4 . Figure 6 The bottom of FIG shows an example circuit configuration in which one FD unit 163 is shared among four photodiodes 121f-1 to 121f-4. Note that for Figure 6 The bottom of the Figure 5 The same components as those shown at the bottom are described.

[0161] Figure 6 The circuit configuration shown at the bottom is Figure 5 The circuit configuration shown at the bottom of FIG. 1 is different in that photodiodes 161-1 to 161-4 (corresponding to Figure 6 The photodiodes 121f-1 to 121f-4 at the top thereof and the transfer transistors 162-1 to 162-4 are used instead of the photodiodes 161 (corresponding to Figure 5 The photodiode 121e) and the transfer transistor 162 in the top, and the FD unit 163 are shared.

[0162] With this configuration, the charges accumulated in the photodiodes 121f-1 to 121f-4 are transferred to the common FD unit 163 having a predetermined capacity provided in the connection portion between the photodiodes 121f-1 to 121f-4 and the gate electrode of the amplifying transistor 165. Then, a signal corresponding to the charge level held in the FD unit 163 is read as a detection signal (pixel signal).

[0163] Thus, the charge accumulated in the photodiodes 121f-1 to 121f-4 can be selectively contributed to the output or detection signal of the pixel 121a in various combinations. That is, the charge can be read independently from each of the photodiodes 121f-1 to 121f-4, and the photodiodes 121f-1 to 121f-4 that contribute to the output (or the degree to which the photodiodes 121f-1 to 121f-4 contribute to the output) can be made different from each other. As a result, different incident angle directivities can be obtained.

[0164] For example, the charges in the photodiode 121f-1 and the photodiode 121f-3 are transferred to the FD unit 163, and the signals obtained by reading the corresponding charges are added, so that the incident angle directivity in the horizontal direction can be obtained. Similarly, the charges in the photodiode 121f-1 and the photodiode 121f-2 are transferred to the FD unit 163, and the signals obtained by reading the corresponding charges are added, so that the incident angle directivity in the vertical direction can be obtained.

[0165] Furthermore, a signal obtained based on the charges selectively read out independently from the four photodiodes 121 f - 1 to 121 f - 4 is a detection signal corresponding to one pixel of the detection image.

[0166] Note that, for example, the contribution of (the charge in) each photodiode 121f to the detection signal depends not only on whether the charge (detection value) in each photodiode 121f is transferred to the FD unit 163, but also on whether the charge accumulated in the photodiode 121f is reset before being transferred to the FD unit 163 using an electronic shutter function or the like. For example, if the charge in the photodiode 121f is reset immediately before being transferred to the FD unit 163, the photodiode 121f does not contribute to the detection signal at all. On the other hand, allowing time between resetting the charge in the photodiode 121f and transferring the charge to the FD unit 163 allows the photodiode 121f to partially contribute to the detection signal.

[0167] As mentioned above, in Figure 6In the case of the imaging device 121 in FIG. 1 , the combination to be used for detecting signals is changed between the four photodiodes 121f-1 to 121f-4 so that different incident angle directivities can be provided for each pixel. Figure 6 The detection signal output by each pixel 121a of the imaging device 121 has a value (output pixel value) modulated according to the incident angle of the incident light from the object, and has a characteristic (directivity) that changes with the incident angle (has incident angle directionality).

[0168] Note that in Figure 6 In the imaging device 121, incident light enters all photodiodes 121f-1 to 121f-4 without being optically modulated. Therefore, the detection signal is not a signal obtained by optical modulation. Meanwhile, the photodiode 121f that does not contribute to the detection signal will hereinafter also be referred to as the photodiode 121f that does not contribute to the pixel or its output.

[0169] also, Figure 6 An example is shown in which the light receiving surface of a pixel (pixel 121a) is divided into four equal areas and each photodiode 121f having a light receiving surface of the same size is arranged in the corresponding area, or an example in which the photodiode is divided into four equal parts. However, the number of divisions and the division positions of the photodiode can be set appropriately.

[0170] For example, a photodiode is not necessarily divided into equal parts, and the position of the photodiode divisions may vary for each pixel. Thus, for example, even if photodiodes 121f located in the same position in multiple pixels contribute to the output, the incident angle directivity varies between pixels. Furthermore, the number of divisions may vary between pixels, for example, to allow for more flexible setting of the incident angle directivity. Furthermore, for example, both the number of divisions and the position of the divisions may vary between pixels.

[0171] also, Figure 5 The imaging device 121 and Figure 6 The imaging devices 121 in both have a configuration in which each pixel can have an independently set incident angle directivity. Note that in Figure 5 In the imaging device 121 in FIG. 1 , the incident angle directivity of each pixel is set by the light shielding film 121b during manufacturing. On the other hand, in Figure 6 In the imaging device 121 in FIG. 1 , the number of divisions and the division position of the photodiode of each pixel are set at the time of manufacturing, but the incident angle directivity (the combination of photodiodes contributing to the output) of each pixel can be set at the time of use (for example, at the time of imaging). Note that in Figure 5 The imaging device 121 and Figure 6In both imaging devices 121, not all pixels necessarily need to have incident angle directivity.

[0172] Note that for Figure 5 In the imaging device 121, the shape of the light shielding film 121b of each pixel 121a will be referred to as a light shielding pattern hereinafter. Figure 6 In the imaging device 121 , the shape of a region of the photodiode 121 f that does not contribute to the output in each pixel 121 a will hereinafter be referred to as a light shielding pattern.

[0173] <Basic Characteristics, Etc. of Imaging Device 121>

[0174] Next, refer to Figures 7 to 14 The basic characteristics and the like of the imaging device 121 are described.

[0175] <Principle of generating incident angle directivity>

[0176] The incident angle directivity of each pixel of the imaging device 121 is determined by, for example Figure 7 The principle shown is generated. Note that Figure 7 The upper left and upper right parts are used to explain Figure 5 A diagram illustrating the principle of generating incident angle directivity in the imaging device 121 is shown. Figure 7 The lower left and lower right parts are used to explain Figure 6 A diagram illustrating the principle of generating incident angle directivity in the imaging device 121 is shown.

[0177] Figure 7 Each pixel in the upper left and upper right portions includes a photodiode 121e. On the other hand, Figure 7 Each pixel in the lower left and lower right parts includes two photodiodes 121f. Note that for ease of explanation, an example in which one pixel includes two photodiodes 121f is shown here. However, the number of photodiodes 121f included in one pixel may not be two.

[0178] exist Figure 7 In the pixel shown in the upper left portion of FIG, a light shielding film 121b-11 is formed to shield the right half of the light receiving surface of the photodiode 121e-11. Figure 7 In the pixel shown in the upper right portion of FIG, a light shielding film 121b-12 is formed to shield the left half of the light receiving surface of the photodiode 121e-12. Note that each dot-dash line in the figure is an auxiliary line that passes through the center of the light receiving surface of the photodiode 121e in the horizontal direction and is perpendicular to the light receiving surface.

[0179] For example, in Figure 7In the pixel shown in the upper left part of the diagram, the left half of the photodiode 121e-11 that is not shielded by the light-shielding film 121b-11 easily receives incident light from the upper right corner, which forms an incident angle θ1 with the dotted line in the diagram. On the other hand, the left half of the photodiode 121e-11 that is not shielded by the light-shielding film 121b-11 hardly receives incident light from the upper left corner, which forms an incident angle θ2 with the dotted line in the diagram. Therefore, Figure 7 The pixel shown in the upper left portion of has incident angle directivity with high light reception sensitivity to incident light from the upper right corner in the figure and low light reception sensitivity to incident light from the upper left corner.

[0180] At the same time, Figure 7 In the pixel shown in the upper right part of , for example, the left half of the photodiode 121e-12, which is shielded by the light-shielding film 121b-12, hardly receives the incident light from the upper right corner at an incident angle θ1. On the other hand, the right half of the photodiode 121e-12, which is not shielded by the light-shielding film 121b-12, easily receives the incident light from the upper left corner, which forms an incident angle θ2 with the dot-dash line. Therefore, Figure 7 The pixel shown in the upper right part of has incident angle directivity with low light reception sensitivity to incident light from the upper right corner in the figure and high light reception sensitivity to incident light from the upper left corner.

[0181] In addition, Figure 7 In the pixel shown in the lower left portion of FIG, photodiodes 121f-11 and 121f-12 are provided on the right and left sides in the figure, and one of the detection signals is read. Therefore, the pixel has incident angle directivity and does not have any light shielding film 121b.

[0182] Specifically, in Figure 7 In the pixel shown in the lower left part of the figure, only the signal of the photodiode 121f-11 set on the left side of the figure is read out. Therefore, the same Figure 7 . That is, incident light from the upper right corner, which forms an incident angle θ1 with the dotted line in the figure, enters photodiode 121f-11, and a signal corresponding to the amount of received light is read out from photodiode 121f-11. Therefore, this incident light contributes to the detection signal output from this pixel. On the other hand, incident light from the upper left corner, which forms an incident angle θ2 with the dotted line in the figure, enters photodiode 121f-12, but is not read out from photodiode 121f-12. Therefore, this incident light does not contribute to the detection signal output from this pixel.

[0183] Similarly, in Figure 7In the case where the pixel shown in the lower right portion includes two photodiodes 121f-13 and 121f-14, only the signal of the photodiode 121f-14 set on the right side of the figure is read out, so that the same Figure 7 . That is, incident light from the upper right corner, forming an incident angle θ1, enters photodiode 121f-13, but no signal is read out from photodiode 121f-13. Therefore, this incident light does not contribute to the detection signal output from this pixel. On the other hand, incident light from the upper left corner, forming an incident angle θ2, enters photodiode 121f-14, and a signal corresponding to the amount of received light is read out from photodiode 121f-14. Therefore, this incident light contributes to the detection signal output from this pixel.

[0184] Note that in Figure 7 In each pixel shown at the top of FIG, in the above example, the light-shielded area and the non-light-shielded area are divided at the center position in the horizontal direction of the pixel (the light-receiving surface of the photodiode 121e). However, these areas may be divided at some other position. Figure 7 In each pixel shown at the bottom, in the above example, two photodiodes 121f are split at the horizontal center of the pixel. However, the two photodiodes may be split at some other location. By changing the light-shielding region or the position where the photodiode 121f is split in the above manner, different incident angle directivities can be generated.

[0185] <Incident Angle Directivity in Configuration Including On-Chip Lenses>

[0186] Next, refer to Figure 8 The incident angle directivity in the configuration including the on-chip lens 121 c is described.

[0187] Figure 8 The top graphic shows Figure 8 The incident angle directionality of the pixel shown in the middle and bottom. Note that the horizontal axis represents the incident angle θ and the vertical axis represents the detection signal level. Note that the direction of the incident light is different from Figure 8 When the dotted line on the left side of the middle part of the Figure 8 The incident angle θ21 side on the left side of the middle part is the positive direction. Figure 8The incident angle θ22 on the right side of the middle portion is negative. Therefore, the incident angle of light entering on-chip lens 121c from the upper right corner is greater than the incident angle of light entering from the upper left corner. Specifically, when the direction of incident light is tilted significantly to the left (or the incident angle θ increases in the positive direction), the incident angle θ is larger, and when the direction of incident light is tilted significantly to the right (or the incident angle θ increases in the negative direction), the incident angle θ is smaller.

[0188] at the same time, Figure 8 The pixels shown in the center left portion of the Figure 7 The pixel shown in the upper left portion of FIG. is obtained by adding an on-chip lens 121c-11 that focuses incident light and a color filter 121d-11 that transmits light of a predetermined wavelength. That is, in this pixel, the on-chip lens 121c-11, the color filter 121d-11, the light shielding film 121b-11, and the photodiode 121e-11 are stacked in this order from the top of the figure in the direction of light incidence.

[0189] same, Figure 8 The pixels shown in the middle right part of the figure, Figure 8 The pixels shown in the lower left part and Figure 8 The pixels shown in the lower right part of Figure 7 The pixels shown in the upper right part of Figure 7 The pixels shown in the lower left part and Figure 7 The pixels shown in the lower right part of FIG are obtained by adding an on-chip lens 121c-11 and a color filter 121d-11 or an on-chip lens 121c-12 and a color filter 121d-12, respectively. Figure 8 In the pixel shown in the middle left part of Figure 8 As shown in the top solid-line waveform, the detection signal level (light reception sensitivity) of the photodiode 121e-11 varies with the incident angle θ of the incident light. Specifically, when the incident angle θ (i.e., the angle formed by the incident light relative to the dashed line in the diagram) is greater (or when the incident angle θ is greater in the positive direction (or tilted to the right in the diagram)), light is concentrated in the area where the light shielding film 121b-11 is not provided, and thus the detection signal level of the photodiode 121e-11 becomes higher. Conversely, when the incident angle θ of the incident light is smaller (or when the incident angle θ is greater in the negative direction (when tilted to the left in the diagram)), light is concentrated in the area where the light shielding film 121b-11 is provided, and accordingly, the detection signal level of the photodiode 121e-11 becomes lower.

[0190] In addition, Figure 8 In the pixel shown in the middle right part of Figure 8As shown in the top dashed waveform, the detection signal level (light reception sensitivity) of the photodiode 121e-12 changes with the incident angle θ of the incident light. Specifically, when the incident angle θ of the incident light is greater (or when the incident angle θ is greater in the positive direction), the light is concentrated in the area where the light shielding film 121b-12 is provided, and accordingly, the detection signal level of the photodiode 121e-12 becomes lower. Conversely, when the incident angle θ of the incident light is smaller (or when the incident angle θ is greater in the negative direction), the light is concentrated in the area where the light shielding film 121b-12 is not provided, and accordingly, the detection signal level of the photodiode 121e-12 becomes higher.

[0191] Figure 8 The solid and dotted waveforms shown at the top may vary depending on the area of ​​the light shielding film 121 b. Therefore, depending on the area of ​​the light shielding film 121 b, different incident angle directivities that vary for each pixel may be generated.

[0192] As mentioned above, the incident angle directivity is a characteristic of the light receiving sensitivity of each pixel that depends on the incident angle θ, but it can also be said that it depends on Figure 8 The characteristic of the light shielding value of the incident angle θ in each pixel in the middle part of FIG. That is, the light shielding film 121b blocks the incident light in a specific direction at a high level, but cannot sufficiently block the incident light from other directions. Figure 8 As shown at the top, the level change caused by this light shielding produces a detection signal level that varies with the incident angle θ. Therefore, if the direction in which light can be blocked at the highest level in each pixel is defined as the light shielding direction of each pixel, pixels having different incident angle directionalities mean pixels having different light shielding directions.

[0193] In addition, Figure 8 In the pixel shown in the lower left part of the figure, only the signal of the photodiode 121f-11 in the left part of the figure is used, so that the same Figure 8 The pixels shown in the middle left part have similar incident angle directionality, as Figure 7 . That is, when the incident angle θ of the incident light becomes larger (or when the incident angle θ becomes larger in the positive direction), light is concentrated in the area of ​​the photodiode 121f-11 from which the signal is to be read, and accordingly, the detection signal level becomes higher. Conversely, when the incident angle θ of the incident light is smaller (or when the incident angle θ is larger in the negative direction), light is concentrated in the area of ​​the photodiode 121f-12 from which the signal is not to be read, and accordingly, the detection signal level becomes lower.

[0194] In addition, similarly, Figure 8In the pixel shown in the lower right part of the figure, only the signal of the photodiode 121f-14 in the right part of the figure is used, so that the same Figure 8 The pixels shown in the middle right part have similar incident angle directivities, such as Figure 7 . That is, when the incident angle θ of the incident light is greater (or when the incident angle θ is greater in the positive direction), the light is concentrated in the area of ​​the photodiode 121f-13 that does not contribute to the output (detection signal), and accordingly, the level of the detection signal of each pixel becomes low. Conversely, when the incident angle θ of the incident light is smaller (or when the incident angle θ is greater in the negative direction), the light is concentrated in the area of ​​the photodiode 121f-14 that contributes to the output (detection signal), and accordingly, the level of the detection signal in each pixel becomes high.

[0195] Here, the center of gravity of the incident angle directivity of the pixel 121a is defined as follows.

[0196] The center of gravity of the incident angle directivity is the center of gravity of the light intensity distribution of the incident light entering the light receiving surface of the pixel 121a. The light receiving surface of the pixel 121a is Figure 8 The light receiving surface of the photodiode 121e in each pixel 121a is shown in the middle part of Figure 8 The light receiving surface of the photodiode 121f in each pixel 121a is shown at the bottom.

[0197] For example, Figure 8 The detection signal level on the vertical axis of the graph shown at the top is represented by a(θ), and a light beam having an incident angle θg calculated according to equation (4) shown below is a gravity-center light beam.

[0198] θg=Σ(a(θ)×θ) / Σa(θ)...(4)

[0199] Furthermore, the point where the centroidal light beam intersects the light receiving surface of the pixel 121 a is the centroid of the incident angle directivity of the pixel 121 a .

[0200] In addition, as in Figure 8 In the pixel shown at the bottom of the figure, in a pixel including multiple photodiodes to vary the photodiode contributing to the output, each photodiode has directivity with respect to the angle of incidence of incident light. An on-chip lens 121c is provided in each pixel to generate directivity at the angle of incidence in each pixel.

[0201] Note that in the following description, the main description will be similar to Figure 5An example of a pixel 121a that uses a light-shielding film 121b to achieve incident angle directivity like the pixel 121a shown. However, unless a light-shielding film 121b is required, pixels 121a that substantially divide a photodiode to obtain incident angle directivity may also be used.

[0202] <Relationship between the light-shielded area and the field of view>

[0203] Next, refer to Figure 9 and 14 to describe the relationship between the light-shielded area and the field of view of the pixel 121a.

[0204] For example, now describe a pixel 121a that is shielded by a light-shielding film 121b with a width d1 from each edge on all four sides as shown at the top of Figure 9 , and a pixel 121a' that is shielded by a light-shielding film 121b with a width d2 (> d1) from each edge on all four sides as shown at the bottom of Figure 9 .

[0205] Figure 10 Examples of the incident angles of incident light from the object surface 102 to the center position C1 of the imaging device 121 are shown. Note that Figure 10 shows examples of the incident angles of incident light in the horizontal direction, but similar incident angles are observed in the vertical direction. In addition, Figure 10 the right part of Figure 9 shows the pixels 121a and 121a' shown in

[0206] For example, when the pixel 121a shown in Figure 9 is arranged at the center position C1 of the imaging device 121, the range of the incident angle of incident light from the object surface 102 to the pixel 121a is represented by an angle A1 as shown in the left part of Figure 10 . Therefore, the pixel 121a can receive incident light with a width W1 of the object surface 102 in the horizontal direction.

[0207] On the other hand, when the pixel 121a' in Figure 9 is arranged at the center position C1 of the imaging device 121, as shown in the left part of Figure 10 , the range of the incident angle of incident light from the object surface 102 to the pixel 121a' is represented by an angle A2 (< A1) because the pixel 121a' has a wider light-shielded area than the pixel 121a. Therefore, the pixel 121a' can receive incident light with a width W2 (< W1) of the object surface 102 in the horizontal direction.

[0208] That is, the pixel 121a having a narrow light shielding area is a wide field pixel suitable for imaging a wide area on the object surface 102, and the pixel 121a' having a wide light shielding area is a narrow field pixel suitable for imaging a narrow area on the object surface 102. Note that the wide field pixel and narrow field pixel mentioned here are for comparison purposes only. Figure 9 , and is not limited to these pixels when comparing pixels having other fields of view.

[0209] Thus, for example, pixel 121a is used to restore Figure 9 The image I1 is shown. The image I1 is included as Figure 11 The image of the entire person H101 of the object shown at the top has a field of view SQ1 corresponding to the object width W1. On the other hand, for example, the pixel 121a' is used to restore Figure 9 The image I2 is shown in an enlarged manner. Figure 11 The top portion of the image shows an image of the face of the person H101 and the area surrounding the face, and has a field of view SQ2 corresponding to the object width W2.

[0210] At the same time, if Figure 11 As shown at the bottom of FIG. 1 , for example, the images in the area ZA surrounded by the dotted line in the imaging device 121 may be collected and arranged. Figure 9 , and a predetermined number of pixels 121a in the area ZB enclosed by a dashed line. Furthermore, for example, when restoring an image of field of view SQ1 corresponding to object width W1, the detection signals of the individual pixels 121a in area ZA are used to appropriately restore the image of field of view SQ1. On the other hand, when restoring an image of field of view SQ2 corresponding to object width W2, the detection signals of the individual pixels 121a′ in area ZB are used to appropriately restore the image of field of view SQ2.

[0211] Note that field of view SQ2 is smaller than field of view SQ1. Therefore, if the image of field of view SQ2 and the image of field of view SQ1 are to be restored with the same number of pixels, a restored image with higher image quality (higher resolution) can be obtained by restoring the image of field of view SQ2 than by restoring the image of field of view SQ1.

[0212] That is, in the case where a restored image is to be obtained with the same number of pixels, a restored image with higher image quality can be obtained by restoring an image with a smaller field of view.

[0213] For example, Figure 12 The right part shows Figure 11 An example configuration within the area ZA in the imaging device 121 is shown. Figure 12The left portion of shows an example configuration of pixels 121 a in the area ZA.

[0214] exist Figure 12 In FIG. 1 , the black area is the light shielding film 121 b, and for example, according to Figure 12 The rule shown in the middle left portion determines the light shielding area of ​​each pixel 121a.

[0215] Figure 12 The main light shielding portion Z101 in the left part ( Figure 12 The black portion in the left portion of the pixel 121a is the area that is shielded from light in each pixel 121a. Specifically, the main light-shielding portion Z101 is an area with a width of dx1 from each of the right and left sides of the pixel 121a toward the interior of the pixel 121a, and an area with a height of dy1 from each of the upper and lower sides of the pixel 121a toward the interior of the pixel 121a. In addition, in each pixel 121a, a rectangular opening Z111 that is not shielded by the light-shielding film 121b is provided in the area Z102 inside the main light-shielding portion Z101. Therefore, in each pixel 121a, the area other than the opening Z111 is shielded from light by the light-shielding film 121b.

[0216] Here, the openings Z111 of each pixel 121a are regularly arranged. Specifically, the horizontal position of the opening Z111 in each pixel 121a is the same among the pixels 121a in the same column in the vertical direction. In addition, the position of the opening Z111 in each pixel 121a is the same among the pixels 121a in the same row in the horizontal direction.

[0217] On the other hand, the horizontal position of opening Z111 in each pixel 121a shifts by a predetermined distance depending on the horizontal position of pixel 121a. That is, as the position of pixel 121a moves closer to the right, the left side of opening Z111 shifts to a position shifted rightward from the left side of pixel 121a by widths dx1, dx2, ..., and dxn. The distances between widths dx1 and dx2, dx2 and dx3, ..., and dxn-1 and dxn each have a value obtained by dividing the length obtained by subtracting the width of opening Z111 from the horizontal width of region Z102 by the number of pixels in the horizontal direction, n-1.

[0218] Furthermore, the vertical position of the opening Z111 in each pixel 121a is shifted by a predetermined distance depending on the vertical position of the pixel 121a. That is, as the position of the pixel 121a moves closer to the bottom, the upper side of the opening Z111 moves to a position shifted downward from the upper side of the pixel 121a by heights dy1, dy2, ..., and dyn. The distances between heights dy1 and dy2, dy2 and dy3, ..., and dyn-1 and dyn-1 each have a value obtained by dividing the length obtained by subtracting the height of the opening Z111 from the vertical height of the region Z102 by the number of pixels in the vertical direction, m-1.

[0219] Figure 13 The right part shows Figure 11 An example configuration within the area ZB in the imaging device 121 is shown. Figure 13 The left portion of shows an example configuration of pixels 121 a ′ in the region ZB.

[0220] exist Figure 13 In FIG. 1 , the black area is the light shielding film 121 b ′, and the light shielding area of ​​each pixel 121 a ′ is, for example, Figure 13 is determined by the rules shown in the left part of .

[0221] Figure 13 The main light shielding part Z151 in the left part ( Figure 13 The black portion in the left portion of the pixel 121a' is a region that is shielded from light in each pixel 121a'. Specifically, the main light-shielding portion Z151 is a region having a width dx1' from each of the right and left sides of the pixel 121a' toward the inside of the pixel 121a', and is a region having a height dy1' from each of the upper and lower sides of the pixel 121a' toward the inside of the pixel 121a'. In addition, in each pixel 121a', a rectangular opening Z161 that is not shielded by the light-shielding film 121b' is provided in the region Z152 on the inner side of the main light-shielding portion Z151. Therefore, in each pixel 121a', the region other than the opening Z161 is shielded from light by the light-shielding film 121b'.

[0222] Here, the openings Z161 of the respective pixels 121a' are regularly arranged, similar to Figure 12 . Specifically, the horizontal position of the opening Z161 in each pixel 121a′ is the same among the pixels 121a′ in the same column in the vertical direction. Furthermore, the position of the opening Z161 in each pixel 121a′ is the same among the pixels 121a′ in the same row in the horizontal direction.

[0223] On the other hand, the horizontal position of opening Z161 in each pixel 121a' shifts by a predetermined distance depending on the horizontal position of pixel 121a'. That is, as the position of pixel 121a' moves closer to the right, the left side of opening Z161 shifts to a position shifted rightward from the left side of pixel 121a' by widths dx1', dx2', ..., and dxn'. The distances between widths dx1' and dx2', dx2' and dx3', ..., and dxn-1' and dxn' all have values ​​obtained by dividing the length obtained by subtracting the width of opening Z161 from the horizontal width of region Z152 by the number of pixels in the horizontal direction, n-1.

[0224] In addition, the position of the opening Z161 in each pixel 121a' in the vertical direction is shifted by a predetermined distance according to the displacement of the pixel 121a' in the vertical direction. That is, when the position of the pixel 121a' becomes closer to the bottom, the upper side of the opening Z161 moves to a position shifted from the upper side of the pixel 121a' toward the bottom by heights dy1', dy2', ..., and dyn'. The distance between the height dy1' and the height dy2', the distance between the height dy2' and the height dy3', ..., and the distance between the height dyn-1' and the height dyn' all have a value obtained by dividing the length obtained by subtracting the height of the opening Z161 from the height of the area Z152 in the vertical direction by the number of pixels in the vertical direction, m-1.

[0225] Here, through Figure 12 The length obtained by subtracting the width of the opening Z111 from the width of the region Z102 in the horizontal direction in each pixel 121a shown in FIG. 1 is greater than the length obtained by subtracting the width of the opening Z111 from the width of the region Z102 in the horizontal direction in each pixel 121a shown in FIG. Figure 13 The width of the region Z152 in each pixel 121a′ shown in FIG is obtained by subtracting the width of the opening Z161 from the width of the horizontal direction. Figure 12 The stepwise difference between the widths dx1, dx2, ..., and dxn in Figure 13 The step difference between the widths dx1', dx2', ... and dxn' in.

[0226] In addition, through Figure 12 The length obtained by subtracting the height of the opening Z111 from the height of the region Z102 in each pixel 121a shown in the vertical direction is greater than the length obtained by subtracting the height of the opening Z111 from the height of the region Z102 in the vertical direction. Figure 13 The length of the region Z152 of each pixel 121a′ shown in FIG is obtained by subtracting the height of the opening Z161 from the height of the region Z152 in the vertical direction. Figure 12 The height difference between dy1, dy2, ... and dyn is greater than Figure 13The step difference between the heights dy1', dy2', ... and dyn' in .

[0227] As mentioned above, Figure 12 The step difference in the position of the opening Z111 of the light shielding film 121b of each pixel 121a in the horizontal direction and the vertical direction is Figure 13 The position of the opening Z161 of the light shielding film 121b' of each pixel 121a' shown in FIG is different in the horizontal and vertical directions. Then, in the restored image, the step difference is converted into a difference in object resolution (angular resolution). That is, Figure 13 The step difference of the position of the opening Z161 of the light shielding film 121b' of each pixel 121a' in the horizontal direction and the vertical direction is less than Figure 12 The difference in the horizontal and vertical positions of the opening Z111 of the light shielding film 121b of each pixel 121a is shown in FIG. Figure 12 The restored image of the detection signal of each pixel 121a is compared with the restored image using Figure 13 The restored image restored by the detection signals of the respective pixels 121 a ′ shown has higher object resolution and higher image quality (higher resolution).

[0228] Since the combination of the light shielding area of ​​the main light shielding portion and the opening area of ​​the opening is changed as described above, the imaging device 121 including pixels with various fields of view (or with various incident angle directivities) can be obtained.

[0229] Note that in the above example, the pixels 121a and the pixels 121a' are arranged in the region ZA and the region ZB, respectively. However, this is for convenience of explanation, and the pixels 121a corresponding to different fields of view are preferably arranged in the same region.

[0230] For example, Figure 14 As shown, four pixels formed by 2×2 pixels indicated by dotted lines are set as a unit U, and each unit U is formed by four pixels: a pixel 121a-W with a wide field of view, a pixel 121a-M with a medium field of view, a pixel 121a-N with a narrow field of view, and a pixel 121a-AN with a very narrow field of view.

[0231] In this case, or when the total number of pixels 121a is X, for example, a restored image can be restored using a detection image of X / 4 pixels for each of the four fields of view. At this stage, four coefficient sets that vary with the corresponding fields of view are used, and restored images having different fields of view are restored using four different simultaneous equations.

[0232] Therefore, by restoring the restored image using the detection image obtained from pixels suitable for imaging in the field of view of the restored image to be restored, appropriate restored images of the four fields of view can be obtained.

[0233] Furthermore, an image having fields of view around an intermediate field of view may be generated by interpolation from images having four fields of view, or pseudo optical zoom may be achieved by seamlessly generating images having various fields of view.

[0234] Note that, for example, when obtaining an image with a wide field of view as a restored image, all or some of the wide field of view pixels may be used. Furthermore, for example, when obtaining an image with a narrow field of view as a restored image, all or some of the narrow field of view pixels may be used.

[0235] <Example Configuration of Driver Identification Unit 23>

[0236] Figure 15 It shows Figure 1 FIG. 1 is a block diagram showing an example configuration of the driver identification unit 23. The driver identification unit 23 includes an imaging unit 181, an instruction motion detection unit 182, a learning unit 183, a storage unit 184, and a transmission unit 185.

[0237] The imaging unit 181 is installed at a position capable of capturing an image of the driver in the vehicle, captures an image of the driver, and supplies the obtained image (hereinafter referred to as the driver image) to the instruction motion detection unit 182 and the learning unit 183.

[0238] Based on the driver image and the detection conditions stored in the storage unit 184, the instruction motion detection unit 182 performs processing to detect a motion of the driver issuing an instruction to change the display range of the side image to be displayed on the display unit 25 (hereinafter, this motion will be referred to as a display change instruction motion). The display change instruction motion is performed by, for example, the driver moving one or more of the following body parts: a hand, a head, a line of sight, etc.

[0239] When the instruction motion detection unit 182 detects the display change instruction motion, the instruction motion detection unit 182 supplies data indicating the instruction content to the transmission unit 185. The instruction motion detection unit 182 also supplies data indicating the detection result of the display change instruction motion to the learning unit 183.

[0240] The learning unit 183 learns the detection conditions of the display change instruction movement based on the driver image, the detection result of the display change instruction movement, and the instruction input by the driver or the like via the input unit (not shown) to issue an instruction to change the display range of the side image to be displayed on the display unit 25. The learning unit 183 stores the detection conditions obtained by learning in the storage unit 184.

[0241] The transmission unit 185 converts the data indicating the content of the instruction supplied from the instruction motion detection unit 182 into data in a communication format, and outputs the data to the bus B1.

[0242] <Example Hardware Configuration of Camera Module 21>

[0243] Figure 16 An example hardware configuration of the camera module 21 is shown.

[0244] exist Figure 16 In the illustrated camera module 21 , two semiconductor chips, namely, an LLC chip 202 and a signal processing chip 203 , are mounted on the same substrate 201 .

[0245] LLC chip 202 includes Figure 1 The semiconductor chip of the imaging unit 41 is shown.

[0246] The signal processing chip 203 includes Figure 1 The camera ECU 42 and MCU 43 are semiconductor chips shown.

[0247] Since the LLC chip 202 and the signal processing chip 203 are arranged on the same substrate 201 as described above, a flexible substrate becomes unnecessary, and unnecessary radiation is reduced.

[0248] <Installation Example of Camera Module 21, Display Unit 25, and Imaging Unit 181>

[0249] Next, refer to Figure 17 and 18 An installation example of the camera module 21 , the display unit 25 , and the imaging unit 181 is described.

[0250] Figure 17 1 is a schematic diagram of the front half of a vehicle 221 equipped with the in-vehicle system 11 as viewed from above.

[0251] The camera module 21L is attached to a side window 232L of the passenger seat (left front seat) of the vehicle 221. The imaging direction AL2 passing through the center of the field of view (FOV) FL1 of the imaging device 121L of the camera module 21L is oriented diagonally rearward to the left side of the vehicle 221, so that the imaging device 121L can capture images of the left side and rear side of the vehicle 221.

[0252] The camera module 21R is attached to a side window 232R of the driver's seat (right front seat) of the vehicle 221. The imaging direction AR2 passing through the center of the field of view (FOV) FR1 of the imaging device 121R of the camera module 21R is oriented diagonally rearward to the right side of the vehicle 221, so that the imaging device 121R can capture images of the right side and rear side of the vehicle 221.

[0253] The display unit 25L is installed at a position diagonally to the left side of the driver 222, in front of the windshield 231 in the vehicle, near the left end of the instrument panel (not shown), and with the display surface slightly facing the direction of the driver 222 or diagonally rearward on the right side of the vehicle 221. The display unit 25L displays the left side image obtained by the camera module 21L.

[0254] The display unit 25R is installed at a position diagonally to the right side of the driver 222, in front of the windshield 231 in the vehicle, and near the right end of the instrument panel (not shown), with the display surface slightly facing the direction of the driver 222 or diagonally rearward toward the left side of the vehicle 221. The display unit 25R displays the right side image obtained by the camera module 21R.

[0255] Note that the display unit 25L and the camera module 21L are preferably installed so that the incident angle θL1 of the line of sight AL1 of the driver 222 with respect to the display surface of the display unit 25L and the angle θL2 of the imaging direction AL2 of the imaging device 121L with respect to the display surface of the display unit 25L become as close as possible. Similarly, the display unit 25R and the camera module 21R are preferably installed so that the incident angle θR1 of the line of sight AR1 of the driver 222 with respect to the display surface of the display unit 25R and the angle θR2 of the imaging direction AR2 of the imaging device 121R with respect to the display surface of the display unit 25R become as close as possible.

[0256] Note that the imaging direction indicates the average value of the incident angles of the centroidal light beams of the respective pixels 121 a of the imaging device 121, and passes through the center of the field of view of the imaging device 121. For example, the imaging direction AL2 indicates the average value of the incident angles of the centroidal light beams of the respective pixels 121 a of the imaging device 121L, and passes through the center of the field of view FL1 of the imaging device 121L. For example, the imaging direction AR2 indicates the average value of the incident angles of the centroidal light beams of the respective pixels 121 a of the imaging device 121R, and passes through the center of the field of view FR1 of the imaging device 121R.

[0257] With this arrangement, the relationship between the direction of the driver's 222 sight line and the display range of the side image displayed on each display unit 25 becomes natural, and the driver 222 can be prevented from feeling strange. In addition, the driver 222 can check the diagonal rear of the vehicle 221 without having to move his sight line to a great extent.

[0258] Figure 15 The imaging unit 181 of the driver recognition unit 23 in the vehicle is installed in front of the windshield 231 in the vehicle and near the center on the instrument panel (not shown). The imaging unit 181 captures an image near the driver 222.

[0259] Figure 18 23 is a schematic diagram of the side window 232L to which the camera module 21L is attached, as seen from above.

[0260] The camera module 21L is detachably attached using the bracket 251 so that the surface on which the LLC chip 202 is mounted extends along the surface of the side window 232L on the vehicle interior side. With this arrangement, the light receiving surface of the imaging device 121L provided on the surface of the LLC chip 202 faces toward and contacts or approaches the surface of the side window 232L on the vehicle interior side, while facing the left side of the vehicle 221 and becoming substantially parallel to the surface of the side window 232L on the vehicle interior side.

[0261] Therefore, the space between the light receiving surface of the imaging device 121 and the side window 232L disappears or becomes very narrow. As a result, reflection of incident light from the side window 232L and condensation between the light receiving surface of the imaging device 121 and the side window 232L are prevented.

[0262] The camera module 21L is also connected to the bus B1 of the in-vehicle system 11 via a cable 252 .

[0263] Note that, although not shown in any drawings or described in detail, the camera module 21R is also attached to the side window 232R using a bracket in a manner similar to the camera module 21L. With this arrangement, the light receiving surface of the imaging device 121R provided on the surface of the LLC chip 202 faces and contacts or approaches the surface of the side window 232R on the vehicle interior side, while facing the right side of the vehicle 221 and becoming substantially parallel to the surface of the side window 232R on the vehicle interior side.

[0264] Furthermore, in the vehicle 221, a space for accommodating the camera module 21L and the bracket 251 is formed in the storage portion that accommodates the side window 232L when the side window 232L is open. Similarly, in the vehicle 221, a space for accommodating the camera module 21R and the bracket is formed in the storage portion that accommodates the side window 232R when the side window 232R is open.

[0265] As described above, the camera module 21L and the camera module 21R are attached to the side windows 232L and 232R, respectively. Therefore, it is possible to prevent the imaging device for the electronic side mirror from occupying space and improve space efficiency in the vehicle.

[0266] Furthermore, since each camera module 21 is disposed inside the vehicle, it is possible to prevent the camera module 21 from blocking the passage or being damaged by contact with the outside of the vehicle, etc. Furthermore, there is no need to provide a drive mechanism or the like for opening and closing the camera module 21 to the left and right. Furthermore, imaging can be performed without being affected by environmental factors such as weather.

[0267] <First Embodiment of Pixel Array Unit of Imaging Device 121>

[0268] Next, refer to Figure 19 and 20 A first embodiment of the pixel array unit of the imaging device 121 is described.

[0269] Figure 19 A first embodiment of a light shielding pattern of a pixel array unit of an imaging device 121L of an imaging unit 41L of a camera module 21L is shown. Figure 20 Shown as a composition Figure 19 FIG. 1 is an example of a light shielding pattern of a pixel PLa of the first embodiment of a pixel 121 a of a pixel array unit shown.

[0270] The opening ALa of the light shielding film SLa of each pixel PLa is set within a rectangular opening setting area RLa indicated by a dotted line. Therefore, the area other than the opening setting area RLa of the light shielding film SLa of each pixel PLa serves as a main light shielding portion of the light shielding film SLa.

[0271] The size, shape, and position of the opening setting area RLa are the same between each pixel PLa. The height of the opening setting area RLa in the vertical direction is slightly smaller than the height of the pixel PLa, and its width in the horizontal direction is almost 1 / 2 of the width of the pixel PLa. In addition, the opening setting area RLa is set at the center of the pixel PLa in the vertical direction and at a position closer to the left in the horizontal direction. Therefore, the center of gravity of the opening setting area RLa is offset to the left from the center of the pixel PLa. In addition, as mentioned above, Figure 17 As described above, in a case where the camera module 21L is attached to the side window 232L, the center of gravity of the opening setting area RLa moves from the center of the pixel PLa toward the front of the vehicle 211 .

[0272] The shape and size of the rectangular opening ALa are the same between the pixels PLa. Figure 12 and 13A similar rule is used to form an aperture ALa in the aperture setting region RLa of each pixel PLa.

[0273] Specifically, the opening ALa is located at the left end of the opening setting area RLa in each pixel PLa in the left end column of the pixel array unit, and is located at the upper end of the opening setting area RLa in each pixel PLa in the upper end row of the pixel array unit. In addition, as the position of the pixel PLa becomes closer to the right, the opening ALa shifts rightward at equal intervals within the opening setting area RLa and is located at the right end of the opening setting area RLa in each pixel PLa in the right end column of the pixel array unit. In addition, as the position of the pixel PLa becomes closer to the bottom, the opening ALa shifts toward the bottom at equal intervals within the opening setting area RLa and is located at the lower end of the opening setting area RLa in each pixel PLa in the lower end row of the pixel array unit.

[0274] Therefore, in each pixel PLa in the same column in the vertical direction, the position of the opening ALa in the horizontal direction is the same. In addition, in each pixel PLa in the same row in the horizontal direction, the position of the opening ALa in the vertical direction is the same. Therefore, the position of the opening ALa in each pixel PLa (which is the position where the incident light enters each pixel PLa) varies with each pixel PLa, and as a result, the directionality of the incident angle of each pixel PLa is different from one another.

[0275] Furthermore, the openings ALa of each pixel PLa overlap the opening setting area RLa. That is, the area where the openings ALa of each pixel PLa overlap each other is equal to the opening setting area RLa. Note that the layout pattern of the openings ALa is not limited to the above configuration and can be any layout as long as the area where the openings ALa overlap each other is equal to the opening setting area RLa. For example, the openings ALa can be randomly arranged within the opening setting area RLa.

[0276] Here, the center of gravity of the incident angle directivity of each pixel PLa substantially coincides with the center of gravity of the aperture ALa of each pixel PLa and is offset to the left from the center of each pixel PLa in most pixels PLa. Therefore, the average value of the center of gravity of the incident angle directivity of each pixel PLa is offset to the left from the center of the pixel PLa. Furthermore, the average value of the incident angle of the centroid light beam in each pixel PLa is offset to the right (toward the rear of the vehicle 221) relative to the normal direction of the light receiving surface of the pixel array unit.

[0277] Therefore, even if the LLC chip 202 is installed parallel to the side window 232L and the light receiving surface of the pixel array unit of the imaging device 121L faces the left side of the vehicle 221, it is possible to Figure 17The imaging shown shows a left side view and a rear view of the vehicle 221 having a field of view FL1.

[0278] Note that, for example, the position of the aperture setting area RLa as the offset amount of its center of gravity relative to the center of the pixel PLa and the shape and size of the aperture setting area RLa are set based on the field of view FL1 in which imaging is to be performed.

[0279] Furthermore, even if the LLC chip 202 (or the light receiving surface of the imaging device 121L) is not directed toward the rear of the left side of the vehicle 221, the left side field of view and the rear field of view of the vehicle 221 can be imaged without any imaging lens. Figure 16 As described above, the LLC chip 202L and the signal processing chip 203L are mounted on the same substrate, and the mounting surface of the LLC chip 202L of the camera module 21L is in contact with or close to the side window 232L, so that the camera module 21L can be attached to the side window 232L.

[0280] Note that, although not shown in any drawings nor specifically described, the light shielding pattern of the pixel array unit of the imaging device 121R of the imaging unit 41R of the camera module 21R is also the same as that of the above referenced Figure 19 and 20 The aperture setting area RLa of each pixel PLa of the imaging device 121L is set in a manner similar to that in the example described above. However, in contrast to the aperture setting area RLa of each pixel PLa of the imaging device 121L, the center of gravity of the aperture setting area RRa of each pixel PRa of the imaging device 121R is offset from the center of the pixel PRa to the right side (the rear of the vehicle 221).

[0281] <First Embodiment of Electronic Side Mirror Display Control Processing>

[0282] Next, refer to Figure 21 The flowchart shown describes a first embodiment of the electronic side mirror display control process to be executed by the vehicle-mounted system 11.

[0283] For example, when the power of the vehicle 221 is turned on, the process starts, and when the power is turned off, the process ends.

[0284] Note that although a process of displaying the left image obtained by the camera module 21L on the display unit 25L is described below, a similar process is performed in the case of displaying the right image obtained by the camera module 21R on the display unit 25R.

[0285] In step S1, the imaging device 121L captures an image around the vehicle 221. As a result, a detection signal indicating a detection signal level corresponding to the amount of incident light from the object is output from each pixel PLa of the imaging device 121L having different incident angle directivities, and the imaging device 121L provides a detection image formed by the detection signals of the respective pixels PLa to the restoration unit 122L.

[0286] In step S2, the restoration unit 122L obtains coefficients to be used for image restoration. Specifically, the restoration unit 122L sets the distance to the object surface 102 to be restored, which is the object distance. Note that any method can be used as a method for setting the object distance. For example, the restoration unit 122L sets the object distance set by the user or the object distance detected by various sensors as the distance to the object surface 102 to be restored.

[0287] Next, the restoration unit 122L reads the coefficient set group associated with the set object distance from the storage unit 124L.

[0288] In step S3, the restoration unit 122L restores the image using the detection image and the coefficients. Specifically, the restoration unit 122L creates the simultaneous equations described with reference to equations (1) to (3) above using the detection signal level of each pixel in the detection image and the coefficient set obtained by the processing in step S2. Next, the restoration unit 122L solves the created simultaneous equations to calculate the light intensity of each point light source on the object surface 102 corresponding to the set object distance. Then, the restoration unit 122L arranges pixels having pixel values ​​corresponding to the calculated light intensities according to the layout of the respective point light sources on the object surface 102. By doing so, the restoration unit 122L generates a restored image in which an image of the object is formed.

[0289] In step S4, the restoration unit 122L performs various processing on the restored image. For example, the restoration unit 122L performs demosaicing, gamma correction, white balance adjustment, conversion to a predetermined compression format, etc. on the restored image as needed. In addition, the restoration unit 122L provides the obtained restored image (left image) to the display control unit 26 via the communication unit 125, the camera ECU 42L, and the MCU 43L.

[0290] In step S5 , the display unit 25L displays the restored image, which is the left side image, under the control of the display control unit 26 .

[0291] After that, the process returns to step S1, and the processes in steps S1 to S5 are repeated.

[0292] When the left image is displayed on the display unit 25L and the right image is displayed on the display unit 25R in the above-described manner, an electronic side mirror is obtained.

[0293] <Second Embodiment of Electronic Side Mirror Display Control Processing>

[0294] Next, refer to Figure 22 The illustrated flowchart describes a second embodiment of the electronic side mirror display control process to be executed by the in-vehicle system 11 .

[0295] For example, when the power of the vehicle 221 is turned on, the process starts, and when the power is turned off, the process ends.

[0296] Note that although a process of displaying the left image obtained by the camera module 21L on the display unit 25L is described below, a similar process is performed in the case of displaying the right image obtained by the camera module 21R on the display unit 25R.

[0297] In steps S51 to S53, the Figure 21 The processing is similar to the processing in steps S1 to S3 in .

[0298] In step S54, the restoration unit 122L performs various processing on the restored image. For example, the restoration unit 122L performs demosaicing, gamma correction, white balance adjustment, conversion to a predetermined compression format, and the like on the restored image as needed. Furthermore, the restoration unit 122L provides the obtained restored image (left image) to the camera ECU 42L via the communication unit 125.

[0299] In step S55 , the driver recognition unit 23 performs a process of detecting the pointing motion of the driver. Specifically, the imaging unit 181 captures an image of the driver and supplies the obtained driver image to the pointing motion detection unit 182 .

[0300] The instruction motion detection unit 182 performs processing for detecting a display change instruction motion based on the driver image and the detection conditions stored in the storage unit 184. If a display change instruction motion is detected, the instruction motion detection unit 182 provides data indicating instruction content to the display control unit 26 via the transmission unit 185.

[0301] In step S56, the camera ECU 42L performs dangerous object detection processing. The camera ECU 42L performs object recognition processing on the left image and detects dangerous objects (e.g., pedestrians, bicycles, motorcycles, or vehicles) on the left and rear sides of the vehicle that may collide with or come into contact with the vehicle. The camera ECU 42L provides the left image and data indicating the dangerous object detection results to the alarm control unit 24 and the display control unit 26 via the MCU 43L.

[0302] Note that the dangerous object detection conditions can be set as needed.

[0303] For example, an object whose relative speed toward the vehicle is equal to or higher than a predetermined threshold or an object approaching the vehicle at a predetermined speed or higher is detected as a dangerous object. For example, an object whose distance from the vehicle is equal to or less than a predetermined threshold is detected as a dangerous object.

[0304] Furthermore, if an object that satisfies the above conditions is detected and does not cause a problem if it collides with or contacts the vehicle, the object is not detected as a dangerous object. For example, such an object may be a vinyl bag or the like.

[0305] In step S57, the alarm control unit 24 determines whether a dangerous object has been detected based on the result of the processing performed by the camera ECU 42L in step S56. If it is determined that a dangerous object has been detected, the process proceeds to step S58.

[0306] In step S58, the alarm control unit 24 superimposes a warning display on the restored image. Specifically, the alarm control unit 24 superimposes a warning display on the left image, which draws attention to the dangerous object. For example, a display effect, such as a frame, is added to emphasize the dangerous object in the left image. The alarm control unit 24 provides the left image with the superimposed warning display to the display control unit 26.

[0307] After that, the processing enters step S59.

[0308] On the other hand, if it is determined in step S57 that any dangerous object has not been detected, the processing in step S58 is skipped, and the processing proceeds to step S59.

[0309] In step S59 , the display control unit 26 sets a display range based on the driver's instruction motion and the result of the dangerous object detection.

[0310] Here, reference Figures 23 to 27 Describes the method used to set the display range.

[0311] Figure 23 and 24 An example is shown in which the display range is set based on the pointing movement depending on the direction of the driver's gaze.

[0312] Figure 23An example of a display range DL1 set within the visual field FL1 according to the line of sight AL11 of the driver 222 is shown. In this case, the display range DL1 is set so that the incident angle θL11 of the line of sight AL11 with respect to the display surface of the display unit 25L and the angle θL12 of the center AL12 of the display range DL1 with respect to the display surface of the display unit 25L become as close as possible.

[0313] Figure 24 An example of a display range DL2 set within the visual field FL1 according to the line of sight AL13 of the driver 222 is shown. In this case, the display range DL2 is set so that the incident angle θL13 of the line of sight AL13 with respect to the display surface of the display unit 25L and the angle θL14 of the center AL14 of the display range DL2 with respect to the display surface of the display unit 25L become as close as possible.

[0314] Therefore, when the driver 222 moves his / her line of sight to the right side of the display surface of the display unit 25L, the display range of the left image displayed on the display unit 25L moves toward the front of the vehicle 221. On the other hand, when the driver 222 moves his / her line of sight to the left side of the display surface of the display unit 25L, the display range of the left image displayed on the display unit 25L moves toward the rear of the vehicle 221.

[0315] When the display range of the left image is moved based on the sight line direction of the driver 222 in this manner, an image in a direction that the driver wants to check can be displayed without giving the driver a sense of strangeness.

[0316] Note that, although not specifically described here, the display range of the right image displayed on the display unit 25R moves in a manner similar to that described above based on the driver's line of sight direction.

[0317] Figures 25 to 27 An example of setting a display range based on a dangerous object detection result is shown.

[0318] For example, when setting the range DL1 to Figure 23 In the example shown, the display range of the left image is as follows: Figure 25 Range DL1 is shown detecting a dangerous object 301 within a field of view FL1 behind vehicle 221 .

[0319] In this case, for example, Figure 26 As shown, the display range of the left image displayed on the display unit 25L is moved to a range DL3 including the dangerous object 301 .

[0320] Or, for example, Figure 27As shown, the display range of the left image displayed on the display unit 25L is expanded to a range DL4 equal to the field of view FL1.

[0321] As a result, the dangerous object 301 is displayed on the display unit 25L, allowing the driver to quickly recognize the presence of the dangerous object 301 .

[0322] Note that in Figure 26 and 27 In FIG, arrow AL15 indicates the direction of sight of the driver 222. Figure 26 In FIG, arrow AL16 indicates the direction of the center of the display range DL3. Figure 27 In FIG, arrow AL17 indicates the direction of the center of the display range DL4.

[0323] Reference again Figure 22 In step S60, the display unit 25L displays the restored image under the control of the display control unit 26. That is, the left image having the display range set by the processing in step S59 is displayed on the display unit 25L.

[0324] Note that in the case of detecting a dangerous object, e.g. Figure 28 A warning is displayed as shown. In the example shown here, in the left image, motorcycle 321 is detected as a dangerous object diagonally to the left of vehicle 221. A frame 322 is then displayed surrounding motorcycle 321, and letters and symbols are displayed around frame 322 to draw attention. This arrangement allows driver 222 to quickly and unambiguously identify motorcycle 321 as a dangerous object.

[0325] Note that even in a case where no dangerous object is detected, such as in a case where there is an area or point in the route where accidents frequently occur, for example, a warning display showing this fact may be displayed.

[0326] Alternatively, warning displays related to dangerous objects may be displayed only under certain conditions (eg, when driving on a curve, at a start, etc.).

[0327] After that, the process returns to step S51, and the processes in steps S51 to S60 are repeated.

[0328] Since the display range of the side image is appropriately set in the above manner, the driver 222 can check the side and rear of the vehicle 221 without feeling strange. In addition, the driver can be notified of the presence of a dangerous object quickly and reliably.

[0329] <<2. Second embodiment>>

[0330] Next, refer to Figures 29 to 33 A second embodiment of the present technology is described.

[0331] The second embodiment differs from the first embodiment in the light shielding pattern in the pixel array unit of the imaging device 121 .

[0332] Figure 29 A second embodiment of the light shielding pattern of the pixel array unit of the imaging device 121L of the imaging unit 41L of the camera module 21L is shown. Figure 30 Shown as a composition Figure 29 1 and 2. An example of a light shielding pattern of a pixel PLb and a pixel PLc of the second embodiment of the pixel 121a of the pixel array unit shown.

[0333] The pixels PLb are arranged in odd-numbered columns in the pixel array unit, and the pixels PLc are arranged in even-numbered columns in the pixel array unit.

[0334] The positions of the opening setting areas are different between the pixel PLb and the pixel PLc. Specifically, the shapes and sizes of the opening setting areas RLb of the light shielding film SLb of the pixel PLb and the opening setting areas RLc of the light shielding film SLc of the pixel PLc are different. Figure 20 The shape and size of the opening setting region RLa of the light shielding film SLa of the middle pixel PLa are the same.

[0335] At the same time, compared to opening setting area RLa, opening setting area RLb is set at a position in pixel PLb that is shifted to the right. However, similar to the center of gravity of opening setting area RLa, the center of gravity of opening setting area RLb is offset to the left (forward with respect to vehicle 221) from the center of pixel PLb. Furthermore, compared to opening setting area RLa, opening setting area RLc is set at a position in pixel PLc that is shifted to the left. Thus, the positions in the horizontal direction (forward / rearward direction of vehicle 221) of the pixel differ between opening setting area RLb and opening setting area RLc.

[0336] In addition, the opening ALb of the pixel PLb has the same shape and size as the opening ALa of the pixel PLa, and is formed according to a method similar to that described above with reference to FIG. Figure 12 and 13 The rule of the rule is located in the opening setting area RLb.

[0337] Specifically, opening ALb is located at the left end of the opening setting area RLb in each pixel PLb in the leftmost column of the pixel array unit, and is located at the upper end of the opening setting area RLb in each pixel PLb in the uppermost row of the pixel array unit. Furthermore, as the position of the pixel PLb becomes closer to the right, opening ALb shifts rightward at equal intervals within the opening setting area RLb and is located at the right end of the opening setting area RLb in each pixel PLb in the second column from the right in the pixel array unit. Furthermore, as the position of the pixel PLb becomes closer to the bottom, opening ALb shifts bottomward at equal intervals within the opening setting area RLb and is located at the lower end of the opening setting area RLb in each pixel PLb in the lowermost row of the pixel array unit.

[0338] Therefore, the horizontal position of the opening A1b in each pixel PLb is the same between the pixels PLb in the same column in the vertical direction. Furthermore, the vertical position of the opening A1b in each pixel PLb is the same between the pixels PLb in the same row in the horizontal direction. Therefore, the position of the opening A1b in each pixel PLb (the position at which incident light enters each pixel PLb) varies with each pixel PLb, and as a result, the incident angle directionality of each pixel PLa differs from one another.

[0339] Furthermore, the openings ALb of each pixel PLb overlap the opening setting region RLb. That is, the area where the openings ALb of each pixel PLb overlap each other is equal to the opening setting region RLb. Note that the layout pattern of the openings ALb is not limited to the above configuration and can be any layout as long as the area where the openings ALb overlap each other is equal to the opening setting region RLb. For example, the openings ALb may be randomly arranged within the opening setting region RLb.

[0340] In addition, the opening ALc of the pixel PLc has the same shape and size as the opening ALa of the pixel PLa, and is formed according to a method similar to that described above with reference to FIG. Figure 12 and 13 The rule of the rule is located in the opening setting region RLc.

[0341] Specifically, the opening ALc is located at the left end of the opening setting area RLc in each pixel PLc in the second column from the left in the pixel array unit, and is located at the upper end of the opening setting area RLc in each pixel PLc in the upper row in the pixel array unit. In addition, as the position of the pixel PLc becomes closer to the right, the opening ALc shifts rightward at equal intervals within the opening setting area RLc and is located at the right end of the opening setting area RLc in each pixel PLc in the right end column in the pixel array unit. In addition, as the position of the pixel PLc becomes closer to the bottom, the opening ALc shifts toward the bottom at equal intervals within the opening setting area RLc and is located at the lower end of the opening setting area RLc in each pixel PLc in the lower row in the pixel array unit.

[0342] Therefore, in each pixel PLc in the same column in the vertical direction, the position of the opening ALc in each pixel PLc in the horizontal direction is the same. In addition, in each pixel PLc in the same row in the horizontal direction, the position of the opening ALc in each pixel PLc in the vertical direction is the same. Therefore, the position of the opening ALc in each pixel PLc (which is the position at which incident light enters each pixel PLc) varies with each pixel PLc, and as a result, the directionality of the incident angle of each pixel PLc differs from one another.

[0343] Furthermore, the openings ALc of each pixel PLc overlap the opening setting region RLc. That is, the area where the openings ALc of each pixel PLc overlap is equal to the opening setting region RLc. Note that the layout pattern of the openings ALc is not limited to the above configuration and can be any layout as long as the area where the openings ALc overlap is equal to the opening setting region RLc. For example, the openings ALc may be randomly arranged within the opening setting region RLc.

[0344] Here, the center of gravity of the incident angle directivity of each pixel PLb substantially coincides with the center of gravity of aperture ALb of each pixel PLb, and in most pixels PLb, is offset to the left (forward with respect to vehicle 221) from the center of each pixel PLb. Therefore, the average value of the center of gravity of the incident angle directivity of each pixel PLb is offset to the left from the center of pixel PLb. Furthermore, the average value of the incident angle of the centroid light beam in each pixel PLb is offset to the right (rearward with respect to vehicle 221) relative to the normal direction of the light receiving surface of the pixel array unit.

[0345] Furthermore, the center of gravity of the incident angle directivity of each pixel PLc substantially coincides with the center of gravity of the aperture ALc of each pixel PLc, and in most pixels PLc, is offset to the left (forward with respect to the vehicle 221) from the center of each pixel PLc. Consequently, the average value of the center of gravity of the incident angle directivity of each pixel PLc is offset to the left from the center of the pixel PLc. Furthermore, the average value of the incident angle of the centroid light beam in each pixel PLc is offset to the right (rearward with respect to the vehicle 221) relative to the normal direction of the light receiving surface of the pixel array unit.

[0346] At the same time, the offset from the center of each pixel PLc in the opening setting area RLc is greater than the offset from the center of each pixel PLb in the opening setting area RLb. Therefore, the average value of the incident angle of the centroid light beam in each pixel PLc is tilted to the right (rearward with respect to the vehicle 221) compared to the average value of the incident angle of the centroid light beam in each pixel PLb.

[0347] Therefore, if Figure 31 and 32 As shown, the pixels PLb and the pixels PLc of the imaging device 121 enable imaging of different fields of view FL21 and FL22 in the front and rear directions of the vehicle 221 .

[0348] Specifically, Figure 31 The field of view FL21 to be imaged by each pixel PLb of the imaging device 121L is shown. Note that arrow AL21 indicates the direction of the driver's line of sight AL21, and arrow AL22 indicates the imaging direction of each pixel PLb of the imaging device 121L. Angle θL21 indicates the angle of incidence of the driver's line of sight AL21 relative to the display surface of the display unit 25L. Angle θL22 indicates the angle of imaging direction AL22 relative to the display surface of the display unit 25L.

[0349] Figure 32 The field of view FL22 to be imaged by each pixel PLv of the imaging device 121L is shown. Note that arrow AL23 indicates the direction of the driver's line of sight AL23, and arrow AL24 indicates the imaging direction of each pixel PLc of the imaging device 121L. Angle θL23 indicates the angle of incidence of the driver's line of sight AL23 relative to the display surface of the display unit 25L. Angle θL24 indicates the angle of imaging direction AL24 relative to the display surface of the display unit 25L. The field of view FL22 is displaced from the field of view FL21 toward the rear of the vehicle 221.

[0350] Note that, although not specifically described, the imaging device 121 of the camera module 21R also includes a camera capable of capturing different fields of view FR21 ( Figure 31 ) and FL22( Figure 32 ) Two types of pixels for imaging.

[0351] <Electronic Side Mirror Display Control Processing>

[0352] Next, refer to Figure 33 The flowchart shown describes the electronic side mirror display control process to be executed by the vehicle-mounted system 11 according to the second embodiment of the present technology.

[0353] For example, when the power of the vehicle 221 is turned on, the process starts, and when the power is turned off, the process ends.

[0354] Note that although a process of displaying the left image obtained by the camera module 21L on the display unit 25L is described below, a similar process is performed in the case of displaying the right image obtained by the camera module 21R on the display unit 25R.

[0355] In step S101, Figure 21 Imaging of the surroundings of the vehicle 221 is performed by way of the processing in step S1 .

[0356] In step S102, Figure 22 The process of detecting the driver's instruction movement is performed in a similar manner to the process in step S55.

[0357] In step S103 , the restoration unit 122L selects pixels and coefficients to be used for image restoration.

[0358] For example, in the case of selecting pixels to be used for image restoration based on an indicated movement depending on the direction of the driver's line of sight, pixels whose imaging directions relative to the display surface of the display unit 25L have an angle closer to the incident angle of the driver's 222 line of sight relative to the display surface of the display unit 25L are selected from pixels PLb and pixels PLc.

[0359] For example, by Figure 31 The angle θL22 of the imaging direction AL22 formed by each pixel PLb of the imaging device 121L is greater than that of the imaging direction AL22 formed by Figure 32 The angle θL24 of the imaging direction AL24 formed by each pixel PLc of the imaging device 121L is closer to Figure 31 Therefore, the pixel PLb is selected based on the line of sight AL21 of the driver 222.

[0360] On the other hand, for example, Figure 32 The angle θL24 of the imaging direction AL24 formed by each pixel PLc of the imaging device 121L is greater than that formed by Figure 31 The angle θL22 of the imaging direction AL22 formed by each pixel PLb of the imaging device 121L is closer to Figure 32Therefore, the pixel PLC is selected based on the line of sight AL23 of the driver 222.

[0361] Furthermore, the restoration unit 122L sets the distance to the object surface 102 to be restored, that is, the object distance. Note that any method can be employed as a method for setting the object distance. For example, the restoration unit 122L sets the object distance set by the user or the object distance detected by various sensors as the distance to the object surface 102 to be restored.

[0362] Next, the restoration unit 122L reads the coefficient set group associated with the selected pixel and the set object distance from the storage unit 124L.

[0363] In steps S104 and S105, the Figure 21 The processing in steps S3 and S4 is similar to the processing in FIG.

[0364] In step S106, the display unit 25L displays the restored image under the control of the display control unit 26. That is, the left image having the field of view selected in the process of step S103 is displayed on the display unit 25L. As a result, the field of view (display range) of the left image displayed on the display unit 25L is appropriately changed based on the instruction movement of the driver 222.

[0365] After that, the process returns to step S101 , and the processes in steps S101 to S106 are performed.

[0366] Since the fields of view (imaging range and display range) of the side images are appropriately set in the above-described manner, the driver 222 can check the sides and rear of the vehicle 221 without feeling strange.

[0367] Note that, as in the above-described first embodiment, the field of view of the side image may be changed based on the result of the dangerous object detection.

[0368] 3. Modifications

[0369] The following is a description of modifications to the above-described embodiments of the present technology.

[0370] <Modifications Related to Camera Module 21>

[0371] In the above description, the camera module 21 is attached to the inner surface of the side window of the vehicle 221. However, the camera module 21 may be attached to the outer surface of the vehicle side window, for example.

[0372] In addition, for example, the camera module 21 may be attached to an outer side surface of the vehicle 221. For example, the camera module 21 may be attached to an outer surface of a pillar or a door.

[0373] Furthermore, as described above, the imaging direction of the camera module 21 can be adjusted using the light shielding pattern of the imaging device 121. Therefore, the degree of freedom of the mounting position with respect to the field of view to be imaged is high. Furthermore, the imaging direction of the imaging device 121L of the camera module 21L and the imaging direction of the imaging device 121R of the camera module 21R can be set independently of each other. Therefore, as Figure 34 As shown, for example, the camera module 21L and the camera module 21R do not necessarily need to be attached to the vehicle 221 at symmetrical positions on both sides thereof.

[0374] In this example, the camera module 21L is mounted on the left side surface of the vehicle 221 at a position substantially identical to that of the driver 222 in the front-rear direction. The camera module 21R is mounted on the right side surface of the vehicle 221 at a position substantially identical to the rear end of the driver's seat 351 in the front-rear direction. Therefore, in the front-rear direction of the vehicle 221, the camera module 21L is attached to a portion further forward than the camera module 21R.

[0375] Furthermore, in the above example, the imaging unit 41, the camera ECU 42, and the MCU 43 are provided on two different semiconductor chips. However, other configurations may be employed. For example, the imaging device 121 of the imaging unit 41, the signal processing control unit 111 of the imaging unit 41, the camera ECU 42, and the MCU 43 may be provided on two different semiconductor chips, or the imaging device 121, the signal processing control unit 111 of the imaging unit 41, the camera ECU 42, and the MCU 43 may be provided on three different semiconductor chips. Alternatively, for example, the imaging unit 41, the camera ECU 42, and the MCU 43 may be provided on a single semiconductor chip.

[0376] In addition, for example, the camera module 21 may be formed by stacking Figure 16 The LLC chip 202 and the signal processing chip 203 are formed of semiconductor chips.

[0377] Furthermore, for example, the imaging unit 41 or the imaging device 121 may be provided in the vehicle separately from the other components of the camera module 21. With this arrangement, the degree of freedom of the installation position of the imaging unit 41 or the imaging device 121 becomes higher.

[0378] Furthermore, two camera modules 21 may be combined with each other so that, for example, the light receiving surfaces face in opposite directions to each other. A camera module formed in this manner and capable of imaging both sides may be mounted on a similar Figure 17 、 34 Positions in the example shown, etc., so that the interior of the vehicle as well as the exterior of the vehicle can be imaged.

[0379] This arrangement allows, for example, monitoring of passengers sitting in the passenger and rear seats. For example, a child safety seat may be installed behind the driver's seat for safety reasons. However, in this case, it is difficult for the driver to see the child safety seat. To overcome this, for example, the aforementioned camera module capable of imaging both sides may be installed in a position adjacent to the vehicle. Figure 34 The camera module 21R is located at the same position as the camera module 21R shown above. Therefore, it is possible to image and monitor a child sitting in a child safety seat behind the driver's seat.

[0380] Note that a lensless camera capable of imaging both the front and rear surfaces may also be used instead of a module formed by bonding camera modules 21 to each other. For example, a lensless camera formed by bonding CMOS image sensors each having a predetermined pattern formed by holes to each other may also be used. Each image sensor performs imaging using a pattern formed on another image sensor as a light shielding pattern. This type of lensless camera is capable of double-sided imaging. For example, Tomoya Nakamura and four other scholars specifically describe this type of lensless camera in “Super Field-of-View Lensless Camera by Coded Image Sensors, 2019”.

[0381] Furthermore, if a camera module capable of dual-sided imaging can only recover one image at a time, it is possible to switch between imaging the vehicle's exterior and interior for each frame. In this case, since monitoring the vehicle's exterior is more important, the exterior can be imaged more frequently than the interior.

[0382] <Modifications related to display unit>

[0383] The installation position of the display unit 25 is not limited to the above-described example but may be changed.

[0384] For example, Figure 35 As shown, the display unit 25R may be provided on the inner surface of a pillar 233R at the right end of the windshield 231 of the vehicle 221. Similarly, although not shown in any figure, the display unit 25L may be provided on the inner surface of a pillar 233L at the left end of the windshield 231 of the vehicle 221.

[0385] In this case, for example, the camera module 21R may be provided on the outer surface of the pillar 233R, and an image in a direction that cannot be seen from the driver 222 due to the pillar 233R may be captured by the camera module 21R and displayed on the display unit 25R. Similarly, although not shown in any figure, the camera module 21L may be provided on the outer surface of the pillar 233L, and an image in a direction that cannot be seen from the driver 222 due to the pillar 233L may be captured by the camera module 21L and displayed on the display unit 25L.

[0386] In addition, the display unit 25 may not be divided into a right display unit and a left display unit. Instead, a display unit may be provided at the center of the front surface of the instrument panel of the vehicle 221 so that, for example, a left image and a right image can be displayed on the left and right sides of the display unit, respectively.

[0387] In addition, Figure 17 In the above-described example shown, for example, when the driver 222 is looking at the display unit 25L or the display unit 25R, he / she does not turn his / her sight line toward the imaging unit 181. Therefore, in the driver image captured by the imaging unit 181, the detection accuracy in the driver's sight line direction may be reduced.

[0388] On the other hand, Figure 36 As shown, for example, instead of the display unit 25L and the display unit 25R, the display unit 401L and the display unit 401R may be provided at substantially the same positions as the display unit 25L and the display unit 25R.

[0389] The display unit 401L displays the left image and also captures the driver's image to provide the obtained driver image to the driver recognition unit 23. Based on the obtained driver image, the driver recognition unit 23 monitors the driver and performs sight line direction detection and the like.

[0390] Likewise, the display unit 401R displays the right image and also captures the driver's image to provide the obtained driver image to the driver recognition unit 23. Based on the obtained driver image, the driver recognition unit 23 monitors the driver and performs sight direction detection and the like.

[0391] With this arrangement, when the driver looks at display unit 401L or display unit 401R, the driver can be imaged from the direction in which the driver's line of sight is directed, and therefore, the detection accuracy of the driver's line of sight direction becomes higher.

[0392] Note that in the case where there is no need to distinguish the display units from each other, the display unit 401L and the display unit 401R will be simply referred to as the display unit 401 hereinafter.

[0393] Figure 37 and 38 An example configuration of the display surface of the display unit 401 in the case where the display unit 401 is formed of a micro LED display having micro LEDs as display elements is shown.

[0394] exist Figure 37 In the example shown, pixels including micro-LEDs 411 and represented by white squares and pixels including imaging devices 412 and represented by shaded squares are arranged alternately in every other row in the array.

[0395] Note that the imaging device 412 is an imaging device similar to the imaging device 121 .

[0396] exist Figure 38 In the example shown, imaging devices 412 are provided at the four corners of a region where pixels including micro LEDs 411 are arranged in a two-dimensional array.

[0397] Here, the area occupied by the micro-LEDs 411, which serve as light sources, is very small in each pixel of the display unit 401. Therefore, the imaging device 412 can be easily arranged in the gap between adjacent micro-LEDs 411, etc. With this arrangement, the driver can be imaged by the imaging device 412, while the side image can be displayed by the micro-LEDs 411.

[0398] Note that the display element does not necessarily have to be a micro LED. For example, the display element may be an organic EL element.

[0399] <Modification Related to Method of Changing Display Range of Side Image>

[0400] As described above, learning unit 183 learns the detection conditions that serve as a basis for detecting display change instruction motions. This arrangement prevents, for example, unintended movements by the driver from being erroneously detected as display change instruction motions, prevents the display range of any side images from being changed, prevents any pressure from being applied to the driver, and prevents monitoring of the desired area from being obstructed. For example, it is possible to prevent the driver from erroneously detecting movements such as shifting their gaze to see a person sitting in the passenger seat or a car navigation device, looking back to see someone sitting in the back seat, or raising their hand to touch their head as display change instruction motions.

[0401] Note that since each driver has characteristic movements (e.g., moving with large movements or excessive activity), it is best to learn the detection conditions for each driver separately. For example, the default detection conditions are used initially, and in the case where the display range of the side image is changed, the detection conditions are learned and updated based on the driver's reaction. For example, in the case where the driver does not respond, the change in the display range is determined to be appropriate (correct). In the case where the driver makes an action to change the display range, the change in the display range is determined to be inappropriate (incorrect). For example, the movement to change the display range performed by the driver can be a new display change indication movement, or a manual operation using a button or the like.

[0402] Alternatively, the display change instruction movement may be detected, for example, taking into account conditions other than the driver's movement. For example, the display change instruction movement may be detected taking into account conditions such as the vehicle's speed, travel route, or travel status.

[0403] Specifically, for example, when the vehicle approaches a curve, a wide display range of the side image is considered ideal. Therefore, the detection conditions can be relaxed so that the display change indication movement can be easily detected.

[0404] In addition, for example, when the vehicle is traveling at high speed or entering a highway, the driver pays close attention to the front, so the side image viewing time is very short. In view of this, the detection conditions can be changed to shorten the time required to detect the display change indication movement.

[0405] Furthermore, for example, detection conditions can be learned not only for a driver but also for each region (e.g., each country), each vehicle, or each vehicle type (e.g., large vehicle, trailer, or small vehicle).

[0406] Furthermore, for example, the display range of the side image may be changed based on conditions other than the driver's display change instruction movement and the result of the above-described dangerous object detection.

[0407] For example, the display range of the side image may be changed based on the driving operation. For example, the display range of the side image may be changed due to sudden starting, sudden braking, sudden turning, reversing, parking, etc.

[0408] Furthermore, for example, in a case where some other sensor detects an alarm object approaching from behind, the display range of the side image is changed so as to display the detected object as quickly as possible.

[0409] In addition, for example, the display range of the side image can be changed based on the situation around the vehicle. For example, the display range of the side image can be changed based on the width of the road the vehicle is traveling on, the weather, the brightness of the surrounding environment, etc.

[0410] Furthermore, for example, the display range of the side image may be changed based on the planned route of the vehicle.

[0411] Furthermore, the display range of the side image may be changed by a manual operation such as a button operation or by a voice command or the like.

[0412] <Modification Related to Imaging Device 121>

[0413] although Figure 29 An example is shown in which two types of opening setting areas are set for the imaging device 121 , but three or more types of opening setting areas may be set.

[0414] For example, three or more types of opening setting areas having different horizontal positions may be set.

[0415] Furthermore, for example, two or more types of opening setting regions that differ in at least height or width may be set.

[0416] Furthermore, for example, each imaging device 121 may include pixels having an aperture setting area capable of imaging not only in the oblique rear direction of the vehicle but also in the oblique front direction of the vehicle.

[0417] Furthermore, for example, drive units can be provided that independently drive pixels 121a with corresponding fields of view, allowing for simultaneous or individual imaging of pixels 121a with corresponding fields of view. Furthermore, for example, only pixels 121a corresponding to the restored image used to display the electronic side mirror can be imaged. This arrangement reduces the processing performed by the imaging device 121.

[0418] also, Figure 5 The following examples are shown: using the light shielding film 121b as a modulation element, or changing the combination of photodiodes contributing to the output to provide different incident angle directivities for each pixel. However, according to the present technology, the filter 902 covering the light receiving surface of the imaging device 901 can be used as a modulation element, so that, for example, Figure 39 As shown, the incident angle directionality is provided for each pixel.

[0419] Specifically, the filter 902 is arranged at a predetermined distance from the light receiving surface 901A of the imaging device 901 so as to cover the entire surface of the light receiving surface 901A. Light from the object surface 102 is modulated by the filter 902 and then enters the light receiving surface 901A of the imaging device 901.

[0420] For example, with Figure 40The filter 902BW with a black and white lattice pattern shown can be used as the filter 902. In the filter 902BW, white pattern portions that transmit light and black pattern portions that block light are randomly arranged. The size of each pattern is set independently of the size of the pixel of the imaging device 901.

[0421] Figure 41 1 and 2 show light reception sensitivity characteristics of the imaging device 901 with respect to light from the point light source PA and the point light source PB on the object surface 102, in the case of using the optical filter 902BW. Light from each of the point light source PA and the point light source PB is modulated by the optical filter 902BW and then enters the light receiving surface 901A of the imaging device 901.

[0422] For example, the light receiving sensitivity characteristics of the imaging device 901 with respect to light from the point light source PA resemble waveform Sa. That is, the shadows are formed by the black pattern portion of the filter 902BW, and thus, a grayscale pattern is formed in the image on the light receiving surface 901A for the light from the point light source PA. Similarly, for example, the light receiving sensitivity characteristics of the imaging device 901 with respect to light from the point light source PB resemble waveform Sb. That is, the shadows are formed by the black pattern portion of the filter 902BW, and thus, a grayscale pattern is formed in the image on the light receiving surface 901A for the light from the point light source PB.

[0423] Note that the light from the point light source PA and the light from the point light source PB have different incident angles with respect to the respective white pattern portions of the filter 902BW, thus causing a difference in the appearance of the grayscale pattern on the light receiving surface. Therefore, each pixel of the imaging device 901 has incident angle directivity with respect to each point light source on the object surface 102.

[0424] For example, the details of this method are disclosed by M. Salman Asif and four others in “Flatcam: Replacing lenses with masks and computation”, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW), 2015, pp. 663-666.

[0425] Note that you can use Figure 42Filter 902HW is shown in FIG. Instead of the black pattern portion of filter 902BW, filter 902HW includes linear polarization element 911A and linear polarization element 911B having the same polarization direction, and half-wavelength plate 912. Half-wavelength plate 912 is interposed between linear polarization element 911A and linear polarization element 911B. Instead of the black pattern portion of filter 902BW, a polarization portion indicated by a hatched portion is provided in half-wavelength plate 912, and white pattern portions and polarization portions are randomly arranged.

[0426] The linear polarization element 911A transmits only light with a predetermined polarization direction within the substantially unpolarized light beam emitted from the point light source PA. In the following description, the linear polarization element 911A transmits only light with a polarization direction parallel to the image. Within the polarized light beam transmitted through the linear polarization element 911A, the polarized light transmitted through the polarization portion of the half-wavelength plate 912 changes its polarization direction to a direction perpendicular to the image as the polarization plane rotates. On the other hand, within the polarized light beam transmitted through the linear polarization element 911A, the polarized light transmitted through the white pattern portion of the half-wavelength plate 912 remains parallel to the image without changing its polarization direction. The linear polarization element 911B transmits the polarized light transmitted through the white pattern portion but transmits almost no polarized light transmitted through the polarization portion. Therefore, the amount of polarized light transmitted through the polarization portion becomes smaller than the amount of polarized light transmitted through the white pattern portion. As a result, a grayscale pattern substantially similar to that in the case of using the filter BW is formed on the light receiving surface 901A of the imaging device 901.

[0427] In addition, if Figure 43 As shown in FIG. 1A , an optical interference mask can be used as a filter 902LF. Light emitted from point light sources PA and PB on the object surface 102 is emitted onto the light receiving surface 901A of the imaging device 901 via the filter 902LF. Figure 43 As shown in the enlarged view of the lower part of A, for example, the light incident surface of the filter 902LF has irregularities of a size similar to the size of the wavelength. In addition, the filter 902LF maximizes the transmission of light of a specific wavelength emitted from the vertical direction. When the change in the angle of incidence (or tilt relative to the vertical direction) of light of a specific wavelength emitted from the point light sources PA and PB on the object surface 102 relative to the filter 902LF becomes larger, the optical path length changes. Here, when the optical path length is an odd multiple of half a wavelength, the light beams weaken each other. When the optical path length is an even multiple of half a wavelength, the light beams reinforce each other. That is, as Figure 43As shown in FIG. 1B , the intensity of transmitted light of a specific wavelength emitted from point light sources PA and PB and transmitted through filter 902LF is modulated according to the incident angle relative to filter 902LF, and then enters light receiving surface 901A of imaging device 901. Therefore, the detection signal output from each pixel of imaging device 901 is a signal obtained by combining the light intensities after the modulation of the individual point light sources for each pixel.

[0428] For example, JP 2016-510910 W discloses details of this method.

[0429] <Modification of Shared Processing in In-Vehicle System 11>

[0430] The distribution of processing in the in-vehicle system 11 can be changed as needed.

[0431] For example, the processing performed by the driver identification unit 23 may also be performed by the control unit 27 .

[0432] For example, the processing to be performed by the alarm control unit 24 may also be performed by the control unit 27 or the camera ECU 42 .

[0433] For example, the processing performed by the restoration unit 122 may also be performed by the camera ECU 42 or the control unit 27 .

[0434] <Other modifications>

[0435] The present technology can be applied to imaging devices and imaging apparatuses that image light of a wavelength different from visible light (e.g., infrared light). In this case, the restored image is not an image from which the user can visually identify the object, but an image from which the user cannot visually identify the object. In this case, the present technology is also used to improve the quality of restored images in image processing devices, etc. that can identify objects. Note that conventional imaging lenses have difficulty transmitting far-infrared light, and therefore, the present technology is effective in cases where, for example, far-infrared light imaging is performed. Therefore, the restored image may be an image of far-infrared light. Alternatively, the restored image is not necessarily an image of far-infrared light, but may be an image of some other visible or invisible light.

[0436] Furthermore, by applying machine learning techniques such as deep learning, for example, it is possible to perform image recognition using pre-recovery detection images without the need for a restored image. In this case, this technology can also be used to improve the accuracy of image recognition using pre-recovery detection images. In other words, the image quality of the pre-recovery detection images becomes higher.

[0437] in this case, Figure 1 The camera ECU 42L and the camera ECU 42R in FIG. 4 perform image recognition using the detection image, for example.

[0438] <<4. Other aspects>>

[0439] The above series of processes can be performed by hardware or by software. In the case where a series of processes are performed by software, the program forming the software is installed in a computer. Here, the computer can be a computer (e.g., control unit 123) incorporated into dedicated hardware.

[0440] For example, the program to be executed by the computer may be recorded on a recording medium as a package medium, etc., and then provided. Alternatively, the program may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0441] Note that the program to be executed by the computer may be a program for executing processing in time series in the sequence described in this specification, or may be a program for executing processing in parallel or when necessary (for example, when there is a call).

[0442] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications may be made thereto without departing from the scope of the present technology.

[0443] For example, the present technology may be implemented in a cloud computing configuration in which one function is shared among multiple devices via a network, and processing is performed by the devices cooperating with each other.

[0444] Furthermore, the various steps described with reference to the above flowcharts may be performed by one device, or may be shared among a plurality of devices.

[0445] Furthermore, in the case where a plurality of processes are included in one step, the plurality of processes included in one step may be executed by one device, or may be shared among a plurality of devices.

[0446] Note that the present technology can also be embodied in the configurations described below.

[0447] (1) An imaging device comprising:

[0448] a plurality of pixels that receive incident light entering from an object after passing through neither an imaging lens nor a pinhole, and each pixel outputs a detection signal indicating an output pixel value modulated according to an incident angle of the incident light,

[0449] The imaging device is attached to a vehicle so that the light receiving surface faces one side of the vehicle, and an average value of the center of gravity of the incident angle directivity indicating the directivity of the incident angle of the plurality of pixels with respect to the incident light deviates from the center of the pixel in one direction.

[0450] (2) The imaging device according to (1), wherein

[0451] The plurality of pixels include a plurality of first pixels into which incident light enters at positions different from one another in a first region in which the center of gravity is offset from the center of the pixels toward the front of the vehicle in a state in which the imaging apparatus is attached to the vehicle.

[0452] (3) The imaging device according to (2), wherein

[0453] Each of the plurality of pixels comprises:

[0454] a photoelectric conversion element; and

[0455] a light-shielding film that blocks part of incident light from entering the photoelectric conversion element, and

[0456] The openings of the light shielding films of the plurality of first pixels are located at positions different from each other in the first region.

[0457] (4) The imaging device according to (3), wherein

[0458] The plurality of pixels also include a plurality of second pixels in which the openings of the light-shielding film are located at different positions from each other in a second area, the second area being different from the first area and being an area in which the center of gravity is offset from the center of the pixel along the front / rear direction of the vehicle when the imaging device is attached to the vehicle.

[0459] (5) The imaging device according to any one of (1) to (4), wherein

[0460] In a state in which the imaging apparatus is attached to the vehicle, the average value of the center of gravity of the incident angle directivity is biased toward the front of the vehicle.

[0461] (6) The imaging device according to any one of (1) to (5), wherein

[0462] Imaging devices are attached to both sides of the vehicle.

[0463] (7) The imaging device according to (6), wherein

[0464] The imaging device is attached to the vehicle so that the light receiving surface faces and contacts or is close to the inner surface of the side window of the vehicle.

[0465] (8) The imaging device according to (6), wherein

[0466] The imaging device is attached to an outer side surface of the vehicle.

[0467] (9) The imaging device according to any one of (6) to (8), wherein

[0468] The imaging device is attached to one side of the vehicle at a position different in the front / rear direction of the vehicle from another imaging device attached to the other side of the vehicle.

[0469] (10) The imaging device according to any one of (1) to (9), wherein

[0470] A restored image restored from a detection image based on detection signals of the plurality of pixels is used for an electronic side mirror of a vehicle.

[0471] (11) A display device provided in a vehicle at an oblique left side or an oblique right side in front of a driver,

[0472] The display device comprises:

[0473] a display surface on which a plurality of display elements are arranged; and

[0474] A plurality of pixels are provided on a display surface, receive incident light entering from an object after passing through neither an imaging lens nor a pinhole, and each pixel outputs a detection signal indicating an output pixel value modulated according to an incident angle of the incident light.

[0475] (12) The display device according to (11), wherein

[0476] The display device displays images of the side and rear of the vehicle.

[0477] (13) The display device according to (11) or (12), wherein

[0478] The detection signals of the plurality of pixels are used to detect the driver's sight direction.

[0479] (14) An imaging system comprising:

[0480] an imaging device including a plurality of pixels that receive incident light entering from an object after passing through neither an imaging lens nor a pinhole, and each pixel outputting a detection signal indicative of an output pixel value modulated according to an incident angle of the incident light, the imaging device being attached to a vehicle so that a light receiving surface faces one side of the vehicle, wherein an average value of centers of gravity of incident angle directivities indicative of directivities of the plurality of pixels with respect to the incident angle of the incident light deviates from a center of the pixel in one direction; and

[0481] A display unit displays a restored image restored from a detection image based on the detection signals of the plurality of pixels.

[0482] (15) The imaging system according to (14), wherein

[0483] The multiple pixels include: a plurality of first pixels where incident light enters at positions different from one another in a first area where the center of gravity is offset from the center of the pixel toward the front of the vehicle when the imaging device is attached to the vehicle; and a plurality of second pixels where incident light enters at positions different from one another in a second area, the second area being an area different from the first area and being an area where the center of gravity is offset from the center of the pixel in the front / rear direction of the vehicle.

[0484] (16) The imaging system according to (15), further comprising

[0485] A display control unit controls display of a restored image restored from a detection image based on a detection signal from a first pixel and a restored image restored from a detection image based on a detection signal from a second pixel, the display being performed by the display unit.

[0486] (17) The imaging system according to (16), wherein

[0487] The display control unit changes the restored image displayed by the display unit based on at least one of a movement of the vehicle driver or a result of the dangerous object detection.

[0488] (18) The imaging system according to any one of (14) to (17), further comprising

[0489] A display control unit controls a range of the restored image displayed by the display unit based on at least one of a movement of a vehicle driver or a result of dangerous object detection.

[0490] (19) The imaging system according to any one of (14) to (18), wherein

[0491] The display unit includes:

[0492] a display surface on which a plurality of display elements are arranged; and

[0493] A plurality of pixels are provided on a display surface, receive incident light entering from an object after passing through neither an imaging lens nor a pinhole, and each pixel outputs a detection signal indicating an output pixel value modulated according to an incident angle of the incident light.

[0494] (20) The imaging system according to (19), wherein

[0495] The detection signals of the plurality of pixels of the display unit are used to detect the sight direction of the vehicle driver.

[0496] Note that the advantageous effects described in this specification are merely examples, and the advantageous effects of the present technology are not limited to these and may include other effects.

[0497] Reference Designation List

[0498] 11 In-vehicle systems

[0499] 21L, 21R camera modules

[0500] 23 Driver identification unit

[0501] 24 Alarm Control Unit

[0502] 25L, 25R display units

[0503] 26 Display control unit

[0504] 27 Control Unit

[0505] 41L, 41R imaging units

[0506] 42L, 42R camera ECU

[0507] 121 Imaging Equipment

[0508] 121a pixels

[0509] 122 Recovery Unit

[0510] 123 Control Unit

[0511] 181 Imaging Unit

[0512] 182 Indicator motion detection unit

[0513] 201 substrate

[0514] 202 LLC chip

[0515] 221 vehicles

[0516] 222 Driver

[0517] 232L, 232R side windows

Claims

1. An imaging device comprising: a plurality of pixels that receive incident light entering from an object after passing through neither an imaging lens nor a pinhole, and each pixel outputs a detection signal indicating an output pixel value modulated according to an incident angle of the incident light, wherein the imaging device is attached to a vehicle so that a light receiving surface faces the left side or the right side of the vehicle, and an average value of a center of gravity of incident angle directivities indicating directivities of the plurality of pixels with respect to an incident angle of incident light deviates in one direction from a center of the pixel, wherein the center of gravity of the incident angle directivities is a center of gravity of a light intensity distribution of incident light entering the light receiving surface of the pixel, Here, in a state where the imaging device is attached to the vehicle, the average value of the center of gravity of the incident angle directivity is biased toward the front of the vehicle.

2. The imaging device according to claim 1, wherein The plurality of pixels include a plurality of first pixels into which incident light enters at positions different from one another in a first region in which the center of gravity is offset from the center of the pixels toward the front of the vehicle in a state in which the imaging apparatus is attached to the vehicle.

3. The imaging device according to claim 2, wherein Each of the plurality of pixels comprises: Photoelectric conversion element; and a light-shielding film that blocks part of incident light from entering the photoelectric conversion element, and The openings of the light shielding films of the plurality of first pixels are located at positions different from each other in the first region.

4. The imaging device according to claim 3, wherein The plurality of pixels also include a plurality of second pixels in which the openings of the light-shielding film are located at different positions from each other in a second area, the second area being different from the first area and being an area in which the center of gravity is offset from the center of the pixel along the front / rear direction of the vehicle when the imaging device is attached to the vehicle.

5. The imaging device according to claim 1, wherein Imaging devices are attached to both sides of the vehicle.

6. The imaging device according to claim 5, wherein The imaging device is attached to the vehicle so that the light receiving surface faces and contacts or is close to the inner surface of the side window of the vehicle.

7. The imaging device according to claim 5, wherein The imaging device is attached to an outer side surface of the vehicle.

8. The imaging device according to claim 5, wherein The imaging device is attached to one side of the vehicle at a position different in the front / rear direction of the vehicle from another imaging device attached to the other side of the vehicle.

9. The imaging device according to claim 1, wherein A restored image restored from a detection image based on detection signals of the plurality of pixels is used for an electronic side mirror of a vehicle.

10. An imaging system comprising: an imaging device comprising a plurality of pixels that receive incident light entering from an object after passing through neither an imaging lens nor a pinhole, and each pixel outputting a detection signal indicative of an output pixel value modulated according to an incident angle of the incident light, the imaging device being attached to a vehicle so that a light receiving surface faces the left side or the right side of the vehicle, wherein an average value of centers of gravity of incident angle directivities indicating directivities of the plurality of pixels with respect to the incident angle of the incident light deviates in one direction from a center of the pixel, wherein the center of gravity of the incident angle directivities is a center of gravity of a light intensity distribution of the incident light entering the light receiving surface of the pixel, wherein the average value of the centers of gravity of the incident angle directivities is biased toward the front of the vehicle in a state in which the imaging device is attached to the vehicle; and A display unit displays a restored image restored from a detection image based on the detection signals of the plurality of pixels.

11. The imaging system of claim 10, wherein The plurality of pixels comprises: a plurality of first pixels into which incident light enters at positions different from one another in a first region in which the center of gravity is offset from the center of the pixel toward the front of the vehicle in a state in which the imaging device is attached to the vehicle; and a plurality of second pixels into which incident light enters at positions different from one another in a second region, the second region being a region different from the first region and being a region in which the center of gravity is offset from the center of the pixel in the front / rear direction of the vehicle.

12. The imaging system of claim 11, further comprising A display control unit controls display of a restored image restored from a detection image based on a detection signal from a first pixel and a restored image restored from a detection image based on a detection signal from a second pixel, the display being performed by the display unit.

13. The imaging system of claim 12, wherein The display control unit changes the restored image displayed by the display unit based on at least one of a movement of the vehicle driver or a result of the dangerous object detection.

14. The imaging system of claim 10, further comprising A display control unit controls a range of the restored image displayed by the display unit based on at least one of a movement of a vehicle driver or a result of dangerous object detection.

15. The imaging system of claim 10, wherein The display unit includes: a display surface on which a plurality of display elements are arranged; as well as A plurality of pixels are provided on a display surface, receive incident light entering from an object after passing through neither an imaging lens nor a pinhole, and each pixel outputs a detection signal indicating an output pixel value modulated according to an incident angle of the incident light.

16. The imaging system of claim 15, wherein The detection signals of the plurality of pixels of the display unit are used to detect the sight direction of the vehicle driver.

Images