Image processing device and method, program, and image processing system

Abstract

The present technology pertains to an image processing device and an image processing method, a program, and an image processing system that enable a to-be-restored region to be specified. The present technology specifies a restoration region where a restoration image is to be restored by using a restoration matrix in a region of a detected image obtained in an imaging element provided with a plurality of pixels that receive incident light entering without passing through an imaging lens or a pin hole, the imaging element having a configuration such that the incident angle directions of output pixel values of at least two of the plurality of pixels with respect to incident light from an object have different characteristics from each other. The present disclosure is applicable to, for example, an image processing device, an imaging device, an imaging element, an electronic device, and a system.

Description

[Technical field]

[0001] The present technology relates to an image processing device, method, program, and image processing system, and in particular to an image processing device, method, program, and image processing system that are capable of specifying an area to be restored. [Background technology]

[0002] Conventionally, imaging elements are generally used in combination with imaging lenses that focus light onto the imaging element. The imaging lens guides light from a subject plane to each pixel of the imaging element so as to reproduce the light intensity distribution of the subject plane, and the imaging element can obtain detection signals at levels corresponding to the light intensity distribution at each pixel, thereby obtaining an overall captured image of the subject.

[0003] However, in this case, the physical size becomes large. Therefore, an imaging element that does not use an imaging lens has been considered (for example, see Patent Document 1). An imaging device that uses such an imaging element performs a predetermined calculation on a detected image generated by the imaging element to restore a restored image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2016 / 123529 Summary of the Invention [Problem to be solved by the invention]

[0005] However, it has not been possible to specify an area to be restored, such as to restore some areas of a detected image or not restore others.

[0006] The present disclosure has been made in consideration of such circumstances, and makes it possible to specify an area to be restored. [Means for solving the problem]

[0007] According to one aspect of the present technology, there is provided an image processing device, comprising: a plurality of pixels that receive incident light that is incident without passing through either an imaging lens or a pinhole; and an image sensor configured to make output pixel values ​​of at least two of the plurality of pixels have different incident angle directivities with respect to incident light from a subject; and a region specifying unit that specifies a restoration region that uses a restoration matrix to restore a restored image within a region of a detected image obtained in the image sensor. a partial restoration matrix setting unit that sets a partial restoration matrix used for restoring only a restored image of the restoration area from the detected image by deleting coefficients of an entire restoration matrix that is a restoration matrix corresponding to the entire detected image and that do not correspond to the restoration area specified by the area specifying unit. The image processing device includes:

[0008] According to one aspect of the present technology, there is provided an image processing method, comprising: a plurality of pixels that receive incident light that is incident without passing through an imaging lens or a pinhole; and an image sensor having a configuration for making output pixel values ​​of at least two of the plurality of pixels have different incident angle directivities with respect to incident light from a subject; and a restoration area for restoring a restored image using a restoration matrix within an area of ​​a detected image obtained in the image sensor. and by deleting coefficients that do not correspond to the specified restoration region from among coefficients of an entire restoration matrix, which is a restoration matrix corresponding to the entire detected image, a partial restoration matrix used for restoring only the restored image of the restoration region from the detected image is set. An image processing method.

[0009] A program according to one aspect of the present technology includes a computer, a region specifying unit that specifies a restoration region for restoring a restored image using a restoration matrix within a region of a detected image obtained in an image sensor having a plurality of pixels that receive incident light that is incident without passing through either an imaging lens or a pinhole, the image sensor having a configuration for making output pixel values ​​of at least two of the plurality of pixels have different incident angle directivities with respect to incident light from a subject. a partial restoration matrix setting unit that sets a partial restoration matrix used to restore only a restored image of the restoration area from the detected image by deleting coefficients of an entire restoration matrix that is a restoration matrix corresponding to the entire detected image and that do not correspond to the restoration area specified by the area specifying unit. It is a program that functions as a

[0010] An image processing system according to another aspect of the present technology includes an imaging device and an image processing device, the imaging device includes a plurality of pixels that receive incident light that is incident without passing through an imaging lens or a pinhole, and an image sensor having a configuration for making output pixel values ​​of at least two of the plurality of pixels have different incident angle directivities with respect to incident light from a subject, and the image processing device includes a region specifying unit that specifies a restoration region for restoring a restored image using a restoration matrix within a region of a detected image obtained by the imaging device. a partial restoration matrix setting unit that sets a partial restoration matrix used for restoring only a restored image of the restoration area from the detected image by deleting coefficients of an entire restoration matrix that is a restoration matrix corresponding to the entire detected image and that do not correspond to the restoration area specified by the area specifying unit. The image processing system includes:

[0011] In an image processing device, method, and program according to one aspect of the present technology, a restoration area in which a restoration image is restored using a restoration matrix is ​​specified within an area of ​​a detected image obtained in an image sensor having a configuration in which an image sensor includes a plurality of pixels that receive incident light that is incident without passing through either an imaging lens or a pinhole, and in which the output pixel values ​​of at least two of the plurality of pixels have different incident angle directivities with respect to incident light from a subject, and A partial restoration matrix is ​​set by deleting coefficients that do not correspond to the specified restoration region from among coefficients of the entire restoration matrix, which is a restoration matrix corresponding to the entire detected image, to be used for restoring only the restored image of the restoration region from the detected image. will be done.

[0012] In an image processing system according to another aspect of the present technology, an image capturing device includes an image capturing element having a plurality of pixels that receive incident light that is incident without passing through an imaging lens or a pinhole, and the image capturing element has a configuration for making output pixel values ​​of at least two of the plurality of pixels have different incident angle directivities with respect to incident light from the object, and an image processing device identifies a restoration region in which a restoration image is restored using a restoration matrix within a region of a detected image obtained by the image capturing element. A partial restoration matrix is ​​set by deleting coefficients that do not correspond to the specified restoration region from among coefficients of the entire restoration matrix, which is a restoration matrix corresponding to the entire detected image, to be used for restoring only the restored image of the restoration region from the detected image. will be done. [Brief description of the drawings]

[0013] [Figure 1] 11A and 11B are diagrams illustrating an example of control of a restoration area. [Diagram 2] 1 is a block diagram showing an example of a main configuration of an imaging apparatus; [Diagram 3]FIG. 2 is a diagram illustrating the principle of imaging in an imaging device. [Figure 4] 1A and 1B are diagrams illustrating the difference in configuration between a conventional imaging element and an imaging element according to the present disclosure. [Diagram 5] FIG. 2 is a diagram illustrating a first example of the configuration of an imaging element. [Figure 6] FIG. 2 is a diagram illustrating a first example of the configuration of an imaging element. [Figure 7] 1A and 1B are diagrams for explaining the principle of occurrence of incident angle directivity. [Figure 8] 1A and 1B are diagrams illustrating a change in incident angle directivity using an on-chip lens. [Figure 9] FIG. 1 is a diagram illustrating the design of incident angle directivity. [Figure 10] 11 is a diagram illustrating the relationship between the subject distance and a coefficient expressing the incident angle directivity. FIG. [Figure 11] 4A and 4B are diagrams illustrating the relationship between narrow-angle pixels and wide-angle pixels; [Figure 12] 4A and 4B are diagrams illustrating the relationship between narrow-angle pixels and wide-angle pixels; [Figure 13] 4A and 4B are diagrams illustrating the relationship between narrow-angle pixels and wide-angle pixels; [Figure 14] FIG. 13 is a diagram illustrating a modified example. [Figure 15] FIG. 13 is a diagram illustrating a modified example. [Figure 16] FIG. 13 is a diagram illustrating a modified example. [Figure 17] 13A and 13B are diagrams illustrating an example in which the angle of view is changed by applying a modified example. [Figure 18] 13A and 13B are diagrams illustrating an example in which pixels of a plurality of angles of view are combined when the angle of view is changed by applying a modified example. [Figure 19] FIG. 13 is a diagram illustrating a modified example. [Figure 20] 11A and 11B are diagrams for explaining the reason why the amount of calculation and memory capacity can be reduced by providing rules for the light blocking range in the horizontal and vertical directions, respectively. [Figure 21] 11A and 11B are diagrams for explaining the reason why the amount of calculation and memory capacity can be reduced by providing rules for the light blocking range in the horizontal and vertical directions, respectively. [Figure 22] 11A and 11B are diagrams for explaining the reason why the amount of calculation and memory capacity can be reduced by providing rules for the light blocking range in the horizontal and vertical directions, respectively. [Figure 23] 11A and 11B are diagrams for explaining the reason why the amount of calculation and memory capacity can be reduced by providing rules for the light blocking range in the horizontal and vertical directions, respectively. [Figure 24] FIG. 13 is a diagram illustrating a modified example. [Diagram 25] FIG. 13 is a diagram illustrating a modified example. [Figure 26] FIG. 11 is a diagram illustrating the specification of a restoration area. [Figure 27] FIG. 13 is a diagram illustrating restoration of a restored image. [Figure 28] FIG. 11 is a diagram illustrating restoration of a restoration area. [Figure 29] 13 is a flowchart illustrating an example of the flow of a restored image generating process. [Diagram 30] 1 is a block diagram showing an example of a main configuration of an image processing system; [Diagram 31] 11 is a flowchart illustrating an example of the flow of image processing. [Diagram 32] FIG. 13 is a block diagram showing another example of the configuration of the image processing system. [Diagram 33] 13 is a flowchart illustrating another example of the flow of image processing. [Diagram 34] FIG. 13 is a block diagram showing yet another example configuration of the image processing system. [Diagram 35] 13 is a flowchart illustrating yet another example of the flow of image processing. [Diagram 36] FIG. 13 is a block diagram showing yet another example configuration of the image processing system. [Figure 37] 13 is a flowchart illustrating yet another example of the flow of image processing. [Figure 38] FIG. 13 is a block diagram showing yet another example configuration of the image processing system. [Figure 39] 13 is a flowchart illustrating yet another example of the flow of image processing. [Diagram 40] 1 is a block diagram showing an example of a main configuration of an in-vehicle device; [Diagram 41] 13 is a flowchart illustrating another example of the flow of the restored image generating process. [Diagram 42] FIG. 2 is a diagram illustrating an example of the main configuration of an imaging element. [Diagram 43] FIG. 13 is a diagram showing a case where a black and white pattern mask is used. [Diagram 44] FIG. 13 is a diagram showing a case where a light interference mask is used. [Diagram 45] FIG. 1 is a block diagram showing an example of a main configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described in the following order. 1. Identifying restored and unrestored areas 2. First embodiment (imaging device) 3. Second embodiment (image processing system) 4.Usage example 5. Third embodiment (vehicle-mounted device) 6. Fourth embodiment (another configuration example of the imaging element and imaging device) 7. Notes

[0015] <1. Identification of restored and unrestored areas> Conventionally, there has been an imaging element that does not use an imaging lens, as described in, for example, Patent Document 1. An imaging device that uses such an imaging element performs a predetermined calculation on a detected image generated by the imaging element to restore a restored image. However, it has not been possible to control the restored area of ​​the detected image, such as restoring some areas or not restoring others.

[0016] For example, in a security system that uses a surveillance camera or the like to capture images of public places, monitor for the occurrence of abnormalities, or use the captured images as evidence in the event of an abnormality, there is a risk of infringement of privacy rights due to unauthorized leakage or unauthorized use of the captured images. Also, for example, when a captured image captured on the street includes a stranger and the captured image is made public on a social networking service (SNS) or broadcast by a broadcasting station, there is a risk of infringement of portrait rights.

[0017] In addition, there is a risk that the load of image analysis will increase unnecessarily due to the presence of unnecessary parts in the captured image.Furthermore, for the purpose of service and information management, it is desirable to control whether a captured image is partially public or private (control of the scope of public access) according to the authority of the user who makes the captured image public.

[0018] The above-mentioned detected image is an image that cannot be recognized as an image even when viewed by a user because the image of the subject is not formed (i.e., the subject is not visible). By using this detected image, it is possible to prevent information leakage, unauthorized use, etc., but in order to make the subject visible to the user, it is necessary to restore the restored image from this detected image using a predetermined restoration matrix. And, since the area to be restored cannot be controlled as described above and the entire detected image is restored, there is a risk of information leakage, unauthorized use, etc. in the restored image, as in the above-mentioned captured image. Also, it is not possible to control the range of disclosure based on user authority, etc.

[0019] For example, it is possible to protect information by encrypting a captured image, but the entire captured image must be decrypted when it is used, and there is still a risk of unauthorized leakage or unauthorized use of the decrypted captured image. In addition, when using such encryption, processing such as encoding (encryption) and decryption is required, which may increase the load.

[0020] Therefore, a restoration area for restoring the restored image is set for the detected image. That is, a restoration area for restoring the restored image is specified using a restoration matrix in the area of ​​the detected image obtained by an imaging device having a configuration in which the incident angle directivity of the output pixel values ​​of at least two of the pixels for the incident light from the subject is different from each other. In other words, a non-restoration area for not restoring the restored image is set for the detected image. That is, a non-restoration area for not restoring the restored image is specified using a restoration matrix in the area of ​​the detected image obtained by an imaging device having a configuration in which the incident angle directivity of the output pixel values ​​of at least two of the pixels for the incident light from the subject is different from each other.

[0021] For example, the detection image 51 shown in A of Fig. 1 is an image made up of detection signals obtained by imaging a subject using an imaging element having a configuration in which the output pixel values ​​of at least two of the plurality of pixels have different incident angle directivities with respect to the incident light from the subject, the output pixel values ​​of which are provided with a plurality of pixels that receive incident light that is incident without passing through an imaging lens or a pinhole. As shown in A of Fig. 1, this detection image 51 is an image that cannot be recognized as an image even when viewed by a user (i.e., the subject cannot be visually recognized) because the image of the subject is not formed.

[0022] A restored image is restored from this detected image 51 using a restoration matrix corresponding to this detected image. A restored image 52 in B of Fig. 1 shows an example of a restored image restored from the detected image 51. As shown in B of Fig. 1, persons 53-1, 53-2, and 53-3 appear as subjects in the restored image 52. In other words, persons 53-1 to 53-3 can be visually recognized as subjects in the restored image 52. When it is not necessary to distinguish between persons 53-1 to 53-3, they will be referred to as person 53.

[0023] Here, for example, if the restored image 52 is made public, and the faces of the people 53 are made public, there is a risk of infringement of portrait rights. In such a case, a non-restoration area including the face of each person 53 is set, and the restored image is restored. In other words, the area other than the non-restoration area is set as the restoration area, and the restored image is restored. A restored image 54 shown in C of FIG. 1 is an example of a restored image restored in this way so that the non-restoration area is not restored.

[0024] In the restored image 54, the non-restoration area 55-1 is a non-restoration area set to include the face of the person 53-1. The non-restoration area 55-2 is a non-restoration area set to include the face of the person 53-2. The non-restoration area 55-3 is a non-restoration area set to include the face of the person 53-3. When it is not necessary to distinguish the non-restoration areas 55-1 to 55-3 from one another, they are referred to as non-restoration areas 55. As shown in C of FIG. 1, in the restored image 54, these non-restoration areas 55 are not restored and remain as detected images. That is, in the restored image 54, the face of each person 53 is not visible.

[0025] In this way, the restored image can be made public while keeping the face of each person 53 private. That is, by setting the restored area / non-restored area, it is possible to control whether or not to restore any area of ​​the detected image. This makes it possible to control, for example, whether any area of ​​the detected image is made public / private, and to control whether or not information is protected against unauthorized leakage or use. In addition, for example, it is possible to control the target range of image analysis, etc., and the level of services provided. Note that the restored area (or non-restored area) may be the entire area of ​​the detected image. That is, the entire area of ​​the detected image can be made the restored area or the non-restored area.

[0026] <2. First embodiment> <Imaging device> Fig. 2 is a diagram showing a main configuration example of an imaging device which is an embodiment of an image processing device to which the present technology is applied. The imaging device 100 shown in Fig. 2 is a device that captures an image of a subject and obtains electronic data related to the captured image. As described above in <1. Identification of Restoration Area / Non-Restoration Area>, the imaging device 100 can set a restoration area (non-restoration area) for a detected image and restore a restored image so as not to restore the non-restoration area.

[0027] Note that Fig. 2 shows the main processing units, data flows, and the like, and is not limited to all that is shown in Fig. 2. In other words, in the imaging device 100, there may be processing units that are not shown as blocks in Fig. 2, and there may be processing or data flows that are not shown as arrows or the like in Fig. 2.

[0028] As shown in FIG. 2, the imaging device 100 includes an imaging unit 101, a region specifying unit 102, a learning data storage unit 103, a restoration matrix updating unit 104, a restoration unit 105, and the like.

[0029] The imaging unit 101 has an imaging element 121, which will be described later, and performs processing related to the generation of a detection image. For example, the imaging unit 101 may capture an image of a subject and generate a detection image by the imaging element 121. The imaging element 121 is an imaging element having a configuration in which the output pixel values ​​of at least two of the pixel output units among the plurality of pixel output units have different incident angle directivities with respect to the incident light from the subject. Details of the imaging element 121 will be described later. The imaging unit 101 may also supply the generated detection image to the region specifying unit 102 and the restoration unit 105. Furthermore, the imaging unit 101 may supply a restoration matrix corresponding to the detection image, that is, used to generate a restoration image from the detection image, to the restoration matrix updating unit 104. This restoration matrix may be stored in advance by the imaging unit 101, or may be generated in the imaging unit 101.

[0030] The angle of view of the image sensor 121 may be fixed or variable in an absolute coordinate system in a three-dimensional space. For example, the position and orientation of the image sensor 121 (or the imaging unit 101) may be fixed or variable with respect to the imaging device 100. Also, the imaging device 100 may be fixed or variable in an absolute coordinate system in a three-dimensional space.

[0031] The region identifying unit 102 performs processing related to identification of a restoration region (or a non-restoration region). For example, the region identifying unit 102 may acquire a detection image supplied from the imaging unit 101. The region identifying unit 102 may also acquire learning data stored in the learning data storage unit 103. The region identifying unit 102 may further set a restoration region (or a non-restoration region) for the detection image supplied from the imaging unit 101, i.e., the detection image obtained by the image sensor 121, based on the learning data read from the learning data storage unit 103. The region identifying unit 102 may also generate restoration region information indicating the restoration region (or non-restoration region information indicating the non-restoration region), and supply the restoration region information to the restoration matrix update unit 104.

[0032] The learning data storage unit 103 performs processing related to storage of learning data. For example, the learning data storage unit 103 has an arbitrary storage medium, and stores learning data related to identification of a restoration region. This learning data will be described later. The learning data storage unit 103 can supply the stored learning data to the region identification unit 102, for example, in response to a request from the region identification unit 102.

[0033] The restoration matrix update unit 104 performs processing related to updating the restoration matrix. For example, the restoration matrix update unit 104 may obtain restoration region information (or non-restoration region information) supplied from the region identification unit 102. The restoration matrix update unit 104 may also obtain a restoration matrix supplied from the imaging unit 101. This restoration matrix is ​​a restoration matrix corresponding to a detected image generated in the imaging unit 101 (used to restore a restored image).

[0034] The restoration matrix update unit 104 may update the restoration matrix acquired from the imaging unit 101 based on the restoration area information, and generate a restoration matrix (also referred to as a non-restoration area corresponding restoration matrix) for restoring only the restoration area without restoring the non-restoration area. In other words, the restoration matrix update unit 104 is a partial restoration matrix generation unit that generates a partial restoration matrix used to restore only the restoration image of the specified restoration area from the detection image. This non-restoration area is an area determined based on the restoration area indicated by the restoration area information, and is an area of ​​the detection image other than the restoration area. Note that, when non-restoration area information is supplied from the area identification unit 102, the restoration matrix update unit 104 may update the restoration matrix acquired from the imaging unit 101 based on the non-restoration area information, and generate a non-restoration area corresponding restoration matrix for restoring only the restoration area without restoring the non-restoration area indicated by the non-restoration area information. This restoration area is an area determined based on the non-restoration area indicated by the non-restoration area information, and is an area of ​​the detection image other than the non-restoration area.

[0035] The restoration matrix update unit 104 may also supply the restoration unit 105 with a restoration matrix corresponding to the non-restoration area, which is the restoration matrix after the update. The restoration matrix update unit 104 may set a partial restoration matrix (also referred to as a restoration matrix corresponding to the non-restoration area). For example, it is previously determined that when a face is detected in the upper left of the detected image, the entire left half is not restored, and a restoration matrix that does not restore only the left half (a dedicated matrix when the left half is not restored) is previously prepared (for example, stored in a storage area in the restoration matrix update unit 104), and the restoration matrix update unit 104 sets the restoration matrix corresponding to the non-restoration area to be applied when a face is detected in the upper left of the detected image based on the restoration area information. In other words, the restoration matrix update unit 104 may set whether or not to apply the restoration matrix corresponding to the non-restoration area prepared in advance, without generating a restoration matrix corresponding to the non-restoration area. When applying a restoration matrix corresponding to the non-restoration area prepared in advance, the restoration matrix update unit 104 may select one of a plurality of restoration matrices corresponding to the non-restoration area prepared in advance based on the restoration area information, and apply it. In other words, the restoration matrix update unit 104 can also be said to be a partial restoration matrix setting unit that sets a partial restoration matrix used for restoring only the restored image of the specified restoration area from the detected image.

[0036] The restoration unit 105 performs a process related to the restoration of the restored image. For example, the restoration unit 105 may acquire a detected image provided from the imaging unit 101. The restoration unit 105 may also acquire a non-restored area corresponding restoration matrix provided from the restoration matrix update unit 104. The restoration unit 105 may also restore the restored image using the non-restored area corresponding restoration matrix. This process may generate a restored image in which only the restored area is restored from the detected image. The restoration unit 105 may also output the generated restored image (a restored image in which only the restored area is restored from the detected image) to the outside of the imaging device 100. Note that the restoration of this restored image may be performed outside the imaging device 100. In this case, the restoration unit 105 does not perform this restoration, but associates the detected image with the non-restored area corresponding restoration matrix and outputs it to a restoration device that performs restoration processing outside the imaging device 100. In this way, the imaging device 100 can transmit an image to a restoration device (another device) in a state in which information is protected. Therefore, it is possible to prevent the occurrence of violations of privacy rights due to the unauthorized leakage or unauthorized use of images.

[0037] The information may be output to the outside by communication via any communication path, or may be output via any storage medium (i.e., the information may be stored in a storage medium and then output).

[0038] Furthermore, the term "associate" means, for example, making one piece of information (data, commands, programs, etc.) available (linked) when processing the other piece of information. In other words, pieces of information associated with each other may be collected together as a single file or the like, or each piece of information may be individual pieces of information. For example, information B associated with information A may be transmitted over a different transmission path from that of information A. Also, for example, information B associated with information A may be recorded on a different recording medium from that of information A (or on a different recording area of ​​the same recording medium).

[0039] As described above, the area specifying unit 102 can control the area to be restored by specifying the restoration area (non-restoration area). This allows the imaging device 100 to restore the restored image so as not to restore any area.

[0040] Each of these processing units (imaging unit 101 to restoration unit 105) of the imaging device 100 may have any configuration. For example, each processing unit may be configured with a logic circuit that realizes the above-mentioned processing. Also, each processing unit may have, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and may realize the above-mentioned processing by executing a program using them. Of course, each processing unit may have both configurations, and may realize part of the above-mentioned processing by a logic circuit and the other by executing a program. The configurations of each processing unit may be independent of each other, and for example, some processing units may realize part of the above-mentioned processing by a logic circuit, other processing units may realize the above-mentioned processing by executing a program, and still other processing units may realize the above-mentioned processing by both a logic circuit and executing a program.

[0041] <About the image sensor> Next, the imaging element 121 of the imaging unit 101 will be described with reference to FIGS.

[0042] <Pixels and pixel output units> In this specification, the term "pixel" (or "pixel output unit") is used to describe this technology. In this specification, the term "pixel" (or "pixel output unit") refers to a division unit including at least one physical configuration capable of receiving light independently of other pixels in an area (also referred to as a pixel area) in which a physical configuration for receiving incident light of the image sensor 121 is formed. The physical configuration capable of receiving light is, for example, a photoelectric conversion element, such as a photodiode (PD (Photo Diode)). The number of such physical configurations (for example, photodiodes) formed in one pixel is arbitrary, and may be one or more. The type, size, shape, etc. of the physical configuration are also arbitrary.

[0043] The physical configuration of the "pixel" unit includes not only the above-mentioned "physical configuration capable of receiving light" but also all physical configurations related to receiving incident light, such as an on-chip lens, a light-shielding film, a color filter, a planarizing film, an anti-reflection film, etc. Furthermore, it may also include configurations such as a readout circuit. In other words, the physical configuration of the pixel unit may be any configuration.

[0044] A detection signal read out from a "pixel" (i.e., the physical configuration of a pixel unit) may also be referred to as a "pixel unit (or pixel output unit) detection signal" or the like. Furthermore, this pixel unit (or pixel output unit) detection signal is also referred to as a "pixel unit detection signal (or pixel output unit detection signal)." This pixel unit detection signal is also referred to as a "pixel output." Furthermore, its value is also referred to as an "output pixel value."

[0045] The value of the detection signal (output pixel value) of each pixel of the image sensor 121 can have an incident angle directivity (characteristics with respect to the incident angle) of the incident light from the subject independent of the others. That is, each pixel unit (pixel output unit) of the image sensor 121 has a configuration in which the incident angle directivity of the output pixel value with respect to the incident light from the subject can be set independently. Therefore, in the image sensor 121, the output pixel values ​​of at least two pixel units have mutually different incident angle directivities with respect to the incident light from the subject.

[0046] As described above, a "pixel (or pixel output unit)" may have any number of "physical structures capable of receiving light," so the pixel unit detection signal may be a detection signal obtained from a single "physical structure capable of receiving light," or may be a detection signal obtained from multiple "physical structures capable of receiving light."

[0047] In addition, the pixel unit detection signals (output pixel values) can be combined into one at any stage. For example, the output pixel values ​​of multiple pixels may be added in the form of analog signals, or may be converted into digital signals and then added.

[0048] In addition, after the detection signals are read out from the image sensor 121, that is, in the detection image, a plurality of detection signals can be collectively made into a singular signal, or a singular detection signal can be made into a plurality of signals. In other words, the resolution (number of data) of the detection image is variable.

[0049] For convenience of explanation, the imaging element 121 will be described below as having a pixel region in which a plurality of pixels are arranged in a matrix (a pixel array is formed) unless otherwise specified. The arrangement pattern of the pixels (or pixel output units) of the imaging element 121 is arbitrary and is not limited to this example. For example, the pixels (or pixel output units) may be arranged in a honeycomb structure. Also, for example, the pixels (or pixel output units) may be arranged in a single row (or single column). In other words, the imaging element 121 may be a line sensor.

[0050] The wavelength range to which the image sensor 121 (its pixels) has sensitivity is arbitrary. For example, the image sensor 121 (its pixels) may be sensitive to visible light, may be sensitive to invisible light such as infrared light or ultraviolet light, or may be sensitive to both visible light and invisible light.

[0051] <Incident angle directivity> The imaging element 121 has a plurality of pixel output units. The imaging element 121 has a configuration for making the output pixel values ​​of at least two of the plurality of pixel output units have different incident angle directivities with respect to incident light from a subject. In other words, the imaging element 121 can obtain detection signals for the plurality of pixel output units (a plurality of pixel output unit detection signals), and the incident angle directivities with respect to incident light from a subject of at least two of the pixel output unit detection signals are different from each other.

[0052] Here, "incident angle directivity" refers to the light receiving sensitivity characteristic according to the incident angle of the incident light, that is, the detection sensitivity for the incident angle of the incident light. For example, even if the incident light has the same light intensity, the detection sensitivity may change depending on the incident angle. Such a bias in detection sensitivity (including the case where there is no bias) is called "incident angle directivity".

[0053] For example, when incident light having the same light intensity is incident on the physical configurations of the two pixel output units at the same incident angle, the signal levels (detection signal levels) of the detection signals of the pixel output units may be different from each other. The image sensor 121 (each pixel output unit) has a physical configuration with such characteristics.

[0054] More specifically, the imaging element 121 may be similar in basic structure to an imaging element such as a general CMOS (Complementary Metal Oxide Semiconductor) image sensor, but has a configuration for making the incident angle directivity of the output pixel value with respect to the incident light from the subject different from each other, as described above, between at least two pixel output units. This incident angle directivity may be realized by any method. For example, the incident angle directivity may be realized by providing a light shielding film in front of the photoelectric conversion element (photodiode, etc.) (on the light incident side).

[0055] When imaging is performed only with an imaging element consisting of pixels with the same general incidence angle directivity, light of approximately the same light intensity is incident on all pixels of the imaging element, and an image of the object cannot be obtained. Therefore, an imaging lens or a pinhole is generally provided in front of the imaging element (on the light incident side). For example, by providing an imaging lens, light from the object plane can be imaged on the imaging surface of the imaging element. Therefore, the imaging element can obtain a detection signal of a level corresponding to the image of the object imaged at each pixel (that is, an image of the object imaged can be obtained). However, in this case, the physical size becomes large, and there is a concern that it becomes difficult to miniaturize the device. In addition, when a pinhole is provided, it is possible to reduce the size compared to when an imaging lens is provided, but the amount of light incident on the imaging element is reduced, so measures such as lengthening the exposure time or increasing the gain are necessary, and there is a concern that blurring is likely to occur in imaging of an object at high speed, or that natural color expression will not be obtained.

[0056] In contrast, the image sensor 121 has an incident angle directivity in which the detection sensitivity of each pixel is different from one another, as shown in the upper left part of Fig. 3. In other words, the light receiving sensitivity characteristic according to the incident angle of the incident light is different for each pixel. However, it is not necessary that the light receiving sensitivity characteristics of all pixels are completely different, and some pixels may have the same light receiving sensitivity characteristics, or some pixels may have different light receiving sensitivity characteristics.

[0057] Assuming that the light source constituting the object plane 131 is a point light source, in the image sensor 121, light rays of the same light intensity emitted from the same point light source are incident on all pixels, but at different angles of incidence for each pixel. Since each pixel of the image sensor 121 has different incident angle directivities, the pixels detect the light rays of the same light intensity with different sensitivities. In other words, detection signals of different signal levels are detected for each pixel.

[0058] More specifically, the sensitivity characteristic according to the angle of incidence of the incident light received at each pixel of the image sensor 121, i.e., the incident angle directivity according to the angle of incidence at each pixel, is expressed by a coefficient representing the light receiving sensitivity according to the angle of incidence, and the signal level of the detection signal according to the incident light at each pixel (also referred to as the detection signal level) is obtained by multiplying the coefficient set corresponding to the light receiving sensitivity according to the angle of incidence of the incident light.

[0059] More specifically, as shown in the upper left part of FIG. 3, detection signal levels DA, DB, and DC at positions Pa, Pb, and Pc are respectively expressed by the following equations (1) to (3).

[0060] DA = α1×a + β1×b + γ1×c (1) DB = α2 × a + β2 × b + γ2 × c (2) DC = α3×a + β3×b + γ3×c (3)

[0061] Here, α1 is a coefficient set according to the angle of incidence of a ray from a point light source PA on object plane 131 to be restored at position Pa on image sensor 121. β1 is a coefficient set according to the angle of incidence of a ray from a point light source PB on object plane 131 to be restored at position Pa on image sensor 121. γ1 is a coefficient set according to the angle of incidence of a ray from a point light source PC on object plane 131 to be restored at position Pa on image sensor 121.

[0062] As shown in equation (1), the detection signal level DA at position Pa is expressed by the sum (composite value) of the product of the light intensity "a" of the light beam from point light source PA at position Pa and coefficient α1, the product of the light intensity "b" of the light beam from point light source PB at position Pa and coefficient β1, and the product of the light intensity "c" of the light beam from point light source PC at position Pa and coefficient γ1. Hereinafter, the coefficients αx, βx, and γx (x is a natural number) will be collectively referred to as a coefficient set.

[0063] Similarly, the coefficient set α2, β2, and γ2 in formula (2) are coefficient sets that are set according to the angles of incidence of light rays from point light sources PA, PB, and PC on the object plane 131 to be restored at the position Pb on the image sensor 121. That is, the detection signal level DB at the position Pb is expressed by the sum (composite value) of the product of the light intensity "a" of the light from the point light source PA at the position Pb and the coefficient α2, the product of the light intensity "b" of the light from the point light source PB at the position Pb and the coefficient β2, and the product of the light intensity "c" of the light from the point light source PC at the position Pb and the coefficient γ2, as in the above formula (2). Also, the coefficients α3, β3, and γ3 in formula (3) are coefficient sets that are set according to the angles of incidence of light rays from the point light sources PA, PB, and PC on the object plane 131 to be restored at the position Pc on the image sensor 121. In other words, the detection signal level DC at position Pc is expressed as the sum (composite value) of the product of the light intensity "a" of the light beam from point light source PA at position Pc and coefficient α3, the product of the light intensity "b" of the light beam from point light source PB at position Pc and coefficient β3, and the product of the light intensity "c" of the light beam from point light source PC at position Pc and coefficient γ3, as in the above-mentioned equation (3).

[0064] As described above, these detection signal levels are a mixture of the light intensities of the light rays emitted from the point light sources PA, PB, and PC, and are therefore different from the image of the subject that is formed. In other words, the detection signal level shown in the upper right part of Fig. 3 is not a detection signal level that corresponds to the image (captured image) in which the image of the subject is formed, and is therefore different from the pixel value shown in the lower right part of Fig. 3 (the two generally do not match).

[0065] However, by forming simultaneous equations using the coefficient sets α1, β1, γ1, coefficient sets α2, β2, γ2, coefficient sets α3, β3, γ3, and detection signal levels DA, DB, DC, and solving the simultaneous equations of the above-mentioned formulas (1) to (3) with a, b, and c as variables, it is possible to obtain pixel values ​​at each position Pa, Pb, and Pc as shown in the lower right part of Fig. 3. This allows a restored image (an image in which the image of the subject is formed) which is a set of pixel values ​​to be restored.

[0066] With this configuration, the image sensor 121 has different incident angle directivities in at least two pixel output units without requiring an imaging lens, an optical filter made of a diffraction grating, etc., a pinhole, etc. As a result, since an imaging lens, an optical filter made of a diffraction grating, etc., a pinhole, etc. are not essential components, it is possible to reduce the height of the imaging device, that is, to reduce the thickness in the light incident direction in a configuration that realizes an imaging function.

[0067] <Formation of incident angle directivity> The left part of Fig. 4 shows a front view of a part of a pixel array section of a general image sensor, and the right part of Fig. 4 shows a front view of a part of the pixel array section of image sensor 121. Note that Fig. 4 shows an example in which the pixel array section has a configuration of 6 pixels x 6 pixels in each of the horizontal and vertical directions, but the pixel number configuration is not limited to this.

[0068] The incident angle directivity can be formed by, for example, a light-shielding film. A typical image sensor 151 is shown in the example on the left side of Fig. 4, in which pixels 151a having the same incident angle directivity are arranged in an array. In contrast, the image sensor 121 in the example on the right side of Fig. 4 is configured such that a light-shielding film 121b is provided for each pixel 121a in a part of the light receiving area of ​​the photodiode, which is different for each pixel 121a, and the light receiving sensitivity to the incident angle of incident light is different for each pixel 121a (i.e., the incident angle directivity is different from one another).

[0069] More specifically, for example, the pixel 121a-1 and the pixel 121a-2 have different ranges of pixels that are shielded from light by the light shielding films 121b-1 and 121b-2 (at least one of the light shielding region (position) and the light shielding area is different). That is, in the pixel 121a-1, the light shielding film 121b-1 is provided so as to shield a part of the left side of the light receiving region of the photodiode by a predetermined width, and in the pixel 121a-2, the light shielding film 121b-2 is provided so as to shield a part of the right side of the light receiving region by a width wider in the horizontal direction than the light shielding film 121b-1. Similarly, in the other pixels 121a, the light shielding films 121b are provided so as to shield different ranges of the light receiving region for each pixel, and are arranged randomly in the pixel array.

[0070] In addition, the area of ​​the light-shielding film 121b is preferably set to an area that can secure a desired amount of light, since the amount of light that can be received decreases as the proportion of the light-receiving region of each pixel that is covered increases, and the area of ​​the light-shielding film 121b may be limited to, for example, about 3 / 4 of the entire range that can be received at most. In this way, it is possible to secure an amount of light greater than the desired amount. However, if an unshielded area with a width equivalent to the wavelength of the light to be received is provided for each pixel, it is possible to receive a minimum amount of light. That is, for example, in the case of a B pixel (blue pixel), the wavelength is about 500 nm, but if the area is not shied beyond the width equivalent to this wavelength, it is possible to receive a minimum amount of light.

[0071] <Example of imaging element configuration> An example of the configuration of the imaging element 121 in this case will be described with reference to Fig. 5. The upper part of Fig. 5 is a side cross-sectional view of the imaging element 121, and the middle part of Fig. 5 is a top view of the imaging element 121. The side cross-sectional view in the upper part of Fig. 5 is a cross-section taken along line AB in the middle part of Fig. 5. Furthermore, the lower part of Fig. 5 shows an example of the circuit configuration of the imaging element 121.

[0072] 5 includes a plurality of pixel output units that receive incident light that does not pass through an imaging lens or a pinhole, and is configured so that the output pixel values ​​of at least two of the plurality of pixel output units have different incident angle directivities with respect to incident light from a subject. Also, the image sensor 121 in this case has a configuration in which the plurality of pixel output units can set the incident angle directivity with respect to incident light from a subject independently for each pixel output unit.

[0073] In the image sensor 121 in the upper part of Fig. 5, incident light is incident from the top to the bottom in the figure. Adjacent pixels 121a-15 and 121a-16 are each provided with a wiring layer Z12 in the bottom layer in the figure, and a photoelectric conversion layer Z11 is provided thereon, which is a so-called back-illuminated type.

[0074] In addition, when there is no need to distinguish between the pixels 121a-15 and 121a-16, they are simply referred to as pixels 121a, and other configurations are also referred to in the same manner. In addition, Fig. 5 shows a side view and a top view of two pixels that constitute the pixel array of the image sensor 121, but it goes without saying that more pixels 121a are arranged but are not shown.

[0075] Furthermore, the pixels 121a-15 and 121a-16 are provided with photodiodes 121e-15 and 121e-16 in the photoelectric conversion layer Z11, respectively. Moreover, on-chip lenses 121c-15 and 121c-16 and color filters 121d-15 and 121d-16 are respectively arranged on the photodiodes 121e-15 and 121e-16 from the top.

[0076] The on-chip lenses 121c-15 and 121c-16 focus the incident light onto the photodiodes 121e-15 and 121e-16.

[0077] The color filters 121d-15 and 121d-16 are optical filters that transmit light of specific wavelengths, such as red, green, blue, infrared, and white. In the case of white, the color filters 121d-15 and 121d-16 may be transparent filters or may not be present.

[0078] In the photoelectric conversion layer Z11 of the pixels 121a-15 and 121a-16, light-shielding films 121p-15 to 121p-17 are formed at the boundaries between the pixels, respectively, to suppress crosstalk between adjacent pixels.

[0079] 5, the light-shielding films 121b-15 and 121b-16 partially shield the light-receiving surface S when viewed from above. Different ranges are shielded from light on the light-receiving surfaces S of the photodiodes 121e-15 and 121e-16 in the pixels 121a-15 and 121a-16 by the light-shielding films 121b-15 and 121b-16, respectively, and different incident angle directivities are set for each pixel. However, the light-shielded ranges are not limited to being different for all pixels 121a of the image sensor 121, and there may be pixels 121a in which the same range is partially shielded from light.

[0080] With the configuration shown in the upper part of Figure 5, the right end of the light-shielding film 121p-15 is connected to the upper end of the light-shielding film 121b-15, and the left end of the light-shielding film 121b-16 is connected to the upper end of the light-shielding film 121p-16, forming an L-shape when viewed from the side.

[0081] Furthermore, the light-shielding films 121b-15 to 121b-17 and the light-shielding films 121p-15 to 121p-17 are made of a metal, for example, tungsten (W), aluminum (Al), or an alloy of Al and copper (Cu). The light-shielding films 121b-15 to 121b-17 and the light-shielding films 121p-15 to 121p-17 may be formed simultaneously with the wiring by the same process as that for forming the wiring in the semiconductor process, using the same metal. The film thicknesses of the light-shielding films 121b-15 to 121b-17 and the light-shielding films 121p-15 to 121p-17 do not have to be the same depending on the position.

[0082] Also, as shown in the lower part of Figure 5, pixel 121a is configured to include a photodiode 161 (corresponding to photodiode 121e), a transfer transistor 162, an FD (Floating Diffusion) section 163, a selection transistor 164, an amplification transistor 165, and a reset transistor 166, and is connected to a current source 168 via a vertical signal line 167.

[0083] The photodiodes 161 have their anode electrodes grounded and their cathode electrodes connected to the gate electrodes of the amplifier transistors 165 via the transfer transistors 162 .

[0084] The transfer transistors 162 are driven in accordance with a transfer signal TG. For example, when the transfer signal TG supplied to the gate electrode of the transfer transistor 162 becomes high level, the transfer transistor 162 is turned on. As a result, the charge accumulated in the photodiode 161 is transferred to the FD section 163 via the transfer transistor 162.

[0085] The amplifying transistor 165 serves as an input section of a source follower, which is a readout circuit that reads out a signal obtained by photoelectric conversion in the photodiode 161, and outputs a pixel signal at a level corresponding to the charge accumulated in the FD section 163 to the vertical signal line 23. That is, the amplifying transistor 165 has a drain terminal connected to a power supply voltage VDD and a source terminal connected to the vertical signal line 167 via the selection transistor 164, thereby constituting a source follower together with a current source 168 connected to one end of the vertical signal line 167.

[0086] The FD (Floating Diffusion) section 163 is a floating diffusion region having a charge capacitance C1 provided between the transfer transistor 162 and the amplification transistor 165, and temporarily stores the charge transferred from the photodiode 161 via the transfer transistor 162. The FD section 163 is a charge detection section that converts charge into a voltage, and the charge stored in the FD section 163 is converted into a voltage in the amplification transistor 165.

[0087] The selection transistor 164 is driven in accordance with a selection signal SEL, and is turned on when the selection signal SEL supplied to the gate electrode becomes high level, thereby connecting the amplification transistor 165 and the vertical signal line 167 .

[0088] The reset transistor 166 is driven in accordance with a reset signal RST. For example, when the reset signal RST supplied to the gate electrode of the reset transistor 166 becomes high level, the reset transistor 166 turns on and discharges the charge accumulated in the FD section 163 to the power supply voltage VDD, thereby resetting the FD section 163.

[0089] With the above circuit configuration, the pixel circuit shown in the lower part of FIG. 5 operates as follows.

[0090] That is, in a first operation, the reset transistor 166 and the transfer transistor 162 are turned on, and the charge stored in the FD unit 163 is discharged to the power supply voltage VDD, thereby resetting the FD unit 163.

[0091] In the second operation, the reset transistor 166 and the transfer transistor 162 are turned off, an exposure period begins, and the photodiode 161 accumulates electric charges according to the amount of incident light.

[0092] As a third operation, the reset transistor 166 is turned on to reset the FD unit 163, and then the reset transistor 166 is turned off. By this operation, the FD unit 163 is reset and set to the reference potential.

[0093] As a fourth operation, the potential of the FD unit 163 in the reset state is output from the amplification transistor 165 as a reference potential.

[0094] As a fifth operation, the transfer transistor 162 is turned on, and the charge accumulated in the photodiode 161 is transferred to the FD section 163 .

[0095] In a sixth operation, the potential of the FD section 163 to which the charge of the photodiode has been transferred is output as a signal potential from the amplifying transistor 165 .

[0096] Through the above process, the reference potential is subtracted from the signal potential, and the result is output as a detection signal by CDS (correlated double sampling).

[0097] In this way, in the case of FIG. 5, one photodiode 121e is provided for each pixel 121a, and a different range for each pixel 121a is shielded from light by the light-shielding film 121b, and a detection signal for one pixel of a detection image having incident angle directivity can be expressed by one pixel 121a by optical modulation using the light-shielding film 121b.

[0098] <Other configuration examples of image sensor> In addition, the incident angle directivity can be formed by, for example, the position, size, shape, etc. of the light receiving element (e.g., photodiode) in the pixel. Pixels with different parameters have different sensitivities to incident light of the same light intensity from the same direction. In other words, by setting these parameters for each pixel, the incident angle directivity can be set for each pixel.

[0099] For example, a pixel may be provided with a plurality of light receiving elements (e.g., photodiodes) that can be selectively used. In this way, the incident angle directivity can be set for each pixel by selecting the light receiving element.

[0100] Fig. 6 is a diagram showing another example of the configuration of the imaging element 121. The upper part of Fig. 6 shows a side cross-sectional view of a pixel 121a of the imaging element 121, and the middle part of Fig. 6 shows a top view of the imaging element 121. The side cross-sectional view in the upper part of Fig. 6 is the cross-section taken along line AB in the middle part of Fig. 6. Furthermore, the lower part of Fig. 6 shows an example of the circuit configuration of the imaging element 121.

[0101] 6 includes a plurality of pixel output units that receive incident light that does not pass through an imaging lens or a pinhole, and is configured to make the output pixel values ​​of at least two of the plurality of pixel output units have different incident angle directivities with respect to incident light from a subject. In addition, the image sensor 121 in this case can set the incident angle directivity of the output pixel values ​​independently for each pixel output unit by making the PDs (Photo Diodes) that contribute to the output different from each other.

[0102] As shown in Fig. 6, the imaging element 121 has a different configuration from the imaging element 121 in Fig. 6 in that four photodiodes 121f-1 to 121f-4 are formed in the pixel 121a, and a light-shielding film 121p is formed in a region that separates the photodiodes 121f-1 to 121f-4 from each other. That is, in the imaging element 121 in Fig. 6, the light-shielding film 121p is formed in a "+" shape when viewed from above. Note that the same reference numerals are used for the common configurations, and detailed description thereof will be omitted.

[0103] In the imaging element 121 configured as shown in Fig. 6, the photodiodes 121f-1 to 121f-4 are separated by the light-shielding film 121p, so that electrical and optical crosstalk between the photodiodes 121f-1 to 121f-4 can be prevented. That is, the light-shielding film 121p in Fig. 6 is for preventing crosstalk, similar to the light-shielding film 121p of the imaging element 121 in Fig. 5, and is not for imparting incident angle directivity.

[0104] Although details will be described later, the photodiodes 121f-1 to 121f-4 have different incident angles at which their light receiving sensitivity characteristics become high. In other words, the output pixel value of the pixel 121a can have a desired incident angle directivity depending on which of the photodiodes 121f-1 to 121f-4 the charge is read from. In other words, the incident angle directivity of the output pixel value of the pixel 121a can be controlled.

[0105] In the configuration example of the image sensor 121 in Fig. 6, one FD section 163 is shared by four photodiodes 121f-1 to 121f-4. The lower part of Fig. 6 shows a circuit configuration example in which one FD section 163 is shared by four photodiodes 121f-1 to 121f-4. Note that in the lower part of Fig. 6, the description of the same configuration as in the lower part of Fig. 5 will be omitted.

[0106] The circuit configuration in the lower part of Figure 6 differs from that in the lower part of Figure 5 in that, instead of the photodiode 161 and the transfer transistor 162, photodiodes 161-1 to 161-4 (corresponding to photodiodes 121f-1 to 121f-4 in the upper part of Figure 6) and transfer transistors 162-1 to 162-4 are provided, and the FD section 163 is shared.

[0107] 6, when it is not necessary to distinguish between the photodiodes 161-1 to 161-4, they will be referred to as photodiodes 161. When it is not necessary to distinguish between the transfer transistors 162-1 to 162-4, they will be referred to as transfer transistors 162.

[0108] In the circuit shown in the lower part of FIG. 6, when any of the transfer transistors 162 is turned on, the charge of the photodiode 161 corresponding to the transfer transistor 162 is read out and transferred to a common FD section 163. Then, a signal according to the level of the charge held in the FD section 163 is read out as a detection signal for each pixel output. That is, the charge of each photodiode 161 can be read out independently of each other, and it is possible to control which photodiode 161 the charge is read out from by turning on which transfer transistor 162. In other words, it is possible to control the degree of contribution of each photodiode 161 to the output pixel value by turning on which transfer transistor 162. For example, by making the photodiodes 161 that read out the charge different from each other between at least two pixels, it is possible to make the photodiodes 161 that contribute to the output pixel value different from each other. That is, by selecting the photodiode 161 that reads out the charge, it is possible to give the output pixel value of the pixel 121a a desired incident angle directivity.

[0109] 6, for example, the charges of the photodiodes 121f-1 and 121f-3 are transferred to the FD section 163 and the signals obtained by reading out the charges are added together, thereby giving the output pixel value of the pixel 121a incident angle directivity in the left-right direction in the figure. Similarly, the charges of the photodiodes 121f-1 and 121f-2 are transferred to the FD section 163 and the signals obtained by reading out the charges are added together, thereby giving the output pixel value of the pixel 121a incident angle directivity in the up-down direction in the figure.

[0110] The signals obtained based on the charges of the photodiodes 121f of the pixel 121a in FIG. 6 may be added after being read out from the pixel, or may be added within the pixel (for example, in the FD unit 163).

[0111] In addition, the combination of photodiodes 121f for adding up the charges (or the signals corresponding to the charges) is arbitrary and is not limited to the above example. For example, the charges (or the signals corresponding to the charges) of three or more photodiodes 121f may be added up. Also, for example, the charge of one photodiode 121f may be read out without addition.

[0112] In addition, the detection value (charge) accumulated in the photodiode 161 (photodiode 121f) may be reset using an electronic shutter function before the charge is read out to the FD section 163, thereby giving the pixel 121a (the detection sensitivity) a desired incident angle directivity.

[0113] When using the electronic shutter function, if the photodiode 121f is reset just before the charge is read out to the FD section 163, the photodiode 121f can be put into a state where it does not contribute to the detection signal level of the pixel 121a, and if there is a time between the reset and the readout to the FD section 163, it can be made to contribute partially.

[0114] As described above, the pixel 121a in FIG. 6 is provided with four photodiodes 121f, and the light-receiving surface is not formed with the light-shielding film 121b. However, the pixel is divided into a plurality of regions by the light-shielding film 121p, and four photodiodes 121f-1 to 121f-4 are formed, and a detection signal for one pixel of a detection image having incident angle directivity is expressed. In other words, for example, the range of the photodiodes 121f-1 to 121f-4 that does not contribute to the output functions similarly to a light-shielded region, and a detection signal for one pixel of a detection image having incident angle directivity is expressed. Note that, when the detection signal for one pixel is expressed using the photodiodes 121f-1 to 121f-4, the light-shielding film 121b is not used, and therefore the detection signal is not a signal obtained by optical modulation.

[0115] Although an example in which four photodiodes are arranged in a pixel has been described above, the number of photodiodes arranged in a pixel is arbitrary and is not limited to the above example. In other words, the number of partial regions in which photodiodes are arranged in a pixel is also arbitrary.

[0116] In the above, the photodiodes are arranged in four partial regions obtained by dividing a pixel into four equal parts, but the partial regions do not have to be equally divided. In other words, the sizes and shapes of the partial regions do not have to be uniform (partial regions may include partial regions whose sizes and shapes are different from the others). Alternatively, the positions (positions within the partial regions), sizes, shapes, etc. of the photodiodes arranged in each partial region may differ for each photodiode (partial region). In this case, the sizes and shapes of the partial regions may or may not be uniform.

[0117] Furthermore, these parameters do not have to be uniform for all pixels of the imaging element 121. That is, for one or more pixels of the imaging element 121, one or more of these parameters may be different from other pixels.

[0118] For example, a pixel having a division position for forming a partial region in which a photodiode is arranged within the pixel that is different from other pixels may be included in the pixel group of the image sensor 121. That is, the image sensor 121 may have one or more pixels having a partial region whose size or shape is different from other pixels. For example, by making the division position different for each pixel, even if only the upper left photodiode is used among the multiple pixels, the incident angle directivities of the detection signals detected in each of the multiple pixels can be made different from each other.

[0119] Also, for example, a pixel in which the position, size, shape, etc. of a plurality of photodiodes arranged within the pixel are different from other pixels may be included in the pixel group of the image sensor 121. In other words, the image sensor 121 may have one or more pixels in which at least one of the position, size, and shape of the plurality of photodiodes arranged therein is different from other pixels. For example, by making the position, size, shape, etc. of the photodiode different for each pixel, even if only the upper left photodiode is used among the plurality of pixels, the incident angle directivities of the detection signals detected in each of the plurality of pixels can be made different from each other.

[0120] Furthermore, for example, one or more pixels may be provided that are different from other pixels in both the parameters (size, shape) of the partial region and the parameters (position, size, shape) of the photodiode.

[0121] Also, for example, a pixel having a different number of divisions for forming a partial region in which photodiodes are arranged within the pixel from other pixels may be included in the pixel group of the image sensor 121. That is, the image sensor 121 may have one or more pixels having a different number of arranged photodiodes from other pixels. For example, by making the number of divisions (number of photodiodes) different for each pixel, the incident angle directivity can be set more freely.

[0122] <The principle behind incidence angle directivity> The incident angle directivity of each pixel in the image sensor 121 occurs, for example, according to the principle shown in Fig. 7. The upper left and upper right parts of Fig. 7 are diagrams for explaining the principle of occurrence of the incident angle directivity in the image sensor 121 of Fig. 5, and the lower left and lower right parts of Fig. 7 are diagrams for explaining the principle of occurrence of the incident angle directivity in the image sensor 121 of Fig. 6.

[0123] Moreover, each pixel in the upper left and upper right parts of Fig. 7 is composed of one photodiode 121e. In contrast, each pixel in the lower left and lower right parts of Fig. 7 is composed of two photodiodes 121f. Note that, although an example in which one pixel is composed of two photodiodes 121f is described here, this is for convenience of explanation, and the number of photodiodes 121f constituting one pixel may be other numbers.

[0124] In the upper left part of Fig. 7, a light-shielding film 121b-11 is formed so as to shield the right half of the light-receiving surface of the photodiode 121e-11 when incident light is incident from the top to the bottom in the figure. Also, in the upper right part of Fig. 7, a light-shielding film 121b-12 is formed so as to shield the left half of the light-receiving surface of the photodiode 121e-12. The dashed dotted line in the figure indicates the horizontal center position of the light-receiving surface of the photodiode 121e in the figure, and indicates the vertical direction to the light-receiving surface.

[0125] For example, in the case of a configuration as shown in the upper left of Fig. 7, incident light from the upper right direction in the figure, indicated by an arrow that forms an incident angle θ1 with respect to the dashed line in the figure, is easily received in the left half of the range not shielded by the light-shielding film 121b-11 of the photodiode 121e-11, but incident light from the upper left direction in the figure, indicated by an arrow that forms an incident angle θ2 with respect to the dashed line in the figure, is difficult to receive in the left half of the range not shielded by the light-shielding film 121b-11 of the photodiode 121e-11. Therefore, in the case of a configuration as shown in the upper left of Fig. 7, the incident angle directivity is provided such that the light-receiving sensitivity characteristic is high for incident light from the upper right in the figure and low for incident light from the upper left.

[0126] In contrast, for example, in the case of a configuration such as that shown in the upper right part of Fig. 7, incident light from the upper right direction in the figure, indicated by an arrow that forms an incident angle θ11 with respect to the dashed line in the figure, is difficult to receive in the left half range that is shielded by the light-shielding film 121b-12 of the photodiode 121e-12, but incident light from the upper left direction in the figure, indicated by an arrow that forms an incident angle θ12 with respect to the dashed line in the figure, is easy to receive in the right half range that is not shielded by the light-shielding film 121b-12 of the photodiode 121e-12. Therefore, in the case of a configuration such as that shown in the upper right part of Fig. 7, the incident angle directivity is provided such that the light-receiving sensitivity characteristic is low for incident light from the upper right in the figure and the light-receiving sensitivity characteristic is high for incident light from the upper left.

[0127] In addition, in the case of the lower left part of Figure 7, photodiodes 121f-1 and 121f-2 are provided on the left and right sides of the figure, and the detection signal of either one is read out, thereby providing a configuration with incident angle directivity without providing a light-shielding film 121b.

[0128] That is, as shown in the lower left of Fig. 7, when two photodiodes 121f-1 and 121f-2 are formed in the pixel 121a, the detection signal of the photodiode 121f-1 provided on the left side of the figure contributes to the detection signal level of this pixel 121a, so that the pixel 121a can have the same incident angle directivity as the configuration in the upper left of Fig. 7. That is, the incident light from the upper right direction in the figure, indicated by the arrow that forms an incident angle θ21 with respect to the dashed line in the figure, is incident on the photodiode 121f-1 and received, and the detection signal is read out and contributes to the detection signal level of this pixel 121a. On the other hand, the incident light from the upper left direction in the figure, indicated by the arrow that forms an incident angle θ22 with respect to the dashed line in the figure, is incident on the photodiode 121f-2, but the detection signal is not read out and does not contribute to the detection signal level of this pixel 121a.

[0129] Similarly, as shown in the lower right of FIG. 7, when two photodiodes 121f-11 and 121f-12 are formed in the pixel 121a, the detection signal of the photodiode 121f-12 provided on the left side of the figure contributes to the detection signal level of the pixel 121a, so that the pixel 121a can have the same incident angle directivity as the configuration in the upper right of FIG. 7. That is, the incident light from the upper right direction in the figure, indicated by the arrow that forms an incident angle θ31 with respect to the dashed line in the figure, is incident on the photodiode 121f-11, but the detection signal is not read out and does not contribute to the detection signal level of the pixel 121a. On the other hand, the incident light from the upper left direction in the figure, indicated by the arrow that forms an incident angle θ32 with respect to the dashed line in the figure, is incident on and received by the photodiode 121f-12, and the detection signal is read out and contributes to the detection signal level of the pixel 121a.

[0130] 7, the vertical dashed line is the horizontal center of the light receiving surface of the photodiode 121e, but may be at another position for convenience of explanation. By changing the horizontal position of the light shielding film 121b indicated by the vertical dashed line, different incident angle directivities can be generated.

[0131] <Incident angle directivity in a configuration including an on-chip lens> The principles of how incident angle directivity is generated have been described above, but here, incident angle directivity in a configuration including the on-chip lens 121c will be described.

[0132] That is, the incident angle directivity of each pixel in the imaging element 121 is set, for example, as shown in Fig. 8 by using the on-chip lens 121c in addition to that by the light-shielding film 121b described above. That is, in the middle left part of Fig. 8, an on-chip lens 121c-11 that collects incident light from the incident direction from the upper part of the figure, a color filter 121d-11 that transmits light of a predetermined wavelength, and a photodiode 121e-11 that generates a pixel signal by photoelectric conversion are laminated in this order, and in the middle right part of Fig. 8, an on-chip lens 121c-12, a color filter 121d-12, and a photodiode 121e-12 are configured in this order from the incident direction from the upper part of the figure.

[0133] When there is no need to distinguish between the on-chip lenses 121c-11 and 121c-12, the color filters 121d-11 and 121d-12, and the photodiodes 121e-11 and 121e-12, they will be simply referred to as the on-chip lens 121c, the color filter 121d, and the photodiode 121e.

[0134] The imaging element 121 is further provided with light shielding films 121b-11 and 121b-12 that shield a part of the region that receives incident light, as shown in the middle left and middle right parts of FIG.

[0135] As shown in the middle left part of Figure 8, when a light-shielding film 121b-11 is provided to block the right half of the photodiode 121e-11 in the figure, the detection signal level of the photodiode 121e-11 changes depending on the incident angle θ of the incident light, as shown by the solid line waveform in the upper part of Figure 8.

[0136] That is, when the incident angle θ, which is the angle of the incident light with respect to the dashed line that is perpendicular to the center positions of the photodiode 121e and the on-chip lens 121c, becomes larger (when the incident angle θ becomes larger in the positive direction (tilts to the right in the figure)), the light is concentrated in an area where the light-shielding film 121b-11 is not provided, and the detection signal level of the photodiode 121e-11 becomes larger. Conversely, when the incident angle θ becomes smaller (when the incident angle θ becomes larger in the negative direction (tilts to the left in the figure)), the light is concentrated in an area where the light-shielding film 121b-11 is provided, and the detection signal level of the photodiode 121e-11 becomes smaller.

[0137] The incident angle θ here is set to 0 degrees when the direction of the incident light coincides with the dashed dotted line, the incident angle θ on the incident angle θ21 side on the left side of the middle row of FIG. 8 where the incident light is incident from the upper right in the figure is set to a positive value, and the incident angle θ on the incident angle θ22 side on the right side of the middle row of FIG. 8 is set to a negative value. Therefore, in FIG. 8, the incident angle of the incident light incident from the upper right to the on-chip lens 121c is larger than that of the incident light incident from the upper left. That is, the incident angle θ becomes larger (increases in the positive direction) as the traveling direction of the incident light inclines to the right in FIG. 8, and becomes smaller (increases in the negative direction) as the traveling direction of the incident light inclines to the left.

[0138] Furthermore, as shown in the middle right part of Figure 8, when a light-shielding film 121b-12 is provided to block the left half of the photodiode 121e-12 in the figure, the detection signal level of the photodiode 121e-12 changes depending on the incident angle θ of the incident light, as shown by the dotted waveform in the upper part of Figure 8.

[0139] 8, the larger the incident angle θ, which is the angle of the incident light with respect to the dashed line perpendicular to the center positions of the photodiode 121e and the on-chip lens 121c (the larger the incident angle θ in the positive direction), the more the light is concentrated in the area where the light-shielding film 121b-12 is provided, thereby decreasing the detection signal level of the photodiode 121e-12. Conversely, the smaller the incident angle θ, which is the larger the incident angle θ in the negative direction), the more the light is incident on the area where the light-shielding film 121b-12 is not provided, thereby increasing the detection signal level of the photodiode 121e-12.

[0140] In the upper part of FIG. 8, the horizontal axis indicates the incident angle θ, and the vertical axis indicates the detection signal level in the photodiode 121e.

[0141] The waveforms shown by the solid and dotted lines in the upper part of Fig. 8, which indicate the detection signal level according to the incident angle θ, can be changed according to the range of the light-shielding film 121b, so that it is possible to give different incident angle directivities to each pixel. The solid waveform in the upper part of Fig. 8 corresponds to the solid arrows showing how the incident light in the middle left part and the lower left part of Fig. 8 is condensed by changing the incident angle θ. The dotted waveform in the upper part of Fig. 8 corresponds to the dotted arrows showing how the incident light in the middle right part and the lower right part of Fig. 8 is condensed by changing the incident angle θ.

[0142] The incident angle directivity here refers to the characteristic (light receiving sensitivity characteristic) of the detection signal level of each pixel according to the incident angle θ, but in the case of the example in the middle part of FIG. 8, it can also be said to be the characteristic of the light blocking value according to the incident angle θ. That is, the light blocking film 121b blocks incident light from a specific direction at a high level, but cannot sufficiently block incident light from directions other than the specific direction. This change in the light blocking level generates different detection signal levels according to the incident angle θ as shown in the upper part of FIG. 8. Therefore, if the direction in which light can be blocked at the highest level in each pixel is defined as the light blocking direction of each pixel, having different incident angle directivities on a pixel-by-pixel basis means, in other words, having different light blocking directions on a pixel-by-pixel basis.

[0143] Furthermore, as shown in the lower left part of FIG. 8, two photodiodes 121f-1 and 121f-2 are provided for one on-chip lens 121c-11 (each pixel output unit is composed of two photodiodes 121f-1 and 121f-2), and by using the detection signal of only the photodiode 121f-1 on the left part of the figure, it is possible to obtain the same detection signal level as that in a state in which the right side of the photodiode 121e-11 in the middle left part of FIG. 8 is shielded from light.

[0144] That is, when the incident angle θ, which is the angle of the incident light with respect to the dashed line perpendicular to the center position of the on-chip lens 121c, becomes larger (when the incident angle θ becomes larger in the positive direction), the light is focused in the range of the photodiode 121f-1 from which the detection signal is read out, and the detection signal level becomes larger. Conversely, when the incident angle θ becomes smaller (when the incident angle θ becomes larger in the negative direction), the light is focused in the range of the photodiode 121f-2 from which the detection value is not read out, and the detection signal level becomes smaller.

[0145] Similarly, as shown in the lower right part of FIG. 8, by providing two photodiodes 121f-11 and 121f-12 for one on-chip lens 121c-12 and using the detection signal of only the photodiode 121f-12 on the right side of the figure, it is possible to obtain a detection signal on an output pixel basis having the same detection signal level as that in a state in which the left side of the photodiode 121e-12 in the middle right part of FIG. 8 is shielded from light.

[0146] That is, when the incident angle θ, which is the angle of the incident light with respect to the dashed line perpendicular to the center position of the on-chip lens 121c, becomes larger (when the incident angle θ becomes larger in the positive direction), the light is collected in the range of the photodiode 121f-11 whose detection signal does not contribute to the detection signal of the output pixel unit, so that the detection signal level of the detection signal of the output pixel unit becomes smaller. Conversely, when the incident angle θ becomes smaller (when the incident angle θ becomes larger in the negative direction), the light is collected in the range of the photodiode 121f-12 whose detection signal contributes to the detection signal of the output pixel unit, so that the detection signal level of the detection signal of the output pixel unit becomes higher.

[0147] Incidentally, it is desirable that the incident angle directivity is highly random. For example, if adjacent pixels have the same incident angle directivity, the above-mentioned formulas (1) to (3) or the formulas (4) to (6) described later may be the same formulas, and the relationship between the unknowns and the number of formulas that are the solutions of the simultaneous equations may not be satisfied, and the pixel values ​​that constitute the restored image may not be obtained. In addition, in the configuration shown in the middle part of FIG. 8, one photodiode 121e-11 and one photodiode 121e-12 are formed in the pixel 121a. In contrast, in the configuration shown in the lower part of FIG. 8, two photodiodes 121f-1 and 121f-2, and two photodiodes 121f-11 and 121f-12 are formed in the pixel 121a. Therefore, for example, in the lower part of FIG. 8, one pixel is not constituted by a single photodiode 121f.

[0148] 8, when one pixel output unit is composed of a plurality of photodiodes 121f, it can be considered that the output pixel value of the pixel output unit is modulated according to the incident angle. Therefore, it is possible to make the characteristics of the output pixel value (incident angle directivity) different for each pixel output unit, and the incident angle directivity for each pixel output unit is set. Furthermore, when one pixel output unit is composed of a plurality of photodiodes 121f, one on-chip lens 121c is essential for one pixel output unit to generate the incident angle directivity for each pixel output unit.

[0149] 8, when each of the photodiodes 121e-11 or 121e-12 constitutes one pixel output unit, the incident light to each of the photodiodes 121e-11 or 121e-12 constituting one pixel output unit is modulated according to the incident angle, resulting in modulation of the output pixel value. Therefore, it is possible to make the characteristics (incident angle directivity) of the output pixel value different, and the incident angle directivity is set for each pixel output unit. Furthermore, when each of the photodiodes 121e-11 or 121e-12 constitutes one pixel output unit, the incident angle directivity is set independently during manufacturing by the light shielding film 121b provided for each pixel output unit.

[0150] Furthermore, as shown in the lower part of Figure 8, when one pixel output unit is composed of multiple photodiodes 121f, the number of multiple photodiodes 121f (the number of divisions of the photodiodes 121f constituting one pixel output unit) and positions for setting the incident angle directivity for each pixel output unit are set independently for each pixel output unit at the time of manufacture, and further, which of these photodiodes 121f is used to set the incident angle directivity can be switched at the time of imaging.

[0151] <Incident angle directivity setting> For example, as shown in the upper part of FIG. 9, the set range of the light blocking film 121b is from the left end to position A in the horizontal direction of the pixel 121a, and from the top end to position B in the vertical direction.

[0152] In this case, a horizontal weight Wx of 0 to 1 is set as an index of incident angle directivity according to the incident angle θx (deg) from the horizontal center position of each pixel. More specifically, assuming that the weight Wx is 0.5 at the incident angle θx=θa corresponding to position A, the weight Wh is set so that the weight Wx is 1 at the incident angle θx<θa-α, the weight Wx is (-(θx-θa) / 2α+1 / 2) at θa-α≦incident angle θx≦θa+α, and the weight Wx is 0 at the incident angle θx>θa+α. Here, an example in which the weight Wh is 0, 0.5, or 1 will be described, but the weight Wh is 0, 0.5, or 1 when ideal conditions are satisfied.

[0153] Similarly, a weight Wy of 0 to 1 in the vertical direction, which is an index of incident angle directivity, is set according to the incident angle θy (deg) from the vertical center position of each pixel. More specifically, assuming that the weight Wv is 0.5 at the incident angle θy=θb corresponding to position B, the weight Wy is set so that the weight Wy is 0 at the incident angle θy<θb-α, the weight Wy is ((θy-θb) / 2α+1 / 2) at θb-α≦incident angle θy≦θb+α, and the weight Wy is 1 at the incident angle θy>θb+α.

[0154] Then, by using the weights Wx and Wy thus obtained, it is possible to obtain a coefficient (coefficient set) corresponding to the incident angle directivity, that is, the light receiving sensitivity characteristic, of each pixel 121a.

[0155] In addition, at this time, the gradient (1 / 2α) indicating the change in weight in the range where the horizontal weight Wx and the vertical weight Wy are around 0.5 can be set by using the on-chip lens 121c with a different focal length.

[0156] That is, by using the on-chip lenses 121c with different curvatures, different focal lengths can be achieved.

[0157] For example, by using the on-chip lens 121c with a different curvature, when light is condensed so that the focal length is on the light-shielding film 121b, the slope (1 / 2α) becomes steep, as shown by the solid line in the lower part of Fig. 9. That is, the horizontal weight Wx and the vertical weight Wy in the upper part of Fig. 9 suddenly change to 0 or 1 near the boundary between the horizontal incidence angle θx=θa, which is near 0.5, and the vertical incidence angle θy=θb.

[0158] Also, for example, by using the on-chip lens 121c with a different curvature, when the focal length is focused on the photodiode 121e, the slope (1 / 2α) becomes gentler, as shown by the dotted line in the lower part of Fig. 9. That is, in the upper part of Fig. 9, the horizontal weight Wx and the vertical weight Wy change gently to 0 or 1 near the boundary between the horizontal incidence angle θx=θa and the vertical incidence angle θy=θb, where the horizontal weight Wx and the vertical weight Wy are near 0.5.

[0159] As described above, by using the on-chip lenses 121c with different curvatures and different focal lengths, different incident angle directivities, that is, different light receiving sensitivity characteristics, can be obtained.

[0160] Therefore, the incident angle directivity of the pixel 121a can be set to a different value by making the range in which the photodiode 121e is shielded by the light shielding film 121b different from the curvature of the on-chip lens 121c. The curvature of the on-chip lens may be the same for all pixels in the image sensor 121, or may be different for some pixels.

[0161] <Difference between on-chip lenses and imaging lenses> As described above, the imaging element 121 does not require an imaging lens. However, the on-chip lens 121c is necessary at least when realizing incident angle directivity using multiple photodiodes in a pixel as described with reference to Fig. 6. The on-chip lens 121c and the imaging lens have different physical functions.

[0162] The imaging lens has a light collecting function for making incident light coming from the same direction incident on multiple adjacent pixels. In contrast, light passing through the on-chip lens 121c is incident only on the light receiving surface of the photodiode 121e or 121f constituting the corresponding pixel. In other words, the on-chip lens 121c is provided for each pixel output unit, and collects the subject light incident on itself only on the corresponding pixel output unit. In other words, the on-chip lens 121c does not have a light collecting function for making the diffused light emitted from the virtual point light source incident on multiple adjacent pixels.

[0163] <Relationship between the distance of the subject and the image sensor> Next, the relationship of the distance between the object plane and the imaging element 121 will be described with reference to FIG.

[0164] As shown in the upper left part of Figure 10, when the subject distance from the image sensor 121 to the subject plane 131 is distance d1, for example, when point light sources PA, PB, and PC are set on the subject plane 131, the detection signal levels DA, DB, and DC at corresponding positions Pa, Pb, and Pc on the image sensor 121 can be expressed by the same equations as the above-mentioned equations (1) to (3).

[0165] DA = α1×a + β1×b + γ1×c (1) DB = α2 × a + β2 × b + γ2 × c (2) DC = α3×a + β3×b + γ3×c (3)

[0166] In contrast, as shown in the lower left part of Figure 10, when the subject distance from image sensor 121 is subject plane 131' which is distance d2 which is greater than distance d1 by d, that is, when subject plane 131' is further back than subject plane 131 as viewed from image sensor 121, the detection signal levels are the same for all of detection signal levels DA, DB, and DC, as shown in the upper center and lower center parts of Figure 10.

[0167] However, in this case, light rays with light intensities a', b', c' from point light sources PA', PB', PC' on the object plane 131' are received by each pixel of the image sensor 121. At this time, the angles of incidence of the light rays with light intensities a', b', c' received on the image sensor 121 are different (change), so different coefficient sets are required, and the detection signal levels DA, DB, DC at each position Pa, Pb, Pc are expressed, for example, as shown in the following equations (4) to (6).

[0168] DA = α11×a' + β11×b' + γ11×c' (4) DB = α12×a' + β12×b' + γ12×c' (5) DC = α13×a' + β13×b' + γ13×c' (6)

[0169] Here, the group of coefficient sets consisting of coefficient sets α11, β11, γ11, coefficient sets α12, β12, γ12, and coefficient sets α13, β13, γ13 are the group of coefficient sets of object plane 131' corresponding to coefficient sets α1, β1, γ1, coefficient sets α2, β2, γ2, and coefficient sets α3, β3, γ3 on object plane 131, respectively.

[0170] Therefore, by solving equations (4) to (6) using the preset coefficient set group α11, β11, γ11, α12, β12, γ12, α13, β13, γ13, it is possible to obtain the light intensities (a', b', c') of light rays from point light sources PA', PB', PC' as shown in the lower right part of Figure 10 in a similar manner to the method used to obtain the light intensities (a, b, c) of light rays at point light sources PA, PB, PC in the case of object plane 131 shown in the upper right part of Figure 10, and as a result, it is possible to obtain a restored image of the object on object plane 131'.

[0171] In other words, by pre-storing a group of coefficient sets for each distance from the image sensor 121 to the subject plane, configuring simultaneous equations by switching between the groups of coefficient sets, and solving the configured simultaneous equations, it is possible to obtain restored images of the subject plane at various subject distances based on a single detected image.

[0172] In other words, it is possible to generate a restored image for any distance by capturing a detection image just once and then switching the coefficient set group according to the distance to the subject plane in subsequent processing to obtain the restored image.

[0173] Furthermore, when image recognition or characteristics of a visible image or a subject other than a visible image are desired, it is possible to apply machine learning such as deep learning to the detection signal of the imaging element and perform image recognition using the detection signal itself, without first obtaining a restored image and then performing image recognition based on the restored image.

[0174] In addition, in cases where the subject distance and the angle of view can be specified, a restored image may be generated using a detection image made up of detection signals from pixels having an incident angle directivity suitable for imaging the subject plane corresponding to the specified subject distance and angle of view, without using all pixels. In this way, a restored image can be obtained using detection signals from pixels suitable for imaging the subject plane corresponding to the specified subject distance and angle of view, making it possible to obtain a restored image for the specified subject distance and angle of view with high accuracy.

[0175] Here, by obtaining a restored image using the detection signals of pixels suitable for imaging the subject plane corresponding to the specified subject distance and angle of view, the reason why the restored image can be obtained with high accuracy will be explained.

[0176] For example, as shown in the upper part of FIG. 11, consider a pixel 121a that is shielded by a light-shielding film 121b by a width d1 from each end of the four sides, and a pixel 121a' that is shielded by a light-shielding film 121b by a width d2 (> d1) from each end of the four sides, as shown in the lower part of FIG. 11.

[0177] The pixel 121a is used, for example, to restore the image I1 in FIG. 11 corresponding to an angle of view SQ1 that includes the entire person H101 serving as the subject, as shown in the upper part of FIG. 12. On the other hand, the pixel 121a' is used, for example, to restore the image I2 in FIG. 11 corresponding to an angle of view SQ2 in which the periphery of the face of the person H101 serving as the subject is zoomed in, as shown in the upper part of FIG. 12.

[0178] This is because, as shown in the left part of FIG. 13, the pixel 121a in FIG. 11 has an incident angle range A of incident light with respect to the imaging element 121, so that on the subject plane 131, incident light corresponding to a subject width W1 in the horizontal direction can be received.

[0179] On the other hand, since the pixel 121a' in FIG. 11 has a wider shielded range than the pixel 121a in FIG. 11, as shown in the left part of FIG. 13, it has an incident angle range B (< A) of incident light with respect to the imaging element 121, so that on the subject plane 131, incident light corresponding to a subject width W2 (< W1) in the horizontal direction is received.

[0180] 11, which has a narrow light-blocking range, is a wide-angle pixel suitable for capturing an image of a wide range on the object plane 131, whereas pixel 121a', which has a wide light-blocking range, is a narrow-angle pixel suitable for capturing an image of a narrow range on the object plane 131. Note that the terms wide-angle pixel and narrow-angle pixel mentioned here are expressions for comparing both pixels 121a and 121a' in FIG. 11, and are not limited to comparisons of pixels with other angles of view.

[0181] 13 shows the relationship between positions on subject plane 131 and the angles of incidence of incident light from each position relative to center position C1 of image sensor 121. Also, while Fig. 13 shows the horizontal relationship between positions on subject plane 131 and the angles of incidence of incident light from each position on subject plane 131, the same relationship applies in the vertical direction. Furthermore, pixels 121a and 121a' in Fig. 11 are shown on the right side of Fig. 13.

[0182] With this configuration, as shown in the lower part of Figure 12, when a predetermined number of pixels 121a of Figure 11 are collected in the range ZA surrounded by a dotted line in the image sensor 121, and a predetermined number of pixels 121a' of Figure 11 are collected in the range ZB surrounded by a dotted line in the image sensor 121, when attempting to restore an image of field angle SQ1 corresponding to subject width W1, the image of subject width W1 on subject plane 131 can be appropriately restored by using pixels 121a of Figure 11 which captures field angle SQ1.

[0183] Similarly, when attempting to restore an image of a field of view SQ2 corresponding to a subject width W2, the detection signal level of pixel 121a' in Figure 11, which captures the field of view SQ2, can be used to appropriately restore the image of the subject width W2.

[0184] In the lower part of Figure 12, a configuration is shown in which a predetermined number of pixels 121a' are provided on the left side of the figure and a predetermined number of pixels 121a are provided on the right side, but this is shown as an example to simplify the explanation, and it is desirable that the pixels 121a and the pixels 121a' are arranged in a randomly mixed manner.

[0185] As such, since the angle of view SQ2 is narrower than the angle of view SQ1, when images of the angles of view SQ2 and SQ1 are restored with the same specified number of pixels, a restored image of higher image quality can be obtained by restoring the image of the angle of view SQ2, which has a narrower angle of view, than the image of the angle of view SQ1.

[0186] In other words, when considering obtaining a restored image using the same number of pixels, a restored image with higher image quality can be obtained by restoring an image with a narrower angle of view.

[0187] When an image with a wide angle of view is to be restored, all or some of the wide-angle pixels may be used, and when an image with a narrow angle of view is to be restored, all or some of the narrow-angle pixels may be used.

[0188] By using the imaging element 121 as described above, optical elements such as imaging lenses and diffraction gratings, pinholes, etc. are not required, which increases the degree of freedom in designing the device and also makes it possible to reduce the size of the device relative to the direction of incidence of the incident light, thereby reducing the manufacturing cost. In addition, lenses such as a focus lens that are equivalent to an imaging lens for forming an optical image are not required.

[0189] Furthermore, by using the image sensor 121, it is possible to generate restored images for various subject distances and angles of view by simply acquiring a detected image and then solving a simultaneous equation constructed by selectively using a group of coefficient sets according to the subject distance and angle of view to obtain a restored image.

[0190] Furthermore, since the image sensor 121 can have incident angle directivity on a pixel-by-pixel basis, it can realize a larger number of pixels than optical filters made of diffraction gratings and conventional image sensors, and can obtain images with high resolution and high angular resolution.

[0191] In addition, since the imaging element 121 does not require an optical filter made of a diffraction grating, the optical filter will not be distorted by heat when the environment in which it is used becomes hot. Therefore, by using such an imaging element 121, it is possible to realize a device with high environmental resistance.

[0192] <First Modification> 4, an example is shown in which the light-shielding film 121b in each pixel 121a of the image sensor 121 is configured to provide a difference in the horizontal incidence angle directivity by providing a vertical light-shielding overall configuration and changing the light-shielding width and position in the horizontal direction, but the configuration of the light-shielding film 121b is not limited to this example. For example, the light-shielding film 121b may be configured to provide a horizontal light-shielding overall configuration and change the vertical width (height) and position to provide a difference in the vertical incidence angle directivity.

[0193] 4, the light shielding film 121b that shields the entire pixel 121a from light in the vertical direction and shields the pixel 121a from light by a predetermined width in the horizontal direction is called a horizontal stripe type light shielding film 121b. In contrast, the light shielding film 121b that shields the entire pixel 121a from light in the horizontal direction and shields the pixel 121a from light by a predetermined height in the vertical direction is called a vertical stripe type light shielding film 121b.

[0194] Also, as shown in the example on the left side of Fig. 14, vertical and horizontal stripe type light shielding films 121b may be combined to provide an L-shaped light shielding film 121b for pixel 121a. In the left side of Fig. 14, the black portion is the light shielding film 121b. That is, light shielding films 121b-21 to 121b-24 are light shielding films for pixels 121a-21 to 121a-24, respectively.

[0195] Each of these pixels (pixels 121a-21 to 121a-24) has an incident angle directivity as shown in the right part of Fig. 14. The graph shown in the right part of Fig. 14 shows the light sensitivity of each pixel. The horizontal axis represents the incident angle θx of the incident light in the horizontal direction (x direction), and the vertical axis represents the incident angle θy of the incident light in the vertical direction (y direction). The light sensitivity within range C4 is higher than that outside range C4, the light sensitivity within range C3 is higher than that outside range C3, the light sensitivity within range C2 is higher than that outside range C2, and the light sensitivity within range C1 is higher than that outside range C1.

[0196] Therefore, for each pixel, the detection signal level of the incident light that satisfies the condition of the horizontal (x-direction) incident angle θx and the vertical (y-direction) incident angle θy within range C1 is the highest, and the detection signal level decreases in the order of range C2, range C3, range C4, and ranges other than range C4. The intensity of such light receiving sensitivity is determined by the range blocked by the light blocking film 121b.

[0197] Also, in the left part of FIG. 14, the alphabets in each pixel 121a indicate the color of the color filter (written for convenience of explanation, not actually written). The pixel 121a-21 is a G pixel in which a green color filter is arranged, the pixel 121a-22 is an R pixel in which a red color filter is arranged, the pixel 121a-23 is a B pixel in which a blue color filter is arranged, and the pixel 121a-24 is a G pixel in which a green color filter is arranged. That is, these pixels form a Bayer array. Of course, this is only an example, and the array pattern of the color filters is arbitrary. The arrangement of the light-shielding film 121b is unrelated to the color filters. For example, a filter other than a color filter may be provided in some or all of the pixels, or no filter may be provided.

[0198] In the left part of FIG. 14, an example is shown in which the "L-shaped" light-shielding film 121b shields the left side and the bottom side of the pixel 121a in the figure, but the orientation of this "L-shaped" light-shielding film 121b is arbitrary and is not limited to the example of FIG. 14. For example, the "L-shaped" light-shielding film 121b may shield the bottom side and the right side of the pixel 121a in the figure, may shield the right side and the top side of the pixel 121a in the figure, or may shield the top side and the left side of the pixel 121a in the figure. Of course, the orientation of this light-shielding film 121b can be set independently for each pixel. This "L-shaped" light-shielding film 121b is also collectively referred to as "L-type light-shielding film 121b".

[0199] Although the light-shielding film has been described above, the description of this example can also be applied to the case where incident angle directivity is provided by selectively using multiple photodiodes arranged in a pixel. That is, for example, by appropriately setting the division position (size and shape of each partial region), the position, size, shape, etc. of each photodiode, or by appropriately selecting the photodiode, it is possible to realize incident light directivity equivalent to that of the above-mentioned L-shaped light-shielding film 121b.

[0200] <Second Modification> In the above, we have described examples in which horizontal band type, vertical band type, and L-shaped type light-shielding films are arranged at each pixel so that the light-shielding range changes randomly. However, for example, when a rectangular opening is provided as shown in image sensor 121' in Figure 15, a light-shielding film 121b (the range shown in black in the figure) may be configured to block light except for the range near the position where the light is received at each pixel.

[0201] In other words, when a rectangular opening is provided for each pixel, the light-shielding film 121b may be provided so as to have an incident angle directivity such that only light rays that pass through the rectangular opening and are received are received among the light rays emitted from a point light source that constitutes an object plane at a specified object distance.

[0202] In FIG. 15, for example, with respect to the pixel array in the horizontal direction, the horizontal width of the light-shielding film 121b changes to widths dx1, dx2, ··· dxn, and this has the relationship of dx1 < dx2 < ··· < dxn. Similarly, with respect to the pixel array in the vertical direction, the vertical height of the light-shielding film 121b changes to heights dy1, dy2, ··· dym, and this has the relationship of dy1 < dy2 < ··· < dym. Also, the intervals of the respective changes in the horizontal width and the vertical width of the light-shielding film 121b depend on the subject resolution (angular resolution) to be restored.

[0203] In other words, it can be said that the configuration of each pixel 121a in the image sensor 121' in FIG. 15 has an incident angle directivity that changes the light-shielding range so as to correspond to the pixel arrangement in the image sensor 121' in the horizontal and vertical directions.

[0204] More specifically, the light-shielding range of each pixel 121a in FIG. 15 is determined according to the rule explained using the pixel 121a shown in the left part of FIG. 16, for example.

[0205] Incidentally, the right part of FIG. 16 shows the configuration of the same image sensor 121' as in FIG. 15. Also, the left part of FIG. 16 shows the configuration of the pixel 121a of the image sensor 121' in the right part of FIG. 16 (the same as FIG. 15).

[0206] As shown in the left part of FIG. 16, from the ends of the upper side and the lower side of the pixel 121a toward the inside of the pixel 121a, the light-shielding film 121b shields by widths dx1 respectively, and from the ends of the left side and the right side toward the inside of the pixel 121a, the light-shielding film 121b shields by heights dy1 respectively. Incidentally, in FIGS. 15 to 17, the light-shielding film 121b is the range shown in black.

[0207] In the left part of FIG. 16, the range shielded by the light-shielding film 121b formed in this way is hereinafter referred to as the main light-shielding part Z101 of the pixel 121a (the black part in the left part of FIG. 16), and the other rectangular range is referred to as the range Z102.

[0208] It is assumed that a rectangular opening Z111 that is not shielded by the light shielding film 121b is provided within the area Z102 in the pixel 121a. Therefore, in the area Z102, the area other than the rectangular opening Z111 is shielded from light by the light shielding film 121b.

[0209] The pixel arrangement in the image sensor 121' in Figure 15 is configured as shown in the right part of Figure 16 (same as Figure 15), with the pixel 121a-1 at the left end and upper end being configured so that the rectangular opening Z111 has its left side spaced a width dx1 from the left side of pixel 121a and its upper side spaced a distance dy1 from the upper side of pixel 121a.

[0210] Similarly, pixel 121a-2, which is adjacent to pixel 121a-1 to the right, has a rectangular opening Z111 whose left side is a distance of width dx2 from the left side of pixel 121a and whose top side is a distance of height dy1 from the top side of pixel 121a, and is configured so that the area other than the rectangular opening Z111 is shielded from light by a light-shielding film 121b.

[0211] Similarly, the right side of the rectangular opening Z111 moves from the right side of the pixel 121a by widths dx1, dx2, dxn as the arrangement of the horizontally adjacent pixels 121a progresses to the right side in the figure. Note that the dotted square portion in the upper right part of the range Z102 in FIG. 16 shows the state when the rectangular opening Z111 is arranged such that its left side is a width dxn from the left side of the pixel 121a and its top side is a height dy1 from the top side of the pixel 121a. Also, the respective intervals of the widths dx1, dx2, dxn are calculated by dividing the width obtained by subtracting the width of the rectangular opening Z111 from the horizontal width of the range Z102 by the number of pixels n in the horizontal direction. In other words, the interval of the change in the horizontal direction is determined by dividing by the number of pixels n in the horizontal direction.

[0212] Furthermore, the horizontal positions of the rectangular openings Z111 in the pixels 121a in the imaging element 121' are the same in the pixels 121a (pixels 121a in the same column) that are located at the same horizontal position in the imaging element 121'.

[0213] Furthermore, pixel 121a-3, which is adjacent and directly below pixel 121a-1, has a rectangular opening Z111 whose left side is a distance of width dx1 from the left side of pixel 121a and whose top side is a distance of height dy2 from the top side of pixel 121a, and is configured so that the area other than the rectangular opening Z111 is shielded from light by light-shielding film 121b.

[0214] Similarly, as the pixel 121a adjacent in the vertical direction moves downward in the figure, the upper side of the rectangular opening Z111 moves by heights dy1, dy2, dyn from the upper side of the pixel 121a. The dotted square portion at the lower left of the range Z102 in FIG. 16 shows the state when the rectangular opening Z111 is arranged with its left side at a width dx1 from the left side of the pixel 121a and its upper side at a height dym from the upper side of the pixel 121a. The respective intervals of the heights dy1, dy2, dym are calculated by dividing the height of the range Z102 in the vertical direction by the height of the rectangular opening Z111, by the number of pixels m in the vertical direction. That is, the interval of the change in the vertical direction is determined by dividing by the number of pixels m in the vertical direction.

[0215] Furthermore, the vertical position of the rectangular opening Z111 in the pixels 121a in the imaging element 121' is the same in the pixels 121a (pixels 121a in the same row) that have the same vertical position in the imaging element 121'.

[0216] Furthermore, by changing the main light shielding portion Z101 and the rectangular opening portion Z111 of each pixel 121a constituting the image sensor 121' shown in FIG. 16 (FIG. 15), the angle of view can be changed.

[0217] The right part of Fig. 17 shows the configuration of an image sensor 121' when the angle of view is wider than that of the image sensor 121' in Fig. 16 (Fig. 15). Moreover, the left part of Fig. 17 shows the configuration of pixels 121a of the image sensor 121' in the right part of Fig. 17.

[0218] That is, as shown in the left part of FIG. 17, for example, within pixel 121a, a main light-shielding portion Z151 (the black portion in the left part of FIG. 17) with a light-shielding range narrower than the main light-shielding portion Z101 in FIG. 16 is set, and the other range is set as range Z152. Further, within range Z152, a rectangular opening Z161 with an opening area larger than that of the rectangular opening Z111 is set.

[0219] More specifically, as shown in the left part of FIG. 17, from the end portions of the upper side and the lower side of pixel 121a towards the inside of pixel 121a, the light-shielding film 121b shields light by widths dx1' (<dx1) respectively, and from the end portions of the left side and the right side of pixel 121a towards the inside of pixel 121a, the light-shielding film 121b shields light by heights dy1' (<dy1) respectively, thereby forming the rectangular opening Z161.

[0220] Here, as shown in the right part of FIG. 17, at the left end and the upper-end pixel 121a-1, the rectangular opening Z161 is arranged such that its left side is at a width dx1' from the left side of pixel 121a and its upper side is at a height dy1' from the upper side of pixel 121a, and the range other than the rectangular opening Z161 is shielded by the light-shielding film 121b.

[0221] Similarly, for the pixel 121a-2 adjacent to the right of pixel 121a-1, the rectangular opening Z161 is arranged such that its left side is at a width dx2' from the left side of pixel 121a and its upper side is at a height dy1' from the upper side of pixel 121a, and the range other than the rectangular opening Z161 is shielded by the light-shielding film 121b.

[0222] Similarly, as the arrangement of horizontally adjacent pixels 121a progresses to the right in the figure, the right side of the rectangular opening Z161 moves from the right side of pixel 121a by widths dx1', dx2', dxn'. Here, the interval between each of the widths dx1', dx2', dxn' is the value obtained by subtracting the horizontal width of the rectangular opening Z161 from the horizontal width of the range Z152, and dividing this by the number of pixels n in the horizontal direction. In other words, the interval between vertical changes is determined by dividing by the number of pixels n in the horizontal direction. Therefore, the interval between changes in widths dx1', dx2', dxn' is greater than the interval between changes in widths dx1, dx2, dxn.

[0223] Furthermore, the horizontal positions of the rectangular openings Z161 in the pixels 121a in the image sensor 121' in FIG. 17 are the same in the pixels 121a (pixels 121a in the same column) that are located at the same horizontal position in the image sensor 121'.

[0224] Furthermore, pixel 121a-3, which is adjacent and directly below pixel 121a-1, has a rectangular opening Z161 whose left side is at a width dx1' from the left side of pixel 121a and whose top side is at a height dy2' from the top side of pixel 121a, and is configured so that the area other than the rectangular opening Z161 is shielded from light by light-shielding film 121b.

[0225] Similarly, for pixels 121a adjacent in the vertical direction, the top side of rectangular opening Z161 changes in height dy1', dy2'...dym' from the top side of pixel 121a as the arrangement proceeds downward in the figure. Here, the interval between changes in height dy1', dy2'...dym' is the value obtained by subtracting the height of rectangular opening Z161 from the vertical height of range Z152, divided by the number of pixels m in the vertical direction. In other words, the interval between changes in the vertical direction is determined by dividing by the number of pixels m in the vertical direction. Therefore, the interval between changes in height dy1', dy2'...dym' is greater than the interval between changes in width height dy1, dy2...dym.

[0226] Furthermore, the vertical position of the rectangular opening Z161 in the pixels 121a in the imaging element 121' in FIG. 17 is the same in the pixels 121a (pixels 121a in the same row) that are in the same vertical position in the imaging element 121'.

[0227] In this way, by changing the combination of the light shielding range of the main light shielding portion and the opening range of the opening, it is possible to realize an image sensor 121' consisting of pixels 121a with various angles of view (various incident angle directivities).

[0228] Furthermore, the imaging element 121 may be realized by combining not only the pixels 121a having the same angle of view, but also the pixels 121a having various angles of view.

[0229] For example, as shown in FIG. 18, four pixels consisting of 2 pixels x 2 pixels indicated by dotted lines are defined as one unit U, and each unit U is composed of four pixels: wide-angle pixel 121a-W, medium-angle pixel 121a-M, narrow-angle pixel 121a-N, and extremely narrow-angle pixel 121a-AN.

[0230] In this case, for example, when the number of all pixels 121a is X, it is possible to restore a restored image using detected images of X / 4 pixels for each of four different angles of view. At this time, four different coefficient sets are used for each angle of view, and restored images for the different angles of view are restored by four different simultaneous equations.

[0231] Therefore, by restoring the restored image for the angle of view to be restored using a detection image obtained from pixels suitable for capturing the angle of view to be restored, it is possible to restore an appropriate restored image for each of the four angles of view.

[0232] In addition, images with intermediate angles of view between the four angles of view or images with angles of view before and after these may be generated by interpolation from images with the four angles of view, and images with various angles of view may be seamlessly generated to achieve a pseudo optical zoom.

[0233] Although the light-shielding film has been described above, the description of this example can also be applied to the case where incident angle directivity is provided by selectively using multiple photodiodes arranged in a pixel. That is, for example, by appropriately setting the division position (size and shape of each partial region), the position, size, shape, etc. of each photodiode, or by appropriately selecting the photodiode, it is possible to realize incident light directivity equivalent to the incident light directivity by the light-shielding film 121b having the rectangular opening described above. Of course, in this case, the image sensor 121 can also be realized by combining pixels 121a with various angles of view. In addition, images with intermediate angles of view or angles of view before and after the intermediate angles of view may be generated by interpolation from images with multiple types of angles of view, and images with various angles of view may be seamlessly generated to realize a pseudo optical zoom.

[0234] <Third Modification> Incidentally, when the light-shielding range of the light-shielding film 121b of the pixel 121a in the image sensor 121 is made random, the greater the randomness of the differences in the light-shielding range of the light-shielding film 121b, the greater the processing load on the restoration unit 321, etc. Therefore, the processing load may be reduced by making some of the changes in the light-shielding range of the light-shielding film 121b of the pixel 121a regular and reducing this randomness.

[0235] That is, for example, an L-shaped light-shielding film 121b may be formed by combining a vertical band type and a horizontal band type, and horizontal band type light-shielding films 121b of the same width may be combined for a predetermined column direction, and vertical band type light-shielding films 121b of the same height may be combined for a predetermined row direction, thereby providing regularity in the column and row directions while causing random changes on a pixel-by-pixel basis, thereby reducing the randomness in the incident angle directivity of each pixel and reducing the processing load outside the image sensor 121, such as the restoration unit 321.

[0236] That is, for example, as shown in the image sensor 121'' in FIG. 19, for pixels in the same column indicated by range Z130, horizontal stripe-type light-shielding films 121b of the same width X0 are used, for pixels in the same row indicated by range Z150, vertical stripe-type light-shielding films 121b of the same height Y0 are used, and for pixels 121a specified by each matrix, an L-shaped light-shielding film 121b that combines these is set.

[0237] Similarly, for pixels in the same column indicated by range Z131 adjacent to range Z130, horizontal stripe-type light-shielding films 121b of the same width X1 are used, for pixels in the same row indicated by range Z151 adjacent to range Z150, vertical stripe-type light-shielding films 121b of the same height Y1 are used, and for pixels 121a specified by each matrix, an L-shaped light-shielding film 121b that combines these is set.

[0238] Furthermore, for pixels in the same column indicated by range Z132 adjacent to range Z131, a horizontal stripe type light-shielding film of the same width X2 is used, for pixels in the same row indicated by range Z152 adjacent to range Z151, a vertical stripe type light-shielding film of the same height Y2 is used, and for pixels 121a specified by each matrix, an L-shaped light-shielding film 121b that combines these is set.

[0239] In this way, the range of the light-shielding film can be changed on a pixel-by-pixel basis while providing regularity to the horizontal width and position, and the vertical height and position, of the light-shielding film 121b, thereby suppressing the randomness of the incident angle directivity. As a result, it is possible to reduce the pattern of the coefficient set, and thus to reduce the processing load of the calculation processing in the subsequent stage (e.g., the restoration unit 321, etc.).

[0240] More specifically, as shown in the upper right part of FIG. 20, when a restored image of N×N pixels is obtained from a detected image Pic of N pixels×N pixels, a relationship as shown in the left part of FIG. 20 is established by vector X whose elements are the pixel values ​​of each pixel of the restored image of N×N rows and 1 column, vector Y whose elements are the pixel values ​​of each pixel of the detected image of N×N rows and 1 column, and matrix A of N×N rows and N×N columns consisting of a set of coefficients.

[0241] In other words, FIG. 20 shows that the result of multiplying each element of matrix A of N×N rows and N×N columns consisting of a coefficient set by vector X of N×N rows and 1 column representing a restored image is vector Y of N×N rows and 1 column representing a detected image, and a simultaneous equation can be found from this relationship.

[0242] In addition, in Figure 20, each element in the first column indicated by range Z201 of matrix A corresponds to the elements in the first row of vector X, and each element in the N x N column indicated by range Z202 of matrix A corresponds to the elements in the N x N row of vector X.

[0243] In other words, the restored image is obtained by solving the simultaneous equations based on the determinant shown in Fig. 20 to obtain each element of the vector X. In addition, when a pinhole is used, or when a light-collecting function such as an imaging lens is used to make incident light from the same direction incident on both of the pixel output units adjacent to each other, the relationship between the position of each pixel and the incident angle of light is uniquely determined, so that the matrix A is a square matrix with all diagonal elements ((i,i) elements) being 1 and all elements other than the diagonal elements being 0. Conversely, when neither a pinhole nor an imaging lens is used as in the imaging element 121 of Fig. 3, the relationship between the position of each pixel and the incident angle of light is not uniquely determined, so that the matrix A is not a diagonal matrix.

[0244] Generally, the determinant in Figure 20 has the inverse matrix A of matrix A on both sides. -1 By multiplying it from the left, it is transformed as shown in Figure 21, and the inverse matrix A is applied to the vector Y of the detected image. -1By multiplying from the right, each element of vector X, which is the detected image, is obtained.

[0245] However, in reality, the simultaneous equations cannot be solved for any one or a combination of reasons, such as the fact that matrix A cannot be accurately obtained, cannot be accurately measured, cannot be solved in cases where the basis vectors of matrix A are close to linearly dependent, and / or each element of the detected image contains noise.

[0246] Therefore, a configuration that is robust against various errors is considered, and the following equation (7) is adopted, which uses the concept of the regularized least squares method.

[0247]

number

[0248] Here, the "^" above x in formula (7) represents vector X, A represents matrix A, Y represents vector Y, γ represents a parameter, and ||A|| represents the L2 norm (square root of the sum of squares). Here, the first term is the norm when minimizing the difference between both sides of Fig. 20, and the second term is a regularization term.

[0249] Solving this equation (7) for x gives the following equation (8).

[0250]

number

[0251] However, since the matrix A is of a huge size, it requires a long calculation time and a large amount of memory for the calculation.

[0252] For example, as shown in FIG. 22, matrix A is divided into matrix AL with N rows and N columns and matrix AR with N rows and N columns. TThe matrix X is decomposed into a matrix of N rows and N columns, which represents the restored image, and the result of multiplying the matrix X from the front and back to the rear is the matrix Y of N rows and N columns, which represents the detected image. This allows us to decompose the matrix A, which has the number of elements (N×N)×(N×N), into matrices AL and AR, which have the number of elements (N×N). T Therefore, the number of elements can be reduced to 1 / (N×N). As a result, two matrices AL and AR with the number of elements (N×N) are T Since it is only necessary to use the above, the amount of calculation and the memory capacity can be reduced.

[0253] Here, A T is the transposed matrix of matrix A, γ is a parameter, and I is an identity matrix. The matrix in parentheses in equation (8) is matrix AL, and the inverse matrix of the transposed matrix of matrix A is matrix AR. T By doing so, the determinant shown in FIG. 22 is realized.

[0254] 22, as shown in FIG. 23, the element group Z222 is obtained by multiplying the element Xp of interest in the matrix X by the element group Z221 of the corresponding column of the matrix AL. T By multiplying the two-dimensional response Z224 corresponding to the element of interest Xp by the element of the row corresponding to the element of interest Xp, a two-dimensional response Z224 corresponding to the element of interest Xp is obtained. Then, the two-dimensional responses Z224 corresponding to all elements of the matrix X are integrated to obtain the matrix Y.

[0255] Therefore, the element group Z221 (Figure 23) corresponding to each row of the matrix AL is given a coefficient set corresponding to the incident angle directivity of horizontal band-type pixels 121a that are set to the same width for each column of the image sensor 121 shown in Figure 19.

[0256] Similarly, the matrix AR T Each row of element group Z223 (Figure 23) has a coefficient set corresponding to the incident angle directivity of vertical band-type pixels 121a that are set at the same height for each row of the image sensor 121 shown in Figure 19.

[0257] As a result, it is possible to reduce the size of the matrix used when restoring the restored image based on the detected image, and the amount of calculation is reduced, which makes it possible to improve the processing speed and reduce the power consumption related to the calculation. In addition, since the matrix can be made smaller, it is possible to reduce the memory capacity used for the calculation, and it is possible to reduce the cost of the device.

[0258] In the example of Fig. 19, an example is shown in which the light-shielded range (light-receiving range) is changed on a pixel-by-pixel basis while having a predetermined regularity in the horizontal and vertical directions, but in the technology of the present disclosure, even if the light-shielded range (light-receiving range) is not set completely randomly but is set to a certain degree, it is considered to be set randomly. In other words, in the present disclosure, not only the light-shielded range (light-receiving range) on a pixel-by-pixel basis is set completely randomly, but also the range that is random to a certain degree (for example, the range that includes a regular range for some of the pixels, but the other ranges are random), or the range that does not seem to have a certain degree of regularity (the range that cannot be confirmed to be arranged according to the rules described with reference to Fig. 19) is considered to be random.

[0259] Although the light-shielding film has been described above, the description of this example can also be applied to the case where incident angle directivity is provided by selectively using multiple photodiodes arranged in a pixel. That is, for example, by appropriately setting the division position (size and shape of each partial area) and the position, size, shape, etc. of each photodiode, or by appropriately selecting the photodiode, it is possible to realize incident light directivity equivalent to the incident light directivity when a part of the change in the light-shielding range of the light-shielding film 121b of the pixel 121a described above is made regular. In this way, it is possible to reduce the randomness in the incident angle directivity of each pixel, and reduce the processing load outside the image sensor 121, such as the restoration unit 122.

[0260] <Fourth modified example> The shape of the light-shielding film 121b in pixel units may vary and is not limited to the above-mentioned examples. For example, the light-shielding film 121b may be set to a triangle and its range may be changed to provide different incident angle directivities, or the light-shielding film 121b may be set to a circle and its range may be changed to provide different incident angle directivities. In addition, for example, a linear light-shielding film in an oblique direction may be used.

[0261] Also, the light-shielding film 121b may be set in a variation (pattern) for each of a plurality of pixels constituting a unit consisting of a predetermined number of pixels. This one unit may be composed of any kind of pixels. For example, the imaging element 121 may be provided with a color filter, and may be composed of pixels constituting the unit of the color arrangement of the color filter. Also, a pixel group combining pixels having different exposure times may be used as a unit. Note that it is preferable that the pattern of the range of light shielding film 121b in each pixel constituting the unit has high randomness, that is, that the pixels constituting the unit have different incident angle directivities.

[0262] In addition, the arrangement pattern of the light-shielding film 121b may be set between the units. For example, the width or position of the light-shielding film may be changed for each unit. Furthermore, a pattern of the range of light-shielding film 121b that is shielded from light may be set within a unit made up of a plurality of pixels classified into different categories or between units.

[0263] Although the light-shielding film has been described above, the description of this example can also be applied to the case where incident angle directivity is provided by selectively using multiple photodiodes arranged in a pixel. That is, for example, by appropriately setting the division position (size and shape of each partial area) and the position, size, shape, etc. of each photodiode, or by appropriately selecting the photodiode, it is possible to realize incident light directivity equivalent to the incident light directivity when a part of the change in the light-shielding range of the light-shielding film 121b of the pixel 121a described above is made regular. In this way, it is possible to reduce the randomness in the incident angle directivity of each pixel, and reduce the processing load outside the image sensor 121, such as the restoration unit 122.

[0264] Although the light-shielding film has been described above, the description of this example can also be applied to the case where incident angle directivity is provided by selectively using multiple photodiodes arranged in a pixel. In other words, by appropriately setting the division position (size and shape of each partial region) and the position, size, shape, etc. of each photodiode, or by appropriately selecting the photodiode, it is possible to realize incident light directivity equivalent to that of an incident light shielding film of any shape, such as a triangle, a circle, or a line in an oblique direction.

[0265] Also, for example, the setting of the division position (size and shape of each partial region), the setting of the position, size, shape, etc. of each photodiode, and the selection of the photodiode, etc. may be set for each unit, as in the case of the light-shielding film 121b described above.

[0266] <Photodiode control> When selectively using multiple photodiodes arranged within a pixel as described above with reference to Figure 6, the incident angle directivity of the output pixel value of the pixel output unit may be changed in various ways by switching the presence or absence and the degree of contribution of each of the multiple photodiodes 121f to the output pixel value of the pixel output unit.

[0267] For example, as shown in Fig. 24, nine photodiodes 121f (three vertically × three horizontally) including photodiodes 121f-111 to 121f-119 are arranged in pixel 121a. In this case, this pixel 121a may be used as pixel 121a-b having photodiodes 121f-111 to 121f-119, or may be used as pixel 121a-s having photodiodes 121f-111, 121f-112, 121f-114, and 121f-115.

[0268] For example, when pixel 121a is pixel 121a-b, the incident angle directivity of the output pixel value is controlled by controlling the presence or absence and the degree of contribution of photodiodes 121f-111 to 121f-119 to the output pixel value of pixel 121a. On the other hand, when pixel 121a is pixel 121a-s, the incident angle directivity of the output pixel value is controlled by controlling the presence or absence and the degree of contribution of photodiodes 121f-111, 121f-112, 121f-114, and 121f-115 to the output pixel value of pixel 121a. In this case, the other photodiodes 121f (photodiodes 121f-113, 121f-116, 121f-117 to 121f-119) are controlled so as not to contribute to the output pixel value.

[0269] That is, for example, when the incident angle directivities of the output pixel values ​​are different between the pixels 121a-b, the presence or absence and the degree of contribution of at least one of the photodiodes 121f-111 to 121f-119 to the output pixel value are different. In contrast, when the incident angle directivities of the output pixel values ​​are different between the pixels 121a-s, the presence or absence and the degree of contribution of at least one of the photodiodes 121f-111, 121f-112, 121f-114, and 121f-115 to the output pixel value are different, and the other photodiodes 121f-113, 121f-116, 121f-117 to 121f-119 are common to these pixels and do not contribute to the output pixel value.

[0270] Whether the pixel 121a is to be the pixel 121a-b or the pixel 121a-s can be set for each pixel. This setting may also be made for each unit (multiple pixels).

[0271] As described above, one on-chip lens is formed for each pixel (each pixel output unit) of the imaging element 121. That is, when the pixel 121a has a configuration as shown in the example of Fig. 24, one on-chip lens 121c is provided for the photodiodes 121f-111 to 121f-119 as shown in Fig. 25. Therefore, as described with reference to Fig. 24, whether the pixel 121a is the pixel 121a-b or the pixel 121a-s, one pixel (one pixel output unit) and one on-chip lens 121c correspond one-to-one.

[0272] The above has described the image sensor 121. The imaging unit 101 in Fig. 2 has the image sensor 121 as described above.

[0273] <Identifying the recovery area> Next, the specification of the restored area (or non-restored area) by the area specifying unit 102 will be described. For example, suppose that persons 302-1, 302-2, and 302-3 are reflected in the restored image 301 shown in A of FIG. 26. When it is not necessary to distinguish the persons 302-1 to 302-3 from one another, they are referred to as persons 302. For example, when it is not desired to disclose the face of each person 302, it is sufficient not to restore a predetermined area including the face of each person 302. For example, in the case of the restored image 301 of A of FIG. 26, it is sufficient not to restore the partial area 303-1 including the face of the person 302-1, the partial area 303-2 including the face of the person 302-2, and the partial area 303-3 including the face of the person 302-3, as shown in B of FIG. 26.

[0274] Therefore, for example, as shown in C of Fig. 26, a non-restoration area 305-1 having the same position, shape, and size as the partial area 303-1, a non-restoration area 305-2 having the same position, shape, and size as the partial area 303-2, and a non-restoration area 305-3 having the same position, shape, and size as the partial area 303-3 are set in the detected image 304. When it is not necessary to distinguish the non-restoration areas 305-1 to 305-3 from one another, they are referred to as non-restoration areas 305. Areas other than the non-restoration areas of the detected image are set as restoration areas. In other words, the restoration area 306 indicates areas other than the non-restoration area 305 of the detected image 304.

[0275] By setting such a non-restoration area 305, it is possible to prevent the non-restoration area 305 from being restored. In other words, it is possible to prevent any partial area of ​​the detected image from being restored. In other words, by setting a restoration area 306, it is possible to restore only the restoration area 306. In other words, it is possible to restore only any partial area of ​​the detected image.

[0276] Also, for example, if it is not desired to disclose the face of each person 302, only the partial region 303 may be restored (the area other than the partial region 303 may not be restored). In this case, the non-restoration region 305 described above may be set as the restoration region, and the restoration region 306 described above may be set as the non-restoration region.

[0277] As described above, the restoration area and the non-restoration area can be arbitrarily set in the detected image, but in reality, the detected image is in a state where the image of the subject is not formed and the user cannot visually recognize the subject, such as the detected image 307 shown in D of Fig. 26. Therefore, in this detected image 307, it is basically impossible for the user to visually identify the face part as described above and set the non-restoration area (or restoration area).

[0278] Therefore, AI (Artificial Intelligence) is used to identify the restored area (non-restored area). For example, machine learning (e.g., deep learning, etc.) is performed using a detected image 304 as shown in C of FIG. 26 and a non-restored area 305 (or a restored area 306) corresponding to the detected image as training data. Note that the non-restored area 305 (restored area 306) used as training data is, for example, one identified from a restored image 301 by a user or the like. Such machine learning is performed using a large amount of training data, and features are learned and modeled.

[0279] Such machine learning is performed in advance, and the learning results (learning data) are stored in the learning data storage unit 103. The region specifying unit 102 uses the learning results (learning data) to specify a restoration region (non-restoration region) from the detected image.

[0280] In this way, it is possible to identify subjects that are not visible to humans in the detected image, and to identify partial regions that should be restoration regions (non-restoration regions).

[0281] <Restored area / Non-restored area> The area to be set as the restoration area (non-restoration area) may be any area that can be specified as an area having a predetermined characteristic. In other words, by setting such a restoration area (or non-restoration area), it is possible to restore only an arbitrary partial area having the characteristic, or to restore other areas so as not to restore the partial area. The characteristic is arbitrary. In other words, it is possible to restore only an arbitrary partial area having the characteristic, or to restore other areas so as not to restore the partial area.

[0282] For example, a partial area of ​​the detected image corresponding to an area including a previously registered subject in the restored image or an area other than the registered subject may be set as the restored area (or non-restored area). In this way, it is possible to restore only the area including the previously registered subject, or to restore other areas without restoring the area. For example, an area including a person may be set as the restored area (or non-restored area), or an area other than the area including the person may be set as the restored area (or non-restored area). Of course, instead of a person, an animal such as a dog or a cat may be set as the target subject, or an object such as a vehicle or a tool may be set as the target subject.

[0283] In addition, a partial area of ​​the detected image that corresponds to an area including a preregistered part of the subject in the restored image or an area other than the registered part may be set as the restored area (or non-restored area). By doing so, it is possible to restore only the area including the preregistered part of the preregistered subject, or to restore other areas without restoring the area.

[0284] For example, an area including a person's face may be set as a restoration area (or a non-restoration area), or an area other than the face may be set as a restoration area (or a non-restoration area). In this way, it is possible to restore only the area including the person's face, or to restore other areas without restoring the area including the face. Of course, a part other than the face, such as the hands, feet, or upper body, may be set as the target part. Also, any part of a subject other than a person, such as the license plate of a vehicle, the arm of a machine, or the sign of a road sign, may be set as the target part.

[0285] Also, a partial area of ​​the detected image corresponding to an area including a person registered in advance in the restored image or an area other than the area may be set as the restored area (or non-restoration area). By doing so, it is possible to restore only the area including the person registered in advance, or to restore other areas so that the area is not restored. For example, an area including a specific individual may be set as the restored area (or non-restoration area), or the area other than the area including the individual may be set as the restored area (or non-restoration area). Of course, an area including a part (such as a face) registered in advance of the individual may be set as the restored area (or non-restoration area), or the area other than the part including the registered part may be set as the restored area (or non-restoration area).

[0286] Also, a partial area of ​​the detected image corresponding to an area including a subject that satisfies a pre-registered condition in the restored image or an area other than the area may be set as the restored area (or non-restored area). By doing so, it is possible to restore only the area including the subject that satisfies the pre-registered condition, or to restore other areas without restoring the area. For example, an area including a person performing a pre-registered action may be set as the restored area (or non-restored area), or an area other than the person performing the pre-registered action may be set as the restored area (or non-restored area).

[0287] The motion as the condition is arbitrary. For example, it may be running, walking, jumping, climbing over a wall, sitting, crouching, driving, or other motions. Also, for example, a subject moving from left to right within the angle of view, a subject moving from top to bottom, or a subject moving from back to front may be set as a subject that satisfies the pre-registered condition. Note that the motion (including movement) of the subject may be relative to an absolute coordinate system in a three-dimensional space, or may be relative. For example, it may be a motion that changes the position or posture of the subject relative to its surroundings (background, etc.), or it may be a motion that changes the position or posture of the subject relative to the imaging device 100 (imaging element 121).

[0288] Moreover, the condition is arbitrary and may be other than the action. For example, the number of people may be the condition, or the sex, age, height, weight, etc. of the person to be estimated may be the condition. Of course, the subject may be something other than a person. Moreover, the condition may be set according to the subject, or may be set for each subject.

[0289] Also, a partial area of ​​the detected image with a position and size registered in advance may be set as the restored area (or non-restored area). In other words, any partial area may be set as the restored area (or non-restored area) regardless of the content (subject) of the image. For example, a partial area of ​​any position, size, or shape of the detected image, such as the right half, left half, peripheral part, or center part of the detected image, may be set as the restored area (or non-restored area).

[0290] More specific examples will be described later as usage examples.

[0291] <Restoration matrix corresponding to unrestored region> Next, a description will be given of the non-restored region corresponding restoration matrix generated (set) by the restoration matrix update unit 104. As described with reference to FIG. 20 and FIG. 21, the restored image is generated by the restoration matrix (matrix A (or matrix A -1 )) is used to restore the restored image from the detected image. That is, the restored image of the restoration region is restored by a matrix operation of coefficients of the restoration matrix corresponding to the restoration region and pixel values ​​of the detected image corresponding to the restoration region. That is, by deriving the result of this matrix operation only for the restoration region, it is possible to restore only the restored image of the restoration region.

[0292] In other words, in this matrix calculation, by not deriving a matrix calculation result corresponding to the non-restored region, it is possible to restore the restored image without restoring the non-restored region.

[0293] Therefore, for example, as shown in FIG. 27, the coefficients of the restoration matrix corresponding to the non-restored region may be set to a predetermined value (for example, "0"). For example, assume that the coefficients of the restoration matrix ("Transfer Matrix (coefficients of simultaneous linear equations)" in FIG. 27) corresponding to the shaded portion of the restored image ("Scene Image (what is to be calculated)" in FIG. 27) are the shaded portion. When the shaded portion of this restored image is treated as the non-restored region, the coefficient values ​​of the shaded portion of the restoration matrix are set to "0". By doing so, even if the restored image is restored from the detected image by performing the matrix calculation as shown in FIG. 21, the pixel values ​​of the shaded portion of the restored image will be "0" (i.e., not restored).

[0294] Based on the restoration region information, the restoration matrix update unit 104 generates (sets) a non-restoration region corresponding restoration matrix in which the coefficients corresponding to the non-restoration region are "0." That is, the restoration matrix update unit 104 may generate (set) a partial restoration matrix used to restore only the restored image of the restoration region by setting the coefficients of the entire restoration matrix, which is the restoration matrix corresponding to the entire detected image, that do not correspond to the restoration region to a pre-registered value (e.g., "0").

[0295] When the restoration unit 105 restores a restored image from the detected image using this non-restored region corresponding restoration matrix (partial restoration matrix), a restored image in which the pixel values ​​of the non-restored region are "0" is obtained. In other words, by using this non-restored region corresponding restoration matrix, the restoration unit 105 can restore the restored image such that the non-restored region is not restored.

[0296] The "preregistered value" of the coefficient of the restoration matrix set by the restoration matrix update unit 104 may be any value other than "0". In other words, the "preregistered value" may be any value that does not make the subject visible through the matrix operation. For example, the "preregistered value" may or may not be uniform for all coefficients corresponding to the non-restoration area. For example, a different value may be set for each coefficient or each block as the "preregistered value".

[0297] Also, for example, as shown in Fig. 28, coefficients corresponding to the non-restoration region of the restoration matrix may be deleted. In the example of Fig. 28, coefficients corresponding to the non-restoration region (shaded area in Fig. 27) are deleted. In this way, a restoration matrix consisting of coefficients corresponding to regions other than the non-restoration region (i.e., a restoration matrix corresponding to the restoration region) is generated. By using this restoration matrix, it is possible to restore only the restored image of the restoration region from the detected image.

[0298] The restoration matrix update unit 104 generates (sets) a non-restoration region corresponding restoration matrix in which the coefficients corresponding to the non-restoration region have been deleted, i.e., the non-restoration region corresponding restoration matrix is ​​composed only of coefficients corresponding to the restoration region, based on the restoration region information. In other words, the restoration matrix update unit 104 may generate (set) a partial restoration matrix used to restore only the restored image of the restoration region by deleting coefficients that do not correspond to the restoration region from among the coefficients of the entire restoration matrix, which is the restoration matrix corresponding to the entire detected image.

[0299] When the restoration unit 105 restores a restored image from the detected image using this non-restored region corresponding restoration matrix (partial restoration matrix), a restored image in which the pixel values ​​of the non-restored region are "0" is obtained. In other words, by using this non-restored region corresponding restoration matrix, the restoration unit 105 can restore the restored image such that the non-restored region is not restored.

[0300] However, in this case, the non-restored region corresponding restoration matrix does not correspond to the entire detected image. That is, as described above, the non-restored region corresponding restoration matrix does not include a coefficient corresponding to the non-restored region. Therefore, information (for example, information indicating the restored region (or non-restored region) in the detected image) that makes each coefficient of the non-restored region corresponding restoration matrix correspond to a pixel of the detected image is required. Based on the information, the restoration unit 105 can correctly associate the restored region in the detected image with the non-restored region corresponding restoration matrix. Note that this information may be known to the restoration unit 105 in advance, or may be supplied from the restoration matrix update unit 104 to the restoration unit 105 together with the non-restored region corresponding restoration matrix.

[0301] <Flow of restored image generation process> An example of the flow of a restored image generation process in which the imaging device 100 captures an image of a subject and restores a restored image from the detected image so as not to restore a non-restored area will be described with reference to the flowchart of FIG.

[0302] When the restored image generation process is started, in step S301, the imaging section 101 of the imaging device 100 (FIG. 1) captures an image of a subject using the image sensor 121 and generates a detected image.

[0303] In step S302, the region identification unit 102 identifies a restoration region (or a non-restoration region) from the detection image generated in step S301 based on the learning data read from the learning data storage unit 103, and generates restoration region information indicating the restoration region (or non-restoration region information indicating the non-restoration region).

[0304] In step S303, the restoration matrix update unit 104 updates the restoration matrix corresponding to the entire detected image based on the restoration region information (or non-restoration region information) generated in step S302, and generates a restoration matrix corresponding to the non-restoration region.

[0305] In step S304, the restoration unit 105 restores the restored image from the detected image generated in step S301, using the unrestored region corresponding restoration matrix generated in step S303, so as not to restore the unrestored region.

[0306] In step S305, the restoration unit 105 outputs a restored image in which only the restored area is restored in step S304. Note that the process of step S304 may be omitted, and in step S305, the restoration unit 105 may output the detected image generated in step S301 and the non-restored area corresponding restoration matrix generated in step S303 in association with each other.

[0307] When the process of step S305 ends, the restored image generation process ends.

[0308] As described above, the area specifying unit 102 can control the area to be restored by specifying the restoration area (non-restoration area). Therefore, by performing each process as described above, the imaging device 100 can restore the restored image so as not to restore any area.

[0309] <3. Second embodiment> <3-1. Example of image processing system 1> In the above, the present technology has been described using an imaging device as an example, but the present technology is not limited to this example and can be applied to any configuration, such as an image processing device, an image encoding device, an image decoding device, an information processing device, a communication device, an electronic device, or a system consisting of multiple devices.

[0310] For example, the configuration of the imaging device 100 described above may be realized by a plurality of devices. FIG. 30 is a diagram showing a main configuration example of an embodiment of an image processing system to which the present technology is applied. The image processing system 400 shown in FIG. 30 is a system having the same functions as the imaging device 100, and is a system that captures an image of a subject and restores a restored image from the obtained detected image. That is, like the imaging device 100, the image processing system 400 can set a restoration area (non-restoration area) for the detected image as described above in <1. Identification of restoration area / non-restoration area>, and restore the restored image so as not to restore the non-restoration area.

[0311] 30, an image processing system 400 includes an imaging device 401 and an image processing device 402. The imaging device 401 can capture an image of a subject to generate a detected image, and output the detected image and a restoration matrix corresponding to the detected image.

[0312] The image processing device 402 can acquire a detection image supplied from the outside and a restoration matrix corresponding to the detection image, set a restoration area (or a non-restoration area) for the detection image, and generate restoration area information indicating the restoration area (or non-restoration area information indicating the non-restoration area). Furthermore, the image processing device 402 can update the acquired restoration matrix based on the restoration area information, and generate a non-restoration area corresponding restoration matrix used when restoring only the restoration area (or restoring the non-restoration area so as not to restore it). Furthermore, the image processing device 402 can use the non-restoration area corresponding restoration matrix to generate and output a restoration image in which only the restoration area is restored (or a restoration image in which the non-restoration area is restored so as not to restore it) from the acquired detection image.

[0313] In this way, the image processing device 402 can control, for example, whether any region of the detected image is made public or private, control whether information is protected or not protected against unauthorized leakage or use, control the scope of image analysis, etc., and control the level of services provided. In addition, since the detected image is transmitted when transmitting an image from the imaging device 401 to the image processing device 402, information can be protected and the occurrence of violation of privacy rights due to unauthorized leakage or use of the image can be suppressed.

[0314] The imaging device 401 has an imaging unit 101. The image processing device 402 has a region specifying unit 102 to a restoration unit 105. Each processing unit performs the same process as in the imaging device 100. That is, each processing unit of the imaging device 401 and the image processing device 402 performs a process, for example, according to a flow of the flowchart shown in FIG.

[0315] In step S401, the imaging unit 101 of the imaging device 401 captures an image of a subject using the image sensor 121 to generate a detection image. In step S402, the imaging device 401 supplies the detection image and a restoration matrix (global restoration matrix) corresponding to the detection image to the image processing device 402. In step S411, the image processing device 402 acquires the supplied detection image and restoration matrix.

[0316] It should be noted that information can be exchanged between the devices in any manner. For example, the detected image and the restoration matrix may be provided from the imaging device 401 to the image processing device 402 by communication via any communication medium. This communication method is arbitrary, and may be wired communication, wireless communication, or both. This communication may be performed via one or more networks, such as the Internet or a LAN (Local Area Network), or may be performed via other devices, such as a communication device or a relay device. Furthermore, this communication may be performed via a server or so-called cloud computing.

[0317] Furthermore, information may be exchanged via any storage medium, such as a removable flash memory, a hard disk, etc. In other words, the information to be exchanged may be stored in any storage medium connected to a source device, the storage medium may be connected to a destination device, and the information may be read from the storage medium.

[0318] In step S412, the area identification unit 102 of the image processing device 402 identifies a restoration area (or a non-restoration area) from the detection image acquired in step S411 based on the learning data read out from the learning data storage unit 103, and generates restoration area information indicating the restoration area (or non-restoration area information indicating the non-restoration area).

[0319] In step S413, the restoration matrix update unit 104 updates the restoration matrix acquired in step S411 based on the restoration region information (or non-restoration region information) to generate a non-restoration region corresponding restoration matrix.

[0320] In step S414, the restoration unit 105 restores (generates) a restored image from the detection image acquired in step S411 using the non-restored area corresponding restoration matrix generated in step S413 so as to restore only the restored area (in other words, so as not to restore the non-restored area).

[0321] In step S415, the image processing device 402 outputs the restored image generated in step S414 in which only the restored area has been restored (in other words, the restored image in such a way that the non-restored area is not restored), to the outside of the image processing device 402.

[0322] This output to the outside may be performed by communication via an arbitrary communication path, or may be performed via an arbitrary storage medium (i.e., the information may be stored in a storage medium and then output).

[0323] By executing each process in this manner, the image processing system 400 can control the area to be restored. Therefore, the image processing system 400 can control, for example, whether any area of ​​the detected image is made public or private, control whether information is protected or not protected against unauthorized leakage or use, control the target range of image analysis, etc., and control the level of service to be provided, for example, for any area of ​​the detected image. In addition, since the detected image is transmitted in transmitting the image from the imaging device 401 to the image processing device 402, information can be protected and the occurrence of violation of privacy rights due to unauthorized leakage or use of the image can be suppressed.

[0324] Note that the process of step S414 may be omitted, and in step S415, the image processing device 402 may associate the detected image with the non-restored area corresponding restoration matrix and output the associated image to the outside. That is, the restoration of the restored image may be performed outside the image processing device 402, and in that case, the restoration unit 105 does not perform this restoration, but associates the detected image with the non-restored area corresponding restoration matrix and outputs the associated image to a restoration device that performs restoration processing outside the image processing device 402. In this way, the image processing device 402 can transmit the image to the restoration device (another device) in a state where the information is protected. Therefore, it is possible to prevent the occurrence of violation of privacy rights due to unauthorized leakage or unauthorized use of images.

[0325] Furthermore, in the image processing system 400 as described above, the restoration unit 105 may be provided in the imaging device 401. In this case, the restoration matrix update unit 104 supplies the derived restoration matrix corresponding to the non-restoration area to the restoration unit 105 of the imaging device 401. That is, the restoration matrix corresponding to the non-restoration area is transmitted from the image processing device 402 to the imaging device 401. Then, the restoration unit 105 of the imaging device 401 restores (generates) a restored image from the detected image supplied from the imaging unit 101, using the restoration matrix corresponding to the non-restoration area supplied from the restoration matrix update unit 104 of the image processing device 402, so as to restore only the restoration area (in other words, so as not to restore the non-restoration area).

[0326] Furthermore, the restoration matrix update unit 104 may also be provided in the imaging device 401. In this case, the region identification unit 102 supplies restoration region information to the restoration matrix update unit 104 of the imaging device 401. That is, the restoration region information is transmitted from the image processing device 402 to the imaging device 401. Then, the restoration matrix update unit 104 of the imaging device 401 updates the restoration matrix supplied from the imaging unit 101 based on the restoration region information supplied from the region identification unit 102 of the image processing device 402, and generates a restoration matrix corresponding to a non-restoration region. The restoration unit 105 of the imaging device 401 restores (generates) a restored image from the detected image supplied from the imaging unit 101 by using the non-restoration region-corresponding restoration matrix so as to restore only the restoration region (in other words, so as not to restore the non-restoration region).

[0327] Even in these cases, when transmitting an image from the imaging device 401 to the image processing device 402, the detected image is transmitted, so that the image processing system 400 can protect information and prevent the occurrence of violations of privacy rights due to unauthorized leakage or unauthorized use of images, etc.

[0328] <3-2. Example of image processing system 2> Fig. 32 is a diagram showing a main configuration example of another embodiment of an image processing system to which the present technology is applied. Similar to the image processing system 400 (Fig. 30), the image processing system 410 shown in Fig. 32 is a system that captures an image of a subject and restores a restored image from the obtained detected image, and can set a restored area (non-restored area) for the detected image and restore the restored image so as not to restore the non-restored area.

[0329] 32, an image processing system 410 includes an imaging device 411, an image processing device 412, and a restoration device 413. Like the imaging device 401, the imaging device 411 can capture an image of a subject to generate a detected image and output the detected image and a restoration matrix corresponding to the detected image.

[0330] The image processing device 412 can obtain a detection image supplied from the outside and a restoration matrix corresponding to the detection image, set a restoration area (or a non-restoration area) for the detection image, and generate restoration area information indicating the restoration area (or non-restoration area information indicating the non-restoration area).The image processing device 412 can also update the obtained restoration matrix based on the restoration area information, and generate and output a non-restoration area corresponding restoration matrix used when restoring only the restoration area (or restoring so as not to restore the non-restoration area).

[0331] The restoration device 413 can acquire a restoration matrix corresponding to the non-restored area and a detected image supplied from the outside, and use the restoration matrix corresponding to the non-restored area to generate and output a restored image in which only the restored area is restored from the detected image (or a restored image in which the non-restored area is not restored).

[0332] The imaging device 411 has an imaging unit 101. The image processing device 412 has components from the region specifying unit 102 to the restoration matrix updating unit 104. The restoration device 413 has a restoration unit 105. Each processing unit performs the same processing as in the imaging device 100. That is, each processing unit of the imaging device 411, the image processing device 412, and the restoration device 413 performs processing, for example, in a flow like the flowchart shown in FIG.

[0333] In step S421, the imaging section 101 of the imaging device 411 captures an image of a subject using the image sensor 121, and generates a detection image.

[0334] In step S422, the imaging device 411 supplies the detected image and a restoration matrix (global restoration matrix) corresponding to the detected image to the image processing device 412. In step S431, the image processing device 412 acquires the supplied detected image and restoration matrix.

[0335] In step S423, the imaging device 411 supplies the detected image to the restoration device 413. In step S441, the restoration device 413 acquires the supplied detected image. Note that, as in the case of the image processing system 400, information exchange between the devices can be performed by any method.

[0336] In step S432, the area identification unit 102 of the image processing device 412 identifies a restoration area (or a non-restoration area) from the detection image acquired in step S431 based on the learning data read out from the learning data storage unit 103, and generates restoration area information indicating the restoration area (or non-restoration area information indicating the non-restoration area).

[0337] In step S433, the restoration matrix update unit 104 updates the restoration matrix acquired in step S431 based on the restoration region information (or non-restoration region information) to generate a non-restoration region corresponding restoration matrix.

[0338] In step S434, the image processing device 412 supplies the restoration matrix corresponding to the non-restored region to the restoration device 413. In step S442, the restoration device 413 acquires the supplied restoration matrix corresponding to the non-restored region.

[0339] In step S443, the restoration unit 105 of the restoration device 413 restores (generates) a restored image from the detection image obtained in step S441 using the non-restored area corresponding restoration matrix obtained in step S442, so as to restore only the restored area (in other words, so as not to restore the non-restored area).

[0340] In step S444, the restoration device 413 outputs the restored image generated in step S443 in which only the restoration area has been restored (in other words, the restored image restored so as not to restore the non-restoration area), to the outside of the restoration device 413.

[0341] This output to the outside may be performed by communication via an arbitrary communication path, or may be performed via an arbitrary storage medium (i.e., the information may be stored in a storage medium and then output).

[0342] In this way, the restoration device 413 can output an image with information protected for a desired area of ​​the image. This can prevent the occurrence of violation of privacy rights due to illegal leakage or unauthorized use of the image. In addition, by executing each process in this way, the image processing system 410 can control the area to be restored. Therefore, the image processing system 410 can control, for example, whether any area of ​​the detected image is made public or private, control the protection or non-protection of information against illegal leakage or unauthorized use, control the target range of image analysis, and control the level of service provided. Furthermore, since an image is transmitted from the imaging device 411 to the restoration device 413 as a detected image, this transmission can be performed with information protected, and the occurrence of violation of privacy rights due to illegal leakage or unauthorized use of the image can be prevented. In addition, a restoration matrix corresponding to the detected image is transmitted from the imaging device 401 to the restoration device 413 via a route different from that of the detected image (via the image processing device 412). Therefore, the illegal leakage of information can be further prevented.

[0343] Note that the process of step S443 may be omitted, and in step S444, the restoration device 413 may associate the detected image with the non-restored area corresponding restoration matrix and output the associated image to the outside. That is, the restoration of the restored image may be performed outside the restoration device 413. In this case, the restoration unit 105 does not perform this restoration, but associates the detected image with the non-restored area corresponding restoration matrix and outputs the associated image to another restoration device that performs restoration processing outside the restoration device 413. In this way, the restoration device 413 can transmit the image to another restoration device while protecting the information. Therefore, it is possible to prevent the occurrence of violation of privacy rights due to unauthorized leakage or unauthorized use of images.

[0344] In the above, the detected image is described as being supplied from the imaging device 411 to the restoration device 413, but the detected image may be supplied via the image processing device 412. That is, the image processing device 412 may supply the detected image supplied from the imaging device 411 and the non-restored area corresponding restoration matrix generated for the detected image to the restoration device 413. Even in this case, the image is transmitted as a detected image from the imaging device 411 to the restoration device 413, so that unauthorized leakage of information can be further suppressed.

[0345] <3-3. Example of image processing system 3> Fig. 34 is a diagram showing a main configuration example of another embodiment of an image processing system to which the present technology is applied. Similar to the image processing system 410 (Fig. 32), the image processing system 420 shown in Fig. 34 is a system that captures an image of a subject and restores a restored image from the obtained detected image, and can set a restored area (non-restored area) for the detected image and restore the restored image so as not to restore the non-restored area.

[0346] 34, the image processing system 420 includes an imaging device 421, an image processing device 422, and a restoration device 423. The imaging device 421 can capture an image of a subject to generate a detected image and output the detected image and a restoration matrix corresponding to the detected image.

[0347] The image processing device 422 can acquire a detection image supplied from outside and a restoration matrix corresponding to that detection image, set a restoration area (or a non-restoration area) for that detection image, and generate and output restoration area information indicating that restoration area (or non-restoration area information indicating the non-restoration area).

[0348] The restoration device 423 can acquire a non-restored area corresponding restoration matrix and a detected image supplied from the outside, update the acquired restoration matrix based on the restoration area information, and generate a non-restored area corresponding restoration matrix used when restoring only the restoration area (or restoring the non-restored area without restoring it). Furthermore, the restoration device 423 can use the non-restored area corresponding restoration matrix to generate and output a restored image in which only the restoration area is restored (or a restored image in which the non-restored area is not restored) from the acquired detected image.

[0349] The imaging device 421 has an imaging unit 101. The image processing device 422 has a region specifying unit 102 and a learning data storage unit 103. The restoration device 423 has a restoration matrix updating unit 104 and a restoration unit 105. Each processing unit performs the same process as in the imaging device 100. That is, each processing unit of the imaging device 421, the image processing device 422, and the restoration device 423 performs a process, for example, according to a flow like the flowchart shown in FIG.

[0350] In step S451, the imaging section 101 of the imaging device 421 captures an image of a subject using the image sensor 121, and generates a detection image.

[0351] In step S452, the imaging device 421 supplies the detected image to the image processing device 422. In step S461, the image processing device 422 acquires the supplied detected image. Note that, as in the case of the image processing system 400, information exchange between the devices can be performed by any method.

[0352] In step S453, the imaging device 421 supplies the detected image and a restoration matrix (global restoration matrix) corresponding to the detected image to the restoration device 423. In step S471, the restoration device 423 acquires the supplied detected image and restoration matrix.

[0353] In step S462, the area identification unit 102 of the image processing device 422 identifies a restoration area (or a non-restoration area) from the detection image acquired in step S461 based on the learning data read out from the learning data storage unit 103, and generates restoration area information indicating the restoration area (or non-restoration area information indicating the non-restoration area).

[0354] In step S463, the image processing device 422 supplies the restoration area information (or non-restoration area information) to the restoration device 423. In step S472, the restoration device 423 acquires the supplied restoration area information (or non-restoration area information).

[0355] In step S473, the restoration matrix update unit 104 of the restoration device 423 updates the restoration matrix acquired in step S471 based on the restoration area information (or non-restoration area information) acquired in step S472, and generates a restoration matrix corresponding to the non-restoration area.

[0356] In step S474, the restoration unit 105 uses the non-restored area corresponding restoration matrix to restore (generate) a restored image from the detection image acquired in step S471 so as to restore only the restored area (in other words, so as not to restore the non-restored area).

[0357] In step S475, the restoration device 423 outputs the restored image in which only the restoration area has been restored (in other words, the restored image restored so as not to restore the non-restoration area) generated in step S474 to the outside of the restoration device 423.

[0358] This output to the outside may be performed by communication via an arbitrary communication path, or may be performed via an arbitrary storage medium (i.e., the information may be stored in a storage medium and then output).

[0359] In this way, the restoration device 423 can output an image with information protected for a desired area of ​​the image. This can prevent the occurrence of an infringement of privacy rights due to illegal leakage or unauthorized use of the image. In addition, by executing each process in this way, the image processing system 420 can control the area to be restored. Therefore, the image processing system 420 can control, for example, whether any area of ​​the detected image is made public or private, control the protection or non-protection of information against illegal leakage or unauthorized use, control the target range of image analysis, and control the level of service to be provided. Furthermore, since an image is transmitted from the imaging device 421 to the restoration device 423 as a detected image, this transmission can be performed with information protected, and the occurrence of an infringement of privacy rights due to illegal leakage or unauthorized use of the image can be prevented. In addition, in this case, the restoration matrix is ​​updated in the restoration device 423, so the load on the image processing device 422 can be reduced compared to the case of FIG. 32.

[0360] Note that the process of step S474 may be omitted, and in step S475, the restoration device 423 may associate the detected image with the non-restored area corresponding restoration matrix and output the associated image to the outside. That is, the restoration of the restored image may be performed outside the restoration device 423. In this case, the restoration unit 105 does not perform this restoration, but associates the detected image with the non-restored area corresponding restoration matrix and outputs the associated image to another restoration device that performs restoration processing outside the restoration device 423. In this way, the restoration device 423 can transmit the image to another restoration device while protecting the information. Therefore, it is possible to prevent the occurrence of violation of privacy rights due to unauthorized leakage or unauthorized use of images.

[0361] In the above, the detected image is described as being supplied from the imaging device 421 to the restoration device 423, but the detected image may be supplied to the restoration device 423 via the image processing device 422. That is, the image processing device 422 may supply the detected image supplied from the imaging device 421 and the restoration area information (or the non-restoration area information) generated for the detected image to the restoration device 423. Similarly, the restoration matrix may be supplied to the restoration device 423 via the image processing device 422. That is, the image processing device 422 may supply the restoration matrix supplied from the imaging device 421 and the restoration area information (or the non-restoration area information) generated for the restoration matrix to the restoration device 423. Of course, the detected image and the restored image may be supplied to the restoration device 423 via the image processing device 422.

[0362] <3-4. Example of image processing system 4> Fig. 36 is a diagram showing a main configuration example of another embodiment of an image processing system to which the present technology is applied. Similar to the image processing system 420 (Fig. 34), the image processing system 430 shown in Fig. 36 is a system that captures an image of a subject and restores a restored image from the obtained detected image, and can set a restored area (non-restored area) for the detected image and restore the restored image so as not to restore the non-restored area.

[0363] 36, the image processing system 430 has an imaging device 431 and a restoration device 432. The imaging device 431 can capture an image of a subject to generate a detection image and output the detection image and a restoration matrix corresponding to the detection image. The imaging device 431 can also set a restoration area (or a non-restoration area) for the detection image, generate and output restoration area information indicating the restoration area (or non-restoration area information indicating the non-restoration area).

[0364] The restoration device 432 can acquire a detection image supplied from the outside, a restoration matrix corresponding to the detection image, and restoration area information. The restoration device 432 can also update the acquired restoration matrix based on the acquired restoration area information, and generate a non-restoration area corresponding restoration matrix used when restoring only the restoration area (or restoring the non-restoration area so as not to restore it). Furthermore, the restoration device 432 can use the non-restoration area corresponding restoration matrix to generate and output a restoration image in which only the restoration area is restored (or a restoration image in which the non-restoration area is not restored) from the acquired detection image.

[0365] The imaging device 431 has an imaging unit 101, a region specifying unit 102, and a learning data storage unit 103. The restoration device 432 has a restoration matrix updating unit 104 and a restoration unit 105. Each processing unit performs the same process as in the imaging device 100. That is, each processing unit of the imaging device 431 and the restoration device 432 performs a process according to a flow such as the flowchart shown in FIG.

[0366] In step S481, the imaging section 101 of the imaging device 431 captures an image of a subject using the image sensor 121, and generates a detection image.

[0367] In step S482, the area identification unit 102 identifies a restoration area (or a non-restoration area) from the detection image generated in step S481 based on the learning data read out from the learning data storage unit 103, and generates restoration area information indicating the restoration area (or non-restoration area information indicating the non-restoration area).

[0368] In step S483, the imaging device 431 supplies the detected image, the restoration matrix corresponding to the detected image, and the restoration area information (or the non-restoration area information) to the restoration device 432. In step S491, the restoration device 432 acquires the supplied detected image, the restoration matrix, and the restoration area information (or the non-restoration area information). Note that, as in the case of the image processing system 400, information can be exchanged between the devices by any method.

[0369] In step S492, the restoration matrix update unit 104 of the restoration device 432 updates the restoration matrix acquired in step S491 based on the restoration area information (or non-restoration area information) acquired in step S491, and generates a restoration matrix corresponding to the non-restoration area.

[0370] In step S493, the restoration unit 105 uses the non-restored area corresponding restoration matrix to restore (generate) a restored image from the detection image acquired in step S491 so as to restore only the restored area (in other words, so as not to restore the non-restored area).

[0371] In step S494, the restoration device 432 outputs the restored image in which only the restoration area has been restored (in other words, the restored image in which the non-restoration area has not been restored), generated in step S493, to the outside of the restoration device 432.

[0372] This output to the outside may be performed by communication via an arbitrary communication path, or may be performed via an arbitrary storage medium (i.e., the information may be stored in a storage medium and then output).

[0373] In this way, the restoration device 432 can output an image with information protected for a desired area of ​​the image. This can prevent the occurrence of an infringement of privacy rights due to the illegal leakage or unauthorized use of an image. In addition, by executing each process in this way, the image processing system 430 can control the area to be restored. Therefore, the image processing system 430 can control, for example, whether any area of ​​the detected image is made public or private, control the protection or non-protection of information against illegal leakage or unauthorized use, control the target range of image analysis, etc., and control the level of services to be provided. Furthermore, since an image is transmitted from the imaging device 431 to the restoration device 432 as a detected image, this transmission can be performed with information protected, and the occurrence of an infringement of privacy rights due to the illegal leakage or unauthorized use of an image can be prevented.

[0374] Note that the process of step S493 may be omitted, and in step S494, the restoration device 432 may associate the detected image with the non-restored area corresponding restoration matrix and output the associated image to the outside. That is, the restoration of the restored image may be performed outside the restoration device 432. In this case, the restoration unit 105 does not perform this restoration, but associates the detected image with the non-restored area corresponding restoration matrix and outputs the associated image to another restoration device that performs restoration processing outside the restoration device 432. In this way, the restoration device 432 can transmit the image to another restoration device while protecting the information. Therefore, it is possible to prevent the occurrence of violation of privacy rights due to unauthorized leakage or unauthorized use of images.

[0375] <3-5. Example of Image Processing System 5> Fig. 38 is a diagram showing a main configuration example of another embodiment of an image processing system to which the present technology is applied. Similar to the image processing system 430 (Fig. 36), the image processing system 440 shown in Fig. 38 is a system that captures an image of a subject and restores a restored image from the obtained detected image, and can set a restored area (non-restored area) for the detected image and restore the restored image so as not to restore the non-restored area.

[0376] As shown in FIG. 38, the image processing system 440 has an imaging device 441 and a restoration device 442. The imaging device 441 can capture an image of a subject to generate and output a detection image. The imaging device 441 can also set a restoration area (or a non-restoration area) for the detection image and generate restoration area information indicating the restoration area (or non-restoration area information indicating the non-restoration area). Furthermore, the imaging device 441 can update a restoration matrix corresponding to the generated detection image based on the restoration area information, and generate and output a non-restoration area corresponding restoration matrix used when restoring only the restoration area (or restoring so as not to restore the non-restoration area).

[0377] The restoration device 442 can obtain a detected image and a restoration matrix corresponding to a non-restored area supplied from the outside. In addition, the restoration device 442 can generate and output a restored image in which only the restoration area is restored from the detected image (or a restored image in which the non-restored area is not restored) using the non-restored area corresponding restoration matrix.

[0378] The imaging device 441 includes an imaging unit 101 to a restoration matrix update unit 104. The restoration device 442 includes a restoration unit 105. Each processing unit performs the same process as in the imaging device 100. That is, each processing unit of the imaging device 441 and the restoration device 442 performs a process according to a flow such as the flowchart shown in FIG.

[0379] In step S501, the imaging section 101 of the imaging device 441 captures an image of a subject using the image sensor 121, and generates a detection image.

[0380] In step S502, the region identification unit 102 identifies a restoration region (or a non-restoration region) from the detection image generated in step S501 based on the learning data read out from the learning data storage unit 103, and generates restoration region information indicating the restoration region (or non-restoration region information indicating the non-restoration region).

[0381] In step S503, the restoration matrix update unit 104 updates the restoration matrix corresponding to the detected image generated in step S501 based on the restoration area information (or non-restoration area information) generated in step S502, and generates a restoration matrix corresponding to the non-restoration area.

[0382] In step S504, the imaging device 441 supplies the detected image and the non-restored region corresponding restoration matrix to the restoration device 442. In step S511, the restoration device 442 acquires the supplied detected image and non-restored region corresponding restoration matrix. Note that, as in the case of the image processing system 400, information exchange between the devices can be performed by any method.

[0383] In step S512, the restoration unit 105 of the restoration device 442 restores (generates) a restored image from the detection image obtained in step S511 using the non-restored area corresponding restoration matrix obtained in step S511 so as to restore only the restored area (in other words, so as not to restore the non-restored area).

[0384] In step S513, the restoration device 442 outputs the restored image generated in step S512 in which only the restoration area has been restored (in other words, the restored image restored so as not to restore the non-restoration area), to the outside of the restoration device 442.

[0385] This output to the outside may be performed by communication via an arbitrary communication path, or may be performed via an arbitrary storage medium (i.e., the information may be stored in a storage medium and then output).

[0386] In this way, the restoration device 442 can output an image with information protected for a desired area of ​​the image. This can prevent the occurrence of an infringement of privacy rights due to the illegal leakage or unauthorized use of an image. In addition, by executing each process in this way, the image processing system 440 can control the area to be restored. Therefore, the image processing system 440 can control, for example, whether any area of ​​the detected image is made public or private, control the protection or non-protection of information against illegal leakage or unauthorized use, control the target range of image analysis, etc., and control the level of services to be provided. Furthermore, since an image is transmitted from the imaging device 441 to the restoration device 442 as a detected image, this transmission can be performed with information protected, and the occurrence of an infringement of privacy rights due to the illegal leakage or unauthorized use of an image can be prevented.

[0387] Note that the process of step S512 may be omitted, and in step S513, the restoration device 442 may associate the detected image and the non-restored area corresponding restoration matrix acquired in step S511 and output them to the outside. That is, the restoration of the restored image may be performed outside the restoration device 442, and in that case, the restoration unit 105 does not perform this restoration, but associates the detected image with the non-restored area corresponding restoration matrix and outputs them to another restoration device that performs restoration processing outside the restoration device 442. In this way, the restoration device 442 can transmit the image to another restoration device in a state where the information is protected. Therefore, it is possible to prevent the occurrence of violation of privacy rights due to illegal leakage or illegal use of images.

[0388] <4. Usage example> Next, examples of the use of devices, systems, etc. to which the present technology is applied will be described.

[0389] <In-vehicle devices and systems> In recent years, devices and systems have been developed that install cameras in vehicles, analyze images captured by the cameras, and provide appropriate warnings, attention calls, guidance, etc. based on the analysis results. Devices and systems have also been developed that control the operation of the accelerator, brakes, steering wheel, etc. (operations related to driving the vehicle) based on the analysis results. Furthermore, so-called drive recorders that store captured images as records have also been developed. The present technology may be applied to in-vehicle devices and in-vehicle systems that are installed in such vehicles.

[0390] In that case, the in-vehicle device (or in-vehicle system) may generate a detected image instead of the above-mentioned captured image, and may set a restoration area (or a non-restoration area) for the detected image.

[0391] In this case, for example, the restoration area or the non-restoration area may be set according to the traveling speed of the vehicle in which the in-vehicle device (or in-vehicle system) is installed. For example, during high-speed driving (driving at a predetermined speed or higher), the far side (far, wide angle) may be set as the restoration area, and during low-speed driving (driving below the predetermined speed), the near side (near, narrow angle) may be set as the restoration area. In other words, during high-speed driving, the near side (near, narrow angle) may be set as the non-restoration area, and during low-speed driving, the far side (far, wide angle) may be set as the non-restoration area.

[0392] Also, a restoration area (or a non-restoration area) may be set according to the subject. For example, a partial area including an obstacle entering the angle of view at a speed higher than a predetermined threshold or an obstacle entering in a predetermined direction may be set as a restoration area or a non-restoration area. Also, a partial area including a child may be set as a restoration area, but the ball may not be included in the restoration area. Also, road signs and the like may be included in the restoration area.

[0393] In this way, the in-vehicle device (or the in-vehicle system) can restore the restored image from the detected image so as to restore only the restored area (not restore the non-restored area), and restore only the desired area (or not restore the desired area). In other words, it is possible to control the area to be restored.

[0394] In this way, for example, when performing image analysis on a restored image, image analysis of the unnecessary area can be omitted (or simplified) by not restoring the unnecessary area, and an increase in load can be suppressed. Also, for example, it is possible to suppress errors in situation judgment and control due to the image analysis results of the unnecessary area (or unnecessary subject, etc.).

[0395] In addition, when the detected image is stored as a record, as in a drive recorder, an area including a person's face, a license plate, a windshield, etc. may be set as a restoration area or a non-restoration area. In this way, it is possible to prevent the occurrence of a privacy violation due to the leakage of personal information, etc.

[0396] In this case, the non-restored area may be restored depending on the purpose (recipient). For example, when providing a video to a news organization, the restored image may be restored using a non-restored area corresponding restoration matrix so as not to restore the non-restored area, and when providing the video to the police, the restored image may be restored using a whole restoration matrix so as to restore the non-restored area as well. That is, for example, a restored image in which only the restoration area is restored (or a restored image in which the non-restored area is not restored) may be provided to the news organization, and a restored image in which the whole (restoration area and non-restoration area) is restored may be provided to the police. Note that the detected image and the non-restored area corresponding restoration matrix may be provided to the news organization, and the restored image in which only the restoration area is restored (or a restored image in which the non-restored area is not restored) may be restored by the news organization. Also, the detected image and the restoration matrix (whole restoration matrix) corresponding to the detected image may be provided to the police, and the restored image in which the whole (restoration area and non-restoration area) is restored may be restored by the police.

[0397] <Smartphones, etc.> The present technology may also be applied to a portable information processing terminal device having an imaging function, such as a so-called smartphone, a tablet terminal, a notebook personal computer, etc. The present technology may also be applied to an imaging device, such as a digital camera or a video camera. In this case, the portable information processing terminal device or the imaging device may be configured to capture an image of a subject and generate a detection image instead of a captured image, and a restoration area (or a non-restoration area) may be set for the detection image.

[0398] In this case, for example, the entire background may be set as a non-restoration area and masked. Also, anything other than one's own face (or a plurality of faces registered in advance) may be set as a non-restoration area and not restored. Furthermore, only the area around the eyes may be set as a non-restoration area and not restored. In this way, it is possible to prevent the occurrence of privacy violations due to the leakage of personal information, etc.

[0399] <Surveillance camera> The present technology may also be applied to a device or system that performs monitoring using a surveillance camera. In this case, the surveillance camera may capture an image of a subject and generate a detection image instead of a captured image, and a restoration area (or a non-restoration area) may be set for the detection image.

[0400] In this case, for example, a person's entire body, a person's face, a person's hands, etc. may be set as a non-restoration area and not restored. For example, in the case of a surveillance camera in a space where an ATM (Automatic Teller Machine) is installed, the hands of a user who enters a PIN may be set as a non-restoration area and not restored. Also, a specific building, vehicle, etc. may be set as a non-restoration area and not restored.

[0401] In this case, too, the non-restored area may be restored depending on the application. For example, when providing the video to a news organization, the restored image may be restored using a non-restored area corresponding restoration matrix so as not to restore the non-restored area, and when providing the video to the police, the restored image may be restored using a whole restoration matrix so as to restore the non-restored area as well.

[0402] <Service provision level control> In addition, for example, the present technology may be applied to an image distribution service or the like. For example, in a system in which the above-mentioned detected image is distributed from a server and a restored image is restored from the detected image in a user's terminal device, the area to be restored may be controlled (the restored area may be specified) according to the authority of the user. For example, if a child of user A is set as a restoration target for the authority of user A in advance and the detected image provided to user A includes the child of user A, the server provides user A with a non-restored area corresponding restoration matrix in which an area including the child of user A is set as the restored area together with the detected image, and the restored image is restored using the non-restored area corresponding restoration matrix. In other words, it is possible to make only a partial area including a subject (child of user A) permitted to user A visible.

[0403] On the other hand, when providing the same detected image to user B, the server provides user B with a non-restored area corresponding restoration matrix in which an area including a restoration target permitted for user B is set as a restoration area. In other words, it is possible to make only a partial area including a subject permitted for user B visible to user B.

[0404] In this way, by setting the restoration area for each user (user authority) and providing the non-restoration area corresponding restoration matrix, the server can control the area to be restored for each user (user authority). In other words, the server can provide services (control the service provision level) for each user (user authority).

[0405] <Other examples> The non-restored area of ​​the restored image may be processed. For example, another image may be synthesized with the non-restored area. For example, when the face part of a person is set as the non-restored area, an appropriate sample face image may be synthesized with the non-restored area. By doing so, the restored image is decorated, and the deterioration of the appearance of the restored image can be suppressed compared to the case where the non-restored area is expressed by a black image or the like.

[0406] <5. Third embodiment> <In-vehicle equipment> Next, an example of an in-vehicle device among the above-mentioned use examples will be described in more detail. The in-vehicle device may have any configuration, but may have a configuration as shown in FIG. 40, for example. FIG. 40 is a diagram showing a main configuration example of an in-vehicle device that is an embodiment of an image processing device to which the present technology is applied. The in-vehicle device 500 shown in FIG. 40 is a device that can capture an image of a subject, analyze the captured image, and control operations (operations related to driving a vehicle) of the accelerator, brake, steering wheel, etc. based on the analysis result, control operations related to vehicle equipment such as turning on headlights, operating turn signals, operating air conditioning devices and display devices, perform warnings, attention calls, guidance, etc., and store the captured image as a record. As in the case of the imaging device 100, etc., the in-vehicle device 500 can set a restoration area (non-restoration area) for the detected image as the captured image to be analyzed, and restore the restored image so as not to restore the non-restoration area.

[0407] Note that Fig. 40 shows the main processing units, data flows, and the like, and is not limited to all that is shown in Fig. 40. That is, in on-vehicle device 500, there may be processing units that are not shown as blocks in Fig. 40, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 40.

[0408] 40, the in-vehicle device 500 includes the imaging unit 101 through the restoration unit 105, etc., in the same manner as the imaging device 100. These processing units basically perform the same processes as those in the imaging device 100.

[0409] The in-vehicle device 500 also has a sensor unit 511. The sensor unit 511 is a detection unit that has any sensor and can detect any information related to the subject or the periphery of the imaging unit 101 (i.e., at least one of the vehicle and the periphery of the vehicle). For example, the sensor unit 511 can have sensors such as a distance measurement sensor, a temperature sensor, and an acceleration sensor. Of course, the sensor unit 511 may have sensors other than these. For example, the sensor unit 511 may have sensors that detect the operation state of the accelerator, brake, steering wheel, etc. Also, the sensor unit 511 may have sensors that detect the operation state of other equipment. Furthermore, the sensor unit 511 may have sensors that detect the collision of the vehicle, such as an acoustic sensor or an impact sensor. The number of sensors is arbitrary, and may be single or multiple. Also, the number of pieces of information detected by the sensor unit 511 is arbitrary. For example, the sensor unit 511 may have multiple types of sensors and detect multiple types of information. The sensor unit 511 can supply the detected information to the area identification unit 102 as sensor information.

[0410] As described above, the region specifying unit 102 sets a restoration region (or a non-restoration region) for the detected image and generates restoration region information (or non-restoration region information). At this time, the region specifying unit 102 can set the restoration region (non-restoration region) based on the learning data and the sensor information.

[0411] For example, the region identification unit 102 may determine whether the subject is a person or not based on sensor information (temperature distribution) detected by a temperature sensor, and set a restoration region (non-restoration region) based on the determination result. The region identification unit 102 may also set a restoration region (non-restoration region) according to sensor information (e.g., vehicle speed) detected by an acceleration sensor. Furthermore, the region identification unit 102 may set a restoration region (non-restoration region) according to sensor information (e.g., distance to another vehicle or obstacle) detected by a distance measurement sensor.

[0412] Furthermore, the region identification unit 102 may detect a sudden operation of the accelerator, brake, steering wheel, etc., based on the operation state of the accelerator, brake, steering wheel, etc. detected by the sensor unit 511, and set a restoration region (non-restoration region) according to the detection result. Furthermore, the region identification unit 102 may set a restoration region (non-restoration region) based on a sound or impact detected by the sensor unit 511.

[0413] The restoration area (non-restoration area) may be set arbitrarily, or may be selected (switched) from among predetermined options prepared in advance according to conditions. For example, the restoration area may be switched when it is determined that there is a high possibility of collision with another vehicle based on sensor information from a distance measurement sensor.

[0414] By setting the restoration region (or non-restoration region) using the sensor information as well in this way, the region specifying unit 102 can set the restoration region (or non-restoration region) more accurately.

[0415] Furthermore, the in-vehicle device 500 has an analysis unit 512, a control unit 513, a bus 520, an input unit 521, a display unit 522, a storage unit 523, a storage medium 524, and a communication unit 525. The display unit 522 to the communication unit 525 are also referred to as an output unit 530 because they can perform processing related to the output of information.

[0416] The analysis unit 512 may acquire the restored image (a restored image in which only the restored area is restored) supplied from the restoration unit 105 via the bus 520. The analysis unit 512 may analyze the restored image in which only the restored area is restored, and may identify, for example, other vehicles, people, obstacles, etc., the speed and attitude of the vehicle, and the conditions around the vehicle, such as temperature and brightness. The analysis unit 512 may supply the analysis result to the control unit 513 via the bus 520.

[0417] The control unit 513 can obtain the analysis results supplied from the analysis unit 512 via the bus 520. Based on the analysis results, the control unit 513 can control operations of the accelerator, brake, steering wheel, etc. (operations related to driving the vehicle), control operations related to vehicle equipment such as turning on the headlights, operating the turn signals, operating the air conditioning device and display device, issue warnings, attention calls, guidance, etc., and store captured images as records.

[0418] The input unit 521 has any input device, such as a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc., and can receive information input by a user's operation or information supplied from another device via the input device. The input unit 521 can supply the received information to another processing unit (any processing unit) as appropriate via the bus 520.

[0419] Furthermore, the restored image output from the restoration unit 105 (a restored image in which only the restored area is restored) can be supplied to a display unit 522 , a storage unit 523 , or a communication unit 525 via a bus 520 .

[0420] The display unit 522 has a monitor capable of displaying an image, which is configured, for example, with a liquid crystal panel or an organic EL (Electro Luminescence) panel. The display unit 522 can display, for example, a restored image supplied from the restoration unit 105. In addition, the display unit 522 can display, for example, a GUI (Graphical User Interface) such as a menu, setting information of the in-vehicle device 500, and the like. Furthermore, the display unit 522 can display a restored image stored in the storage medium 524. For example, the display unit 522 can display a thumbnail image of the restored image stored in the storage medium 524.

[0421] The storage unit 523 controls storage of a storage medium 524 formed of, for example, a semiconductor memory. This storage medium 524 may be a removable storage medium, or may be a storage medium built into the in-vehicle device 500. For example, the storage unit 523 can store a restored image supplied via the bus 520 in the storage medium 524. The storage unit 523 can also store a detected image and a non-restored area corresponding restoration matrix, which are supplied via the bus 520 and are associated with each other, in the storage medium 524.

[0422] Also, for example, the storage unit 523 can read out data stored in the storage medium 524 and supply it to other processing units (for example, the display unit 522, the communication unit 525, etc.) via the bus 520. For example, the storage unit 523 can read out a restored image (or a thumbnail image thereof) from the storage medium and supply it to the display unit 522 via the bus 520 for display. Also, for example, the storage unit 523 can read out a restored image, or a detected image and a non-restored area corresponding restoration matrix from the storage medium and supply it to the communication unit 525 via the bus 520 for transmission to another device.

[0423] The communication unit 525 communicates with a server on the Internet, other devices on a wired or wireless LAN, other external devices, etc., by any communication method. For example, the communication unit 525 transmits, by that communication, to a communication partner (external device) a restored image or a detected image and a non-restored region corresponding restoration matrix supplied from the restoration unit 105, the storage unit 523, etc. via the bus 520. For example, the communication unit 525 can transmit the data by a streaming method or an upload method.

[0424] The region specifying unit 102 can control the region to be restored by specifying the restoration region (non-restoration region). Therefore, by having the above-mentioned configuration, the in-vehicle device 500 can restore the restored image so as not to restore any region, similarly to the case of the imaging device 100.

[0425] This allows the in-vehicle device 500 to obtain the effects described above in <4. Usage Examples>, for example, in image analysis, image storage, and the like.

[0426] Each of these processing units (imaging unit 101 to restoration unit 105, sensor unit 511, input unit 521 to communication unit 525) of the in-vehicle device 500 may have any configuration. For example, each processing unit may be configured by a logic circuit that realizes the above-mentioned processing. Also, each processing unit may have, for example, a CPU, a ROM, a RAM, etc., and may realize the above-mentioned processing by executing a program using them. Of course, each processing unit may have both configurations, and may realize part of the above-mentioned processing by a logic circuit and the other by executing a program. The configurations of each processing unit may be independent of each other, and for example, some processing units may realize part of the above-mentioned processing by a logic circuit, other processing units may realize the above-mentioned processing by executing a program, and still other processing units may realize the above-mentioned processing by both a logic circuit and executing a program.

[0427] <Flow of restored image generation process> An example of the flow of a restored image generation process in which the in-vehicle device 500 captures an image of a subject and restores a restored image from the detected image so as not to restore a non-restored area will be described with reference to the flowchart of FIG.

[0428] When the restored image generating process is started, in step S551, the imaging unit 101 of the in-vehicle device 500 (FIG. 40) captures an image of a subject using the imaging element 121 to generate a detected image.

[0429] In step S552, the sensor unit 511 performs sensing to detect predetermined information relating to the subject and the periphery of the imaging unit 101, and generates sensor information.

[0430] In step S553, the area identification unit 102 identifies a restoration area (or a non-restoration area) from the detection image generated in step S551 based on the learning data read from the learning data storage unit 103 and the sensor information obtained in step S552, and generates restoration area information indicating the restoration area (or non-restoration area information indicating the non-restoration area).

[0431] In step S554, the restoration matrix update unit 104 updates the restoration matrix corresponding to the entire detected image based on the restoration region information (or non-restoration region information) generated in step S553, and generates a restoration matrix corresponding to the non-restoration region.

[0432] In step S555, the restoration unit 105 restores the restored image from the detected image generated in step S551, using the unrestored region corresponding restoration matrix generated in step S554, so as not to restore the unrestored region.

[0433] In step S556, the output unit 530 outputs the restored image in which only the restored area has been restored in step S555. For example, the display unit 522 displays the restored image. The storage unit 523 stores the data of the restored image in the storage medium 524. The communication unit 525 transmits the data of the restored image to another device.

[0434] Note that the process of step S555 may be omitted, and the output unit 530 may output in step S556 the detected image generated in step S551 and the non-restored region corresponding restoration matrix generated in step S554 in association with each other. For example, the storage unit 523 may store the data of the detected image and the non-restored region corresponding restoration matrix in the storage medium 524. Furthermore, the communication unit 525 may transmit the data of the detected image and the non-restored region corresponding restoration matrix to another device.

[0435] When the process of step S556 ends, the restored image generation process ends.

[0436] As described above, the area specifying unit 102 can control the area to be restored by specifying the restored area (non-restored area). Therefore, by executing each process as described above, the in-vehicle device 500 can restore the restored image so as not to restore any area. As a result, the in-vehicle device 500 can obtain the effects as described above in <4. Use Example>, for example.

[0437] <6. Fourth embodiment> <Other configuration examples of image sensor> While an example of the image sensor 121 has been described above, the image sensor 121 may have any configuration as long as it has a plurality of pixel output units, and at least two of the plurality of pixel output units have different characteristics with respect to the angle of incidence of light incident from a subject.

[0438] For example, a random black and white pattern mask or an optical interference mask may be used as the modulation element, and the light incident on the imaging surface of the image sensor 121 may be modulated according to the black and white pattern or optical interference.

[0439] 42 shows another configuration of the image sensor. The image sensor 821 is configured such that a mask 823 is fixed to the image sensor 822 at a predetermined distance from the image sensor surface IP of the image sensor 822, and light from an object surface OP is modulated by the mask 823 and then enters the image sensor surface IP of the image sensor 822.

[0440] FIG. 43 shows a case where a black-and-white pattern mask is used. A of FIG. 43 illustrates a black-and-white pattern mask. The black-and-white pattern mask 823BW has a configuration in which white pattern parts that transmit light and black pattern parts that block light are randomly arranged, and the pattern size is set independently of the pixel size of the image sensor 822. B of FIG. 43 shows a schematic illustration of the irradiation state of the light emitted from the point light source PA and the light emitted from the point light source PB on the image sensor surface IP. In addition, B of FIG. 43 also shows a schematic illustration of an example of the response of the image sensor when the black-and-white pattern mask 823BW is used, for each of the light emitted from the point light source PA and the light emitted from the point light source PB. The light from the object surface OP is modulated by the black-and-white pattern mask 823BW and then enters the image sensor surface IP of the image sensor 822. Therefore, the response of the image sensor corresponding to the light emitted from the point light source PA on the object surface OP is Sbwa. The response of the image sensor corresponding to the light emitted from the point light source PB on the object plane OP is Sbwb. Therefore, the pixel output information output from the image sensor 822 is information obtained by combining the responses of each point light source for each pixel output unit.

[0441] FIG. 44 shows a case where an optical interference mask is used. As shown in A of FIG. 44, light emitted from point light sources PA and PB on the object plane OP is irradiated onto the imaging plane IP of the image sensor 822 through the optical interference mask 823LF. For example, the light incident surface of the optical interference mask 823LF is provided with unevenness of about the wavelength as shown in B of FIG. 44. Furthermore, the optical interference mask 823LF has a maximum transmission of light of a specific wavelength irradiated from the vertical direction. When the change in the incident angle (inclination with respect to the vertical direction) of the light of a specific wavelength emitted from the point light sources PA and PB on the object plane OP with respect to the optical interference mask 823LF increases, the optical path length changes. Here, when the optical path length is an odd multiple of a half wavelength, the light weakens each other, and when the optical path length is an even multiple of a half wavelength, the light strengthens each other. That is, the intensity of transmitted light of a specific wavelength that is emitted from the point light sources PA and PB and transmitted through the optical interference mask 823LF is modulated according to the angle of incidence with respect to the optical interference mask 823LF as shown in C of Fig. 44, and is incident on the imaging plane IP of the image sensor 822. Therefore, the pixel output information output from the image sensor 822 is information obtained by combining the light intensities after modulation of the point light sources for each pixel output unit.

[0442] Furthermore, the mask 823 may be configured such that the black pattern shown in Fig. 43 is used as a λ / 2 wave plate (λ is the wavelength) and linear polarizing elements are provided on the front and back sides of the mask with the same polarization direction. In this case, if light with wavelength λ is emitted from a point light source, the polarization plane of the polarized light transmitted through the λ / 2 wave plate is rotated, so that the amount of light is reduced compared to the white pattern portion, and pixel output information can be generated as in the case of Fig. 43. Note that if far-infrared light, for example, is used as the light of a specific wavelength emitted from the point light source, pixel output information corresponding to the subject in the imaging area can be generated even if the imaging area is dark.

[0443] However, in the case of these configurations, other configurations such as a mask must be added to the imaging element, so the imaging element 121 (FIG. 3) described above can be made smaller.

[0444] The imaging element 121 may have any of the configurations described with reference to Fig. 5, the configuration described with reference to Fig. 6, the configuration described with reference to Fig. 42 or Fig. 43, or the configuration described with reference to Fig. 44. In other words, as long as the imaging element 121 has any of these configurations, the present technology can be applied to them or to devices having them. In other words, the imaging element 121 may have a configuration that includes a plurality of pixel output units that receive incident light that enters without passing through an imaging lens or a pinhole, and that allows the incident angle directivities of output pixel values ​​of at least two pixel output units among the plurality of pixel output units with respect to incident light from a subject to have different characteristics from each other.

[0445] Moreover, the imaging element 121 may have a configuration as described with reference to Fig. 5 or a configuration as described with reference to Fig. 6. In other words, as long as the imaging element 121 has any of these configurations, the present technology can be applied to them or to a device having them. In other words, the multiple pixel output units of the imaging element 121 may have a configuration in which the incident angle directivity of the output pixel value can be set independently for each pixel output unit.

[0446] For example, the image sensor 121 may have the configuration described with reference to Fig. 5. In other words, as long as the image sensor 121 has such a configuration, the present technology can be applied to the image sensor 121 or to a device having the image sensor 121. In other words, the image sensor 121 may have a configuration in which the incident angle directivity for the incident light from the subject can be set independently for each pixel output unit.

[0447] Also, for example, the imaging element 121 may have a configuration as described with reference to Fig. 6. That is, as long as the imaging element 121 has such a configuration, the present technology can be applied to the imaging element 121 or an apparatus having the imaging element 121. That is, the multiple pixel output units of the imaging element 121 may have different PDs (Photo Diodes) that contribute to the output, so that the incident angle directivity of the output pixel value can be set independently for each pixel output unit.

[0448] <7. Notes> <Computer> The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, etc., capable of executing various functions by installing various programs.

[0449] FIG. 45 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes by a program.

[0450] In a computer 900 shown in FIG. 45, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0451] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0452] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives a removable recording medium 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0453] In the computer configured as above, the series of processes described above are performed by the CPU 901, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data and the like necessary for the CPU 901 to execute various processes as appropriate.

[0454] The program executed by the computer can be applied by recording it on a removable recording medium 921 such as a package medium, for example. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by mounting the removable recording medium 921 on the drive 915.

[0455] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0456] In addition, this program can also be installed in advance in the ROM 902 or the storage unit 913.

[0457] <Target of application of this technology> In addition, a system, device, processing unit, etc. to which the present technology is applied can be used in any field, such as transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, nature monitoring, etc. In addition, the use thereof is also arbitrary.

[0458] For example, the present technology can be applied to systems and devices used for providing ornamental content, etc. Furthermore, for example, the present technology can be applied to systems and devices used for traffic, such as monitoring traffic conditions and automatic driving control. Furthermore, for example, the present technology can be applied to systems and devices used for security. Furthermore, for example, the present technology can be applied to systems and devices used for automatic control of machines, etc. Furthermore, for example, the present technology can be applied to systems and devices used for agriculture and livestock farming. Furthermore, for example, the present technology can be applied to systems and devices that monitor natural conditions, such as volcanoes, forests, and oceans, and wildlife, etc. Furthermore, for example, the present technology can be applied to systems and devices used for sports.

[0459] <Other> The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0460] For example, the present technology can be implemented as any configuration that constitutes an apparatus or system, such as a processor as a system LSI (Large Scale Integration), a module using multiple processors, a unit using multiple modules, a set in which other functions are added to a unit, etc. (i.e., a configuration of a part of an apparatus).

[0461] Moreover, each of the above-mentioned processing units can be realized by any configuration. For example, it may be configured by a circuit, an LSI, a system LSI, a processor, a module, a unit, a set, a device, an apparatus, or a system. It may also be configured by combining a plurality of them. In this case, for example, it may be possible to combine the same type of configuration such as a plurality of circuits, a plurality of processors, etc., or it may be possible to combine different types of configuration such as a circuit and an LSI, etc.

[0462] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same case. Therefore, multiple devices housed in separate cases and connected via a network, and a single device in which multiple modules are housed in a single case, are both systems.

[0463] Also, for example, the configuration described above as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Also, it is of course possible to add configurations other than those described above to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the system as a whole are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0464] Furthermore, for example, the present technology can be configured as cloud computing in which one function is shared and processed jointly by multiple devices via a network. For example, the technology can be applied to a cloud service that provides services related to captured images (video images) to any terminal such as a computer, AV (Audio Visual) device, portable information processing terminal, or IoT (Internet of Things) device.

[0465] Furthermore, for example, the above-mentioned program can be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0466] Also, for example, each step described in the above flow chart can be executed by one device, or can be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes included in the one step can be executed by one device, or can be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0467] The steps of a program executed by a computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel or individually at a required timing such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no contradiction occurs. Furthermore, the steps of the program may be executed in parallel with the processing of other programs, or may be executed in combination with the processing of other programs.

[0468] The present technologies described in this specification can be implemented independently and individually, as long as no contradiction occurs. Of course, any of the present technologies can be implemented in combination. For example, a part or all of the present technology described in any of the embodiments can be implemented in combination with a part or all of the present technology described in another embodiment. In addition, any of the present technologies described above can be implemented in combination with other technologies not described above.

[0469] The present technology can also be configured as follows. (1) A region specifying unit that specifies a restoration region for restoring a restored image using a restoration matrix within a region of a detected image obtained in an imaging element having a plurality of pixels that receive incident light that is incident without passing through either an imaging lens or a pinhole, the imaging element having a configuration in which output pixel values ​​of at least two of the plurality of pixels have different incident angle directivities with respect to incident light from a subject. An image processing device comprising: (2) The restoration area is a partial area of ​​the detection image that corresponds to an area including a previously registered subject in the restoration image or to an area other than the area. An image processing device as described in (1). (3) The restoration region is a partial region of the detection image that corresponds to a region including a preregistered part of the subject in the restoration image or a region other than the preregistered part. An image processing device as described in (2). (4) The restoration area is a partial area of ​​the detection image that corresponds to an area including a human face in the restoration image or an area other than the human face. An image processing device as described in (3). (5) The restoration area is a partial area of ​​the detection image that corresponds to an area including a person registered in advance in the restoration image or to an area other than the area. An image processing device according to any one of (2) to (4). (6) The restoration area is a partial area of ​​the detection image that corresponds to an area including a subject that satisfies a preregistered condition in the restoration image or to an area other than the area. An image processing device according to any one of (1) to (5). (7) The restoration area is a partial area of ​​the detected image having a position and size registered in advance. An image processing device according to any one of (1) to (6). (8) A learning result storage unit that stores a learning result using the detection image and information on the restoration area corresponding to the detection image as teacher data, The region specifying unit specifies the restoration region of the detection image based on the learning result stored in the learning result storage unit. An image processing device according to any one of (1) to (7). (9) The image processing apparatus further includes a partial restoration matrix setting unit that sets a partial restoration matrix used for restoring only the restored image of the restoration area specified by the area specifying unit from the detected image. An image processing device according to any one of (1) to (8). (10) The partial restoration matrix setting unit sets the partial restoration matrix by setting coefficients of an entire restoration matrix, which is a restoration matrix corresponding to the entire detected image, that do not correspond to the restoration region to preregistered values. An image processing device according to (9). (11) The partial restoration matrix setting unit sets the partial restoration matrix by deleting coefficients that do not correspond to the restoration region from coefficients of an entire restoration matrix that is a restoration matrix corresponding to the entire detected image. The image processing device according to (9) or (10). (12) The image processing method according to the present invention further includes a restoration unit that restores a restored image of the restoration area from the detection image by using the partial restoration matrix set by the partial restoration matrix setting unit. An image processing device according to any one of (9) to (11). (13) The imaging element is further provided. An image processing device according to any one of (1) to (12). (14) The plurality of pixel output units have a configuration in which the incident angle directivity of the output pixel value can be set independently for each of the pixel output units. An image processing device according to (13). (15) The plurality of pixel output units have a configuration in which an incident angle directivity for incident light from the subject can be set independently for each of the pixel output units. The image processing device according to (13) or (14). (16) The plurality of pixel output units may have different PDs (Photo Diodes) contributing to output, so that the incident angle directivity of the output pixel value may be set independently for each of the pixel output units. An image processing device according to any one of (13) to (15). (17) A detection unit that detects at least one of information about the vehicle and information about the surroundings of the vehicle, The region specifying unit specifies the restoration region of the detection image obtained by the imaging element, using the information detected by the detection unit. An image processing device according to any one of (13) to (16). (18) The area specifying unit specifies the restoration area in accordance with the speed of the vehicle detected by the detection unit. The image processing device according to (17). (19) The region specifying unit specifies, as the restoration region, a partial region including an obstacle entering an angle of view detected by the detection unit at a speed faster than a threshold or an obstacle entering in a predetermined direction. The image processing device according to (17). (20) The area specifying unit specifies the restored area depending on a destination of the restored image. An image processing device according to any one of (1) to (19). (21) The area specifying unit specifies the restoration area according to a user's authority. An image processing device according to any one of (1) to (20). (22) A restoration area for restoring a restored image using a restoration matrix of a detected image obtained in an image sensor having a plurality of pixel output units that receive incident light that is incident without passing through an imaging lens or a pinhole, the image sensor having a configuration for making output pixel values ​​of at least two of the plurality of pixel output units have different incident angle directivities with respect to incident light from a subject. Image processing methods. (23) A computer A region specifying unit that specifies a restoration region for restoring a restored image using a restoration matrix of a detected image obtained in an image sensor having a configuration that includes a plurality of pixel output units that receive incident light that is incident without passing through either an imaging lens or a pinhole, and that is configured so that output pixel values ​​of at least two of the plurality of pixel output units have mutually different incident angle directivities with respect to incident light from a subject. A program that functions as a (24) An imaging device and an image processing device, The imaging device, An imaging device comprising a plurality of pixel output units that receive incident light that is incident without passing through either an imaging lens or a pinhole, the imaging device being configured so that output pixel values ​​of at least two of the plurality of pixel output units have mutually different incident angle directivities with respect to incident light from a subject. Equipped with The image processing device, A region specifying unit that specifies a restoration region for restoring a restored image by using a restoration matrix of a detected image obtained by the imaging element. An image processing system comprising: [Explanation of symbols]

[0470] 100 imaging device, 101 imaging unit, 102 region identification unit, 103 learning data storage unit, 104 restoration matrix update unit, 105 restoration unit, 121 imaging element, 400 image processing system, 401 imaging device, 402 image processing device, 410 image processing system, 411 imaging device, 412 image processing device, 413 restoration device, 420 image processing system, 421 imaging device, 422 image processing device, 423 restoration device, 430 image processing system, 431 imaging device, 432 restoration device, 440 image processing system, 441 imaging device, 442 restoration device, 500 in-vehicle device, 511 sensor unit, 520 bus, 521 input unit, 522 display unit, 523 storage section, 524 storage medium, 525 communication section, 530 output section

Claims

1. an area specifying unit that specifies a restoration area for restoring a restored image using a restoration matrix within an area of ​​a detected image obtained in an image sensor having a plurality of pixels that receive incident light that is incident without passing through either an imaging lens or a pinhole, the image sensor having a configuration in which output pixel values ​​of at least two of the plurality of pixels have different incident angle directivities with respect to incident light from a subject; a partial restoration matrix setting unit that sets a partial restoration matrix used for restoring only a restored image of the restoration area from the detected image by deleting coefficients that do not correspond to the restoration area specified by the area specifying unit from coefficients of an entire restoration matrix that is a restoration matrix corresponding to the entire detected image; An image processing device comprising:

2. The restoration area is a partial area of ​​the detection image that corresponds to an area including a previously registered subject in the restoration image or to an area other than the area. The image processing device according to claim 1 .

3. The restoration area is a partial area of ​​the detection image that corresponds to an area including a preregistered part of the subject in the restoration image or an area other than the area. The image processing device according to claim 2 .

4. The restoration area is a partial area of ​​the detection image that corresponds to an area including a person's face in the restoration image or an area other than the face. The image processing device according to claim 3 .

5. The restoration area is a partial area of ​​the detection image that corresponds to an area including a person registered in advance in the restoration image or to an area other than the area. The image processing device according to claim 2 .

6. The restoration area is a partial area of ​​the detection image that corresponds to an area including a subject that satisfies a preregistered condition in the restoration image or to an area other than the area. The image processing device according to claim 1 .

7. The restoration area is a partial area of ​​the detected image having a position and size registered in advance. The image processing device according to claim 1 .

8. a learning result storage unit configured to store a learning result using the detection image and information on the restoration area corresponding to the detection image as teacher data; The region specifying unit specifies the restoration region of the detection image based on the learning result stored in the learning result storage unit. The image processing device according to claim 1 .

9. The imaging element further comprises: The image processing device according to claim 1 .

10. The partial restoration matrix setting unit sets the partial restoration matrix by setting coefficients of an entire restoration matrix, which is a restoration matrix corresponding to the entire detected image, that do not correspond to the restoration region to pre-registered values. The image processing device according to claim 1 .

11. The partial restoration matrix setting unit sets the partial restoration matrix by deleting coefficients that do not correspond to the restoration region from coefficients of an entire restoration matrix that is a restoration matrix corresponding to the entire detected image. The image processing device according to claim 1 .

12. a restoration unit that restores a restored image of the restoration area from the detection image by using the partial restoration matrix set by the partial restoration matrix setting unit. The image processing device according to claim 1 .

13. A detection unit that detects at least one of information about the vehicle and information about the surroundings of the vehicle, The region specifying unit specifies the restoration region of the detected image by using the information detected by the detection unit. The image processing device according to claim 1 .

14. The region specifying unit specifies the restoration region in accordance with the speed of the vehicle detected by the detection unit. The image processing device according to claim 13.

15. The region specifying unit specifies, as the restoration region, a partial region including an obstacle entering the angle of view detected by the detection unit at a speed faster than a threshold or an obstacle entering in a predetermined direction. The image processing device according to claim 13.

16. The area specifying unit specifies the restored area depending on a destination to which the restored image is provided. The image processing device according to claim 1 .

17. The area specifying unit specifies the restoration area according to the authority of the user. The image processing device according to claim 1 .

18. a restoration area for restoring a restored image using a restoration matrix within an area of ​​a detected image obtained in an image sensor having a plurality of pixels that receive incident light that is incident without passing through either an imaging lens or a pinhole, the image sensor having a configuration for making output pixel values ​​of at least two of the plurality of pixels have mutually different incident angle directivities with respect to incident light from a subject; A partial restoration matrix used for restoring only the restored image of the restoration region from the detected image is set by deleting coefficients that do not correspond to the specified restoration region from coefficients of an entire restoration matrix, which is a restoration matrix corresponding to the entire detected image. Image processing methods.

19. Computer, an area specifying unit that specifies a restoration area for restoring a restored image using a restoration matrix within an area of ​​a detected image obtained in an image sensor having a plurality of pixels that receive incident light that is incident without passing through either an imaging lens or a pinhole, the image sensor having a configuration in which output pixel values ​​of at least two of the plurality of pixels have different incident angle directivities with respect to incident light from a subject; a partial restoration matrix setting unit that sets a partial restoration matrix used for restoring only a restored image of the restoration area from the detected image by deleting coefficients that do not correspond to the restoration area specified by the area specifying unit from coefficients of an entire restoration matrix that is a restoration matrix corresponding to the entire detected image. A program that functions as a

20. An imaging device and an image processing device are provided, The imaging device, An imaging element including a plurality of pixels that receive incident light that is incident without passing through either an imaging lens or a pinhole, the imaging element being configured so that output pixel values ​​of at least two of the plurality of pixels have different incident angle directivities with respect to the incident light from a subject. Equipped with The image processing device, a region specifying unit that specifies a restoration region for restoring a restored image by using a restoration matrix within a region of a detected image obtained by the imaging element; a partial restoration matrix setting unit that sets a partial restoration matrix used for restoring only a restored image of the restoration area from the detected image by deleting coefficients that do not correspond to the restoration area specified by the area specifying unit from coefficients of an entire restoration matrix that is a restoration matrix corresponding to the entire detected image; An image processing system comprising:

Images