Optical element and imaging device

Abstract

The present technology relates to an optical element and an imaging device capable of reducing the cost required to remove a light shield attached to an aperture. The optical element is arranged on an optical path from a subject to the imaging element, and has a plurality of minute concave or convex portions arranged on a surface thereof to have different light diffusion characteristics according to positions thereof on a cross-section of the optical path. The present technology can be applied to an optical element provided in an installed imaging device of an imaging system including, for example, an installed imaging device installed in an automobile, drone, telegraph pole, ship, etc. An image processing means that performs image processing for converting the captured image captured by the imaging means into a restored image that is an image of the subject; and a display device.

Description

Technical Field

[0001] This technology relates to optical elements and imaging devices, and more specifically, to optical elements and imaging devices capable of reducing the cost required to remove obstructions attached to the surface of an aperture. Background Technology

[0002] Traditionally, cameras are used near the user (the photographer), and even when obstructions such as dirt adhere to the lens set on the aperture surface, the user can usually easily remove them. Furthermore, even if an obstruction is attached to the lens, the light from the obstruction is scattered across the light-receiving surface of the imaging element, and therefore, the obstruction usually has no effect on the captured image. This phenomenon occurs when the obstruction is close to the aperture surface, and is similar to the phenomenon where, for example, an image of a landscape taken through a screen does not include the screen. Therefore, traditionally, obstructions on the lens are not considered.

[0003] Recently, there has been progress in camera miniaturization, cost reduction, and power consumption reduction, and mounted cameras (such as vehicle cameras, surveillance cameras, and live cameras) have become popular. These mounted cameras capture images of the surrounding environment for users to remotely monitor their surroundings, and there are usually no users nearby. Therefore, when obstructions are attached to the lens, it is difficult for users to remove the obstructions, and the cost of removing the obstructions is high.

[0004] However, when residual obstructions remain attached to the lens, these obstructions can easily accumulate to a degree that makes it difficult to ensure the quality of the captured image.

[0005] Therefore, a method for removing dirt from vehicle cameras using cleaning fluid has been invented (see, for example, PTL 1 and PTL 2).

[0006] However, this method requires significant costs to remove obstructions, including the power consumption of the cleaning fluid spray mechanism, the labor involved in filling the cleaning fluid, the installation of the cleaning fluid spray nozzles near the vehicle camera, and the assembly of various components that make up the spray mechanism.

[0007] [List of Citations]

[0008] [Patent Literature]

[0009] [PTL 1]

[0010] Japanese translation of PCT application No. 2022-547672

[0011] [PTL 2]

[0012] JP 2022-131226A Summary of the Invention

[0013] [Technical Issues]

[0014] Given the above, although there is a need to provide a method for reducing the cost of removing obstructions attached to the aperture surface, the current method does not adequately meet this need.

[0015] In view of this situation, this technology can reduce the cost required to remove obstructions attached to the aperture surface.

[0016] [Solution to the problem]

[0017] According to a first aspect of the present technology, the optical element is an optical element arranged in the optical path from the subject to the imaging element, and is configured to have a plurality of tiny recesses or protrusions arranged on the surface of the optical element to have different light diffusion characteristics depending on their positions in the cross section of the optical path.

[0018] According to a first aspect of the present technology, a plurality of tiny recesses or protrusions are arranged on the optical path from the subject to the imaging element, and different light diffusion characteristics are arranged on the surface, depending on their positions on the cross section of the optical path.

[0019] An imaging apparatus according to a second aspect of the present invention includes an optical element and an imaging element, the optical element being arranged in an optical path from a subject to the imaging element, and configured to have a plurality of minute recesses or protrusions on the surface of the optical element to have different light diffusion characteristics depending on their positions in the cross section of the optical path.

[0020] According to a second aspect of the present technology, an optical element and an imaging element are provided, the optical element being arranged in an optical path from the subject to the imaging element, and configured to have a plurality of tiny recesses or protrusions on the surface of the optical element to have different light diffusion characteristics depending on their positions in the cross section of the optical path. Attached Figure Description

[0021] Figure 1 This is a block diagram illustrating a configuration example of an imaging system according to a first embodiment of the present technology.

[0022] Figure 2 It shows Figure 1 A perspective view of a first configuration example of an imaging device.

[0023] Figure 3 This is a top view showing an example of the configuration of the protrusion.

[0024] Figure 4 This is a perspective view of an optical element showing an example of the shape of a protrusion.

[0025] Figure 5 This is a perspective view showing other examples of convex parts.

[0026] Figure 6 It is used for explanation Figure 1 A diagram illustrating the effect of the imaging system.

[0027] Figure 7 It is used for explanation Figure 1 The flowchart of image processing of the image processing device.

[0028] Figure 8 It is used for explanation Figure 7 The flowchart for the distribution generation process.

[0029] Figure 9 It is used for explanation Figure 7 The flowchart for the classification process.

[0030] Figure 10 This is a diagram used to illustrate the ray tracing model for the generation process.

[0031] Figure 11 It shows Figure 1 A perspective view of a second configuration example of the imaging device.

[0032] Figure 12 It shows Figure 1 A perspective view of a third configuration example of the imaging device.

[0033] Figure 13 This is a perspective view showing a first configuration example of an imaging apparatus according to a second embodiment of an imaging system applying the present technology.

[0034] Figure 14 It shows Figure 13 A perspective view of a second configuration example of the imaging device.

[0035] Figure 15 It shows Figure 13 A perspective view of a third configuration example of the imaging device.

[0036] Figure 16 This is a perspective view showing an example of the configuration of optical elements according to a third embodiment of an imaging system to which this technology has been applied.

[0037] Figure 17 This is a block diagram illustrating a configuration example of an imaging system according to a fourth embodiment of the present technology.

[0038] Figure 18 It is used for explanation Figure 17 The flowchart of image processing of the image processing device.

[0039] Figure 19 It is used for explanation Figure 18 The flowchart for calculating the occlusion rate.

[0040] Figure 20 It is used for explanation Figure 18 The flowchart for the prediction process.

[0041] Figure 21 This is a block diagram illustrating an example of a computer's hardware configuration.

[0042] Figure 22 It is a block diagram illustrating an example of a vehicle control system configuration.

[0043] Figure 23 This is an explanatory diagram showing an example of the mounting location of the imaging unit. Detailed Implementation

[0044] The modes for implementing this technology (hereinafter referred to as implementation methods) will be described below. Here, the descriptions will be given in the following order.

[0045] 1. First Embodiment (Imaging System with Anti-fouling Structure)

[0046] 2. Second Embodiment (Imaging System with Biological Rejection Structure)

[0047] 3. Third embodiment (imaging system with another biological rejection structure)

[0048] 4. Fourth Implementation (Imaging System for Predicting Occlusion Rate)

[0049] 5. Computer

[0050] 6. Examples of applications to moving bodies

[0051] In the accompanying drawings referenced in the following description, the same or similar reference numerals are used to denote the same or similar parts. However, the drawings are schematic, and the relationships between thicknesses and plan dimensions, the thickness proportions of each layer, etc., may differ from reality. Furthermore, the drawings may include, in some cases, portions whose dimensional relationships and scales differ from those in the accompanying drawings.

[0052] Furthermore, it should be understood that the definitions of directions such as up and down in the following description are provided merely for the sake of brevity and are not intended to limit the technical spirit of this disclosure. For example, when viewing an object after rotating it 90 degrees, up and down are converted to and interpreted as left and right, and when viewing an object after rotating it 180 degrees, up and down are interpreted as reversed.

[0053] <1. First Implementation Method>

[0054] <Imaging System Configuration>

[0055] Figure 1 This is a block diagram of a configuration example of an imaging system according to a first embodiment of the present technology.

[0056] Figure 1 The imaging system 10 includes an mounted imaging device 11, an image processing device 12, and a display device 13. The imaging system 10 captures images of the subject and outputs images of the subject, and displays information indicating the type of material attached to the imaging device 11, such as water, snow, mud, factory liquids, excrement, dirt, and burrowing organisms.

[0057] More specifically, the imaging device 11 is a mounted camera (mounted sensor) including an optical element unit 31 and an imaging element 32. The optical element unit 31 includes an optical element arranged as an coded aperture in the optical path from the subject to the imaging element 32. The surface of the optical element on the subject side (incident light side) has an anti-fouling structure to prevent (suppress) obstructions from adhering to the aperture surface. (Refer to the description below.) Figures 2 to 5 Describe the antifouling structure in detail.

[0058] Imaging element 32 includes a synchronous image sensor, an asynchronous vision sensor, and a sensor comprising both a synchronous image sensor and an asynchronous vision sensor, wherein multiple light-receiving elements for receiving visible or invisible light are aligned in one or two dimensions. Imaging element 32 receives light incident from the subject via optical element unit 31 and captures (acquires) a captured image corresponding to that light. Since the light incident on imaging element 32 is diffused light without a focusing element, the captured image is a non-imaging image. Imaging element 32 provides the captured image to image processing device 12.

[0059] The image processing device 12 includes a restoration unit 41, an image generation unit 42, a difference calculation unit 43, a distribution generation unit 44, a classification unit 45, and a storage unit 46.

[0060] The recovery unit 41 performs recovery processing to convert the captured image input from the imaging element 32 into an image of the subject. The principle of this recovery processing is similar to that of signal processing in a lensless camera, which simulates the imaging process (the inverse transformation of the shooting process) through signal processing without setting a lens.

[0061] More specifically, the captured image is a non-imaging image showing the shadow cast by the optical element unit 31 due to light from the subject. This shadow varies depending on the position of the point light source corresponding to the subject. For example, assuming the image of the subject is f and the shadow of the optical element unit 31 is a function of h, the captured image g can be represented by the following equation (1).

[0062] g = f*h ... (1)

[0063] Therefore, the restoration process is to obtain the image f of the subject from the captured image g in equation (1) by deconvolution as the restored image.

[0064] For example, the restoration process is performed using a restoration function, which is an image transformation function that converts a captured image into a restored image. The restoration function is an approximate function of a machine learning model designed to receive the captured image as input and output the restored image. Examples of such models include decoders of deep learning autoencoders, generative models of generative adversarial networks, transformers (specifically image transformers), diffusion models, sparse models, etc. Learning is performed statistically using learning data (including imaging images corresponding to multiple restored images of the subject and non-imaging images corresponding to the captured image), and the restoration parameters, which are parameters of the restoration function, are pre-optimized for the learning data.

[0065] Here, the recovery parameters vary depending on the structure of the optical element unit 31 (such as the arrangement and size of the anti-fouling structure), and therefore, the learning device for learning the recovery parameters needs to prepare learning data and learn the recovery parameters for each structure of the optical element unit 31. However, since the structure of the optical element unit 31 varies depending on each target shield, it is assumed that there are various types of structures for the optical element unit 31.

[0066] Therefore, the learning device may include a simulator that simulates the imaging process via optical element units 31 having each of various types of structures from an image of the subject, and generates a non-imaging image. In this case, by generating non-imaging images associated with optical element units 31 having each of various types of structures from an image of the subject, the learning device can efficiently generate learning data.

[0067] The learning device can set the recovery parameter associated with a predetermined structure of optical element unit 31 as the initial learning value of the recovery parameter associated with the structure of another optical element unit 31, and efficiently learn by adaptively learning (transfer learning) the recovery parameter. As a method for adaptively learning the recovery parameter, learning methods such as transfer learning, domain adaptation, one-sample learning, and meta-learning can be used.

[0068] The restoration parameters can be determined from a physical model, such as a ray tracing model. Alternatively, self-learning can be used to determine the restoration parameters, which feeds back the difference between the predicted captured image and the actual captured image from the ray tracing model (capture image simulator), described later, to the parameters.

[0069] The restoration parameters associated with the structure of the optical element unit 31 are stored in the storage unit 46. Therefore, the restoration unit 41 reads the restoration parameters stored in the storage unit 46 and performs restoration processing by using the restoration parameters to calculate a restoration function for the captured image.

[0070] The recovery unit 41 provides the recovered image converted by the recovery process to the image generation unit 42, and outputs the recovered image as the final captured image taken by the imaging device 11 to the outside of the imaging system 10.

[0071] Image generation unit 42 reads generation parameters stored in storage unit 46 as parameters of the generation function. The generation function is an image transformation function that converts the restored image into a predicted image of the captured subject when no obstruction is attached to optical element unit 31. More specifically, the generation function is an approximation function of a machine learning model designed to receive the restored image as input and output the predicted image. Although instances of this model are considered encoders of deep learning autoencoders, the model can be implemented as generative models of generative adversarial networks, transformers, diffusion models, sparse models, etc.

[0072] The generation parameters are pre-optimized using training data (including imaging images corresponding to multiple reconstructed images of the subject and non-imaging images corresponding to the predicted images) performed statistically. Similar to the reconstructed parameters, the generation parameters can be learned efficiently. The generation parameters can be determined from a physical model such as a ray tracing model. Alternatively, similar to the reconstructed parameters, the generation parameters can be determined using self-learning, which feeds back the difference between the predicted image and the actual predicted image predicted using the generation function to the parameters.

[0073] Image generation unit 42 performs a generation process to generate a predicted image from the restored image by calculating a generation function based on the generation parameters provided by restoration unit 41. Image generation unit 42 then provides the predicted image obtained as a result of the generation process to difference calculation unit 43.

[0074] The difference calculation unit 43 acquires the captured image input from the imaging element 32 as the actual image. Based on the actual image and the predicted image provided by the image generation unit 42, the difference calculation unit 43 calculates the difference between the pixel values ​​of the actual image and the predicted image, and generates a difference image including the difference between the pixel values.

[0075] Here, the predicted image is the captured image without any obstruction attached to the optical element unit 31, and the actual image is the captured image actually captured via the optical element unit 31 to which the obstruction may be attached. Therefore, the differential image can be estimated as the non-imaging image of the obstruction attached to the optical element unit 31. The differential calculation unit 43 provides the differential image to the distribution generation unit 44.

[0076] The distribution generation unit 44 reads the restoration parameters from the storage unit 46. Based on these restoration parameters, the distribution generation unit 44 calculates the restoration function of the difference image provided by the difference calculation unit 43, and performs a differential transformation process to convert the difference image into an occlusion distribution. The occlusion distribution is an estimate of the image of the occlusion on the optical element unit 31. The distribution generation unit 44 provides the occlusion distribution generated by the differential transformation process to the classification unit 45.

[0077] The classification unit 45 (output unit) reads the occlusion parameters for each occlusion category stored in the storage unit 46. The occlusion parameters are parameters designed, for example, to indicate the occlusion distribution function of a deep learning model used to indicate the occlusion distribution. The occlusion parameters are intended not to be used as occlusion parameters when the optical element unit 31 used to prevent occlusion attachment is itself an occlusion, but rather as occlusion parameters of the occlusion attached to the optical element unit 31. The occlusion parameters for each occlusion category are determined, for example, by performing a deep learning-based fitting by category using learning data (including the occlusion distributions of multiple occlusions and the categories of these occlusions).

[0078] The classification unit 45 calculates the occlusion distribution function using occlusion parameters for each occlusion category, generating an occlusion distribution model. Based on the occlusion distribution provided by the distribution generation unit 44, the classification unit 45 selects the category associated with the occlusion distribution model that most closely approximates the occlusion distribution among all categories as the occlusion category. The classification unit 45 outputs the category information to the display device 13 as occlusion information related to the occlusion on the optical element unit 31.

[0079] Storage unit 46 stores recovery parameters, generation parameters, and shielding parameters.

[0080] Display device 13 displays category information provided by classification unit 45. When the category indicated by the category information is a type of obstruction that is unlikely to disappear naturally, the user removes the obstruction by, for example, cleaning optical element unit 31. For example, when the category indicated by the category information is mud or excrement that is unlikely to disappear naturally, the user removes the obstruction. On the other hand, when the category indicated by the category information is snow that is more likely to disappear naturally due to melting, the user does not remove the obstruction. Note that at this time, based on the weather conditions at the installation location of imaging device 11, when the snow acting as an obstruction is unlikely to disappear naturally due to melting, the user can remove the obstruction.

[0081] As mentioned above, users can remove obscurations at the appropriate time based on category information, and therefore remove obscurations more efficiently compared to users removing obscurations regardless of their category. Consequently, the cost of removing obscurations can be reduced.

[0082] Note that the image processing device 12 may not output the category information to the display device 13, but instead output it to a subsequent occlusion removal device (not shown), which removes occlusions through cleaning or the like. In this case, the occlusion removal device determines whether to remove the occlusions based on the category information.

[0083] The image processing device 12 may include a recovery unit 41 and a storage unit 46, and only outputs the recovered image to the outside.

[0084] The image processing apparatus 12 includes a restoration unit 41, an image generation unit 42, a difference calculation unit 43, a distribution generation unit 44, and a storage unit 46. It can output a restored image and display the distribution of the obstructions to a display device 13. In this case, the user can determine the presence and quantity of the obstructions based on the distribution displayed on the display device 13. Therefore, the user can remove the obstructions at an appropriate time based on their presence or quantity. As a result, the cost of removing the obstructions can be reduced.

[0085] The pixel values ​​of pixels in captured, restored, predicted, and differencing images can be luminance values ​​(such as RGB or grayscale values) or values ​​corresponding to luminance values ​​(such as time differences between luminance values).

[0086] The processing of each unit of the image processing apparatus 12 can be performed on each captured image, or it can be performed on multiple captured images simultaneously as parallel processing or batch processing.

[0087] The classification unit 45 can output warning information to promote cleaning, replacement, etc., of the optical element unit 31 based on the type of obstruction, rather than using obstruction information as obstruction information. In this case, the classification unit 45 outputs a warning message, for example, when the obstruction is mud or excrement that is unlikely to disappear. On the other hand, the classification unit 45 does not output a warning message when the obstruction is snow that is more likely to disappear naturally due to melting. Note that this is based on the weather conditions at the installation location of the imaging device 11. When snow, as an obstruction, is unlikely to disappear naturally due to melting, the classification unit 45 can output a warning message.

[0088] The classification unit 45 can output removal information (such as adhesion strength and the best remover for the occupant) based on the occupant category instead of category information as occupant information. The classification unit 45 can output two or more of the following: category information, warning information, and removal information.

[0089] The recovery unit 41 can provide the recovered image to the display device 13 so that the display device 13 can display the recovered image.

[0090] <Imaging Device Configuration Example>

[0091] Figure 2 It shows Figure 1 A perspective view of a first configuration example of the imaging device 11.

[0092] like Figure 2 As shown, in the imaging device 11, the optical element unit 31 is installed at a predetermined distance away from the light receiving surface 32a of the imaging element 32, facing the subject side.

[0093] The optical element unit 31 is a three-dimensional structure with a plurality of tiny protrusions 62 on the subject-side surface 61a as an anti-fouling structure. By forming a plurality of protrusions 62 on the surface 61a of the optical element 61, a waterproof effect, namely the so-called lotus effect that prevents droplets from adhering to the surface 61a, can be obtained. As a result, the optical element unit 31 can prevent obstructions from adhering to the surface 61a.

[0094] The protrusion 62 is formed of a light-transmitting material to allow light received by the imaging element 32 to pass through. Therefore, the imaging element 32 is able to receive light from the subject and capture an image. The light received by the imaging element 32 includes visible light, infrared light, ultraviolet light, etc.

[0095] The optical element unit 31 and the imaging element 32 can be arranged apart from each other by a predetermined interval, or the optical element unit 31 can be attached to the imaging element 32 in such a way as bonding.

[0096] <Example of Convex Arrangement>

[0097] Figure 3 This is a top view showing an example of the arrangement of the protrusion 62.

[0098] Figure 3 The lower section of section A is a diagram showing the surface 61a of the optical element 61 as viewed from the subject side, and Figure 3 The upper part of A is Figure 3 An enlarged view of rectangle P in A.

[0099] like Figure 3 As shown in Figure A, the protrusions 62 are arranged non-periodically on the surface 61a of the optical element 61. That is, the arrangement pattern of the plurality of protrusions 62 is a pattern in which the positions of each protrusion 62 are not highly correlated (hereinafter referred to as a low autocorrelation pattern). In the first embodiment, the low autocorrelation pattern is a pattern of tiny random points arranged in two dimensions. When the low autocorrelation pattern is a random point pattern, the low autocorrelation pattern can be a pattern based on pseudo-random numbers generated by binary encoding (such as M-sequence encoding implemented by cascading M-level (M is an integer greater than 2) shift registers). In this case, the randomness of the low autocorrelation pattern is ensured by using a shift register with a long repetition period.

[0100] Note that a low autocorrelation pattern can be a point-symmetric sector pattern (e.g., a Newton's rings or spiral pattern), wherein a sector pattern with low correlation between the positions of points aligned in the radial direction is attached in a point-symmetric manner.

[0101] As described above, because the multiple protrusions 62 are arranged non-periodically—that is, the multiple protrusions 62 are arranged to have different light diffusion characteristics depending on their position on the cross-section of the optical path—the captured image has the characteristics of an image of the subject. Therefore, the robustness of the restoration process of the restoration unit 41 is improved, enabling the restoration unit 41 to generate the restored image more reliably.

[0102] On the contrary, such as Figure 3 As shown in B, when multiple protrusions 71 are periodically arranged on surface 61a, that is, when the arrangement pattern of the multiple protrusions 71 is a pattern in which the positions of the protrusions 71 are highly correlated, it may be difficult to perform restoration processing. For example, when the periodicity of the arrangement pattern of the multiple protrusions 71 coincides with the periodicity of the imaging image of the subject, the captured image does not have the characteristics of the imaging image of the subject, and it is difficult to perform restoration processing.

[0103] <Examples of convex shapes>

[0104] Figure 4 This is a perspective view of an optical element unit 31 showing an example of the shape of the protrusion 62.

[0105] exist Figure 4 In one example, the protrusion 62 of the optical element unit 31 is shaped as a roughly elliptical cylinder with a concave portion at the center. Figure 4 In one example, at least some of the protrusions 62 arranged on the optical element unit 31 have different shapes and sizes, but they can all be the same.

[0106] <Other examples of convex shapes>

[0107] Figure 5 This is a perspective view showing other shape examples of the protrusion 62.

[0108] exist Figure 5 In example A, the convex portion 62 is shaped like an approximately elliptical cylinder with a convex portion at the center. Figure 5 In the example of B, the convex part 62 is hemispherical in shape. Figure 5 In the example of C, the convex part 62 has a three-dimensional shape with identical rounded rectangles as its two bottom surfaces. Figure 5 In the example of D, the convex part 62 has a spiral shape. Figure 5 In the example of E, the shape of the protrusion 62 is formed by overlapping multiple spheres.

[0109] Note that the shape of the protrusion 62 is not limited to... Figure 4 and Figure 5 The shape of the optical element 61. The optical element 61 may be provided with a concave portion instead of a convex portion 62, or it may be provided with both a concave portion and a convex portion.

[0110] <Description of the imaging system's effects>

[0111] Figure 6 This is a diagram used to illustrate the effect of the imaging system 10.

[0112] As in Figure 6 As shown in A, based on the portable imaging device 82 carried by user 81, user 81 is present near the imaging device 82, such that when an obstruction is attached to the imaging device 82, user 81 can notice and remove the obstruction.

[0113] However, as Figure 6 As shown in Figure B, when the imaging device 11 is installed in a vehicle 91, drone 92, utility pole 93, ship 94, etc., the user is not present near the imaging device 11. Therefore, when an obstruction is attached to the imaging device 11, it is difficult for the user to notice and remove the obstruction.

[0114] Therefore, in the imaging device 11, the optical element 61 of the optical element unit 31 has a plurality of protrusions 62 on the surface 61a as an anti-fouling structure. Thus, it is possible to prevent obstructions from adhering to the surface 61a. In addition, the imaging device 11 displays category information on the display device 13, so that the user can notice the obstruction even when the user is not in the vicinity of the imaging device 11.

[0115] <Instructions for Image Processing>

[0116] Figure 7 It is used for explanation Figure 1 The flowchart of image processing by the image processing device 12 is as follows. For example, the image processing begins when the power to the image processing device 12 is turned on and the image processing device 12 is started.

[0117] exist Figure 7 In step S11, the recovery unit 41 and the distribution generation unit 44 of the image processing device 12 load (read) the recovery parameters stored in the storage unit 46. The image generation unit 42 loads the generation parameters stored in the storage unit 46. The classification unit 45 loads the occlusion parameters stored in the storage unit 46.

[0118] In step S12, the recovery unit 41 determines whether a captured image has been input from the imaging element 32. If it is determined in step S12 that no captured image has been input, the recovery unit 41 waits until a captured image is input. On the other hand, if it is determined in step S12 that a captured image has been input, the process proceeds to step S13.

[0119] In step S13, the recovery unit 41 performs recovery processing on the captured image input from the imaging element 32 based on the recovery parameters read in step S11, and generates a recovered image.

[0120] In step S14, the recovery unit 41 outputs the recovered image generated in step S13 to the image generation unit 42, and outputs the recovered image as the final captured image to the outside of the imaging system 10.

[0121] In step S15, the image processing apparatus 12 performs a distribution generation process to generate the distribution of occlusions. This will be described later. Figure 8 Describe the distribution generation process in detail.

[0122] Following the processing in step S15, in step S16, classification unit 45 performs a classification process to categorize the occlusions by type based on the occlusion distribution generated by the distribution generation process in step S15. (Refer to the description below.) Figure 9 Please describe this classification process in detail.

[0123] Following the processing in step S16, in step S17, the image processing device 12 determines whether to end the image processing, that is, determines, for example, whether the power supply to the image processing device 12 has been turned off. If it is determined in step S17 that the image processing should not be ended, that is, for example, if the power supply to the image processing device 12 has not been turned off, the processing then returns to step S12, and subsequent processing is repeated.

[0124] On the other hand, if it is determined in step S17 that the image processing is to end, that is, for example, if the power supply to the image processing device 12 has been turned off, the processing is ended.

[0125] Note that the image processing device 12 can save drive power by performing image processing only during a preset time period or only when an event such as capturing an image is detected.

[0126] <Explanation of Distribution Generation Processing>

[0127] Figure 8 It is used for explanation Figure 7 The flowchart for the distribution generation process in step S15.

[0128] exist Figure 8 In step S31, the image generation unit 42 acquires the image generated in the image. Figure 7 The restored image output in step S14. In step S32, the image generation unit 42 is based on... Figure 7 The generation parameters read in step S11 are used to perform generation processing on the restored image obtained in step S31, and a predicted image is generated. In step S33, the image generation unit 42 outputs the predicted image generated by the processing in step S32 to the difference calculation unit 43.

[0129] In step S34, the difference calculation unit 43 acquires the predicted image output in step S33. In step S35, the difference calculation unit 43 generates a difference image based on the predicted image acquired in step S34 and the actual image, which is the captured image input from the imaging element 32. In step S36, the difference calculation unit 43 outputs the difference image generated in step S35 to the distribution generation unit 44.

[0130] In step S37, the distribution generation unit 44 performs differential transformation processing on the differential image output in step S36 based on the recovery parameters read in step S11, and generates an occlusion distribution. The distribution generation unit 44 provides this occlusion distribution to the classification unit 45. Furthermore, the processing returns to... Figure 7 Step S15 is performed, and then proceeds to step S16.

[0131] <Instructions for Classification and Processing>

[0132] Figure 9 It is used for explanation Figure 7 The flowchart for the classification process in step S16.

[0133] exist Figure 9 In step S51, classification unit 45 obtains in Figure 8 The distribution of occlusions generated in step S37. In step S52, classification unit 45 is based on the distribution of occlusions generated in step S37. Figure 7 The occlusion parameters read in step S11 and the occlusion distribution obtained in step S51 are used to select the occlusion category. In step S53, the classification unit 45 outputs the category information of the category selected in step S52 to the display device 13. Thus, the display device 13 displays the category information. Furthermore, the processing returns to... Figure 7 Step S16 is performed, and then proceeds to step S17.

[0134] Note that the image generation unit 42 can perform the generation process by simulating the shooting process using a physical model such as a ray tracing model.

[0135] <Explanation of Ray Tracing Model>

[0136] Figure 10 This is a diagram illustrating the ray tracing model used to explain the generation process in this case.

[0137] like Figure 10 As shown, the ray tracing model for the generation process is a model in which the light is incident on the light-receiving surface 32a from the subject 111 corresponding to the restored image via the optical element unit 31 without any attached obstructions. In this ray tracing model, the subject 111 is sufficiently separated from the optical element unit 31, and the light reaching the optical element unit 31 from the subject 111 is parallel. The light reaching the light-receiving surface 32a from the optical element unit 31 is light that diffuses widely across the entire light-receiving surface 32a from each protrusion 62 arranged on the surface 61a, and is represented by a point spread function. Therefore, the generation process is a process of performing a convolution operation of the spread function on the restored image.

[0138] <Second Configuration Example of Imaging Device>

[0139] Figure 11 It shows Figure 1 A perspective view of a second configuration example of the imaging device 11.

[0140] exist Figure 11 In the imaging device 11, with Figure 2 The units corresponding to the units of the imaging device 11 are indicated by the same reference numerals. Therefore, the description of the corresponding units will be omitted as appropriate, and the description will focus on those related to the imaging device 11. Figure 2The imaging device 11 has different units. Figure 11 Imaging device 11 and Figure 2 The difference between the imaging device 11 and the other components is that it includes an optical element unit 131 instead of an optical element unit 31, and the other components are... Figure 2 The imaging device 11 is similarly configured.

[0141] The optical element unit 131 includes, sequentially from the subject side, an optical element 61 and a condenser element 141. In this optical element, multiple protrusions 62 are arranged aperiodically on a surface 61a. That is, in the optical element unit 131, the condenser element 141 is mounted between the optical element 61 and the imaging element 32 at a position corresponding to the light-receiving surface 32a of the imaging element 32 (position in the optical path). The condenser element 141 includes a convex lens, a concave mirror, etc. When the condenser element 141 includes a concave mirror, the positional relationship between the optical element unit 131, the imaging element 32, and the subject is... Figure 11 The positional relationship is different. In this case, when light is incident on the imaging element 32 via the protrusion 62, the protrusion 62 may not be transparent.

[0142] exist Figure 11 In the imaging apparatus 11, light incident on the imaging element 32 is focused by the light-concentrating element 141 and formed into an image on the light-receiving surface 32a. Therefore, the captured image is an imaging image. Thus, the distribution generation unit 44 may not perform differential conversion processing and can use the differential image as a masking distribution. The recovery unit 41 can output the captured image input from the imaging element 32 as the final captured image.

[0143] <Third Configuration Example of Imaging Device>

[0144] Figure 12 It shows Figure 1 A perspective view of a third configuration example of the imaging device 11.

[0145] exist Figure 12 In the imaging device 11, with Figure 2 The units corresponding to the units of the imaging device 11 are indicated by the same reference numerals. Therefore, the description of the corresponding units will be omitted as appropriate, and the description will focus on those related to the imaging device 11. Figure 2 The imaging device 11 has different units. Figure 12 Imaging device 11 and Figure 2 The difference between the imaging device 11 and the other components is that it includes an optical element unit 151 instead of an optical element unit 31. Figure 2 The imaging device 11 is similarly configured.

[0146] The optical element unit 151 includes a condenser element 161, wherein the optical element 61, comprising a plurality of protrusions 62 arranged non-periodically on the surface 61a, is formed on the surface on the subject side. The condenser element 161 includes a convex lens, a concave mirror, etc. When the condenser element 161 includes a concave mirror, the positional relationship between the optical element unit 131, the imaging element 32, and the subject is... Figure 12 The positional relationship is different. In this case, when light is incident on the imaging element 32 via the protrusion 62, the protrusion 62 may not be transparent.

[0147] exist Figure 12 In the imaging device 11, light incident on the imaging element 32 is focused by the light-concentrating element 161 and formed into an image on the light-receiving surface 32a. Therefore, the captured image is an imaging image. Thus, similar to... Figure 11 The imaging device 11, the distribution generation unit 44 can use the differential image as a masking distribution as is, or the recovery unit 41 can output the captured image input from the imaging element 32 as the final captured image as is.

[0148] As described above, the optical element units 31 (131 and 151) include an optical element 61 mounted closer to the subject than the imaging element 32, and a plurality of protrusions 62 are arranged on the surface 61a of the optical element 61. Therefore, it is possible to suppress the adhesion of obstructions to the surface 61a, and as a result, the cost required to remove obstructions is reduced.

[0149] Multiple protrusions 62 are arranged non-periodically on surface 61a, enabling recovery unit 41 to more reliably convert captured images into recovered images.

[0150] The recovery unit 41 recovers the recovered image from the captured image. Therefore, by outputting the recovered image as the final captured image taken by the imaging device 11, the recovery unit 41 is able to suppress the image quality degradation of the captured image caused by the optical element units 31 (131 and 151) and the obstruction.

[0151] Note that the recovery function and the generation function of the image processing device 12 are different depending on the optical element units 31 (131 and 151).

[0152] <2. Second Implementation Method>

[0153] <First Configuration Example of Imaging Device>

[0154] The configuration of the optical elements, as well as the recovery and generation functions, in the second embodiment of the imaging system applying this technology differ from those in the first embodiment, but other aspects are similar to those in the first embodiment. Therefore, the description will focus on the imaging apparatus including the optical elements, and descriptions of units other than the imaging apparatus will be omitted.

[0155] Figure 13 This is a perspective view showing a first configuration example of an imaging apparatus according to a second embodiment of an imaging system applying the present technology.

[0156] exist Figure 13 In the imaging device 210, with Figure 2 The units corresponding to the units of the imaging device 11 are indicated by the same reference numerals. Therefore, the description of the corresponding units will be omitted as appropriate, and the description will focus on the units that are different from the units of the imaging device 11. The imaging device 210 differs from the imaging device 11 in that it includes optical element unit 211 instead of optical element unit 31, and other components are arranged similarly to those of the imaging device 11. The imaging device 210 is installed in ships, etc., and prevents the attachment of attached organisms such as bacteria, barnacles, and algae in the shielding material.

[0157] The optical element unit 211 includes an optical element 221 arranged as an coded aperture in the optical path from the subject to the imaging element 32. The optical element 221 is formed of an opaque repellent material for sessile organisms and has a bio-repellent structure that prevents (inhibits) sessile organisms from attaching to the aperture surface.

[0158] The repellent material against sessile organisms is a sheet-like (plate-like) material that is likely to be disliked by the sessile organisms that are likely to attach to the optical element unit 211. It is a sheet coated with a synthetic metal such as copper or a copper alloy, a sheet such as a polymer, or a chemical substance such as an isonitrile compound that the sessile organisms dislike. If the repellent material against sessile organisms is not a sheet coated with a chemical substance, it is not necessary to recoat the chemical substance or replace the optical element 221 to maintain the repellency effect.

[0159] Multiple apertures 222 are arranged non-periodically in the optical element 221. That is, the arrangement pattern of the multiple apertures 222 is a low autocorrelation pattern. Therefore, the optical element 221 has light diffusion characteristics that vary depending on its position on the cross-section of the optical path. The imaging element 32 receives light from the subject through the multiple apertures 222. Since the light is diffracted by the multiple apertures 222, the recovery function and generation function according to the second embodiment are different from those in the first embodiment. Note that the optical element 221 may be provided with pinholes instead of apertures 222.

[0160] <Second Configuration Example of Imaging Device>

[0161] Figure 14 It shows Figure 13 A perspective view of a second configuration example of the imaging device 210 shown.

[0162] exist Figure 14 In the imaging device 210, with Figure 13 The units corresponding to the units of the imaging device 210 are indicated by the same reference numerals. Therefore, the description of the corresponding units will be omitted as appropriate, and the description will focus on those related to the imaging device 210. Figure 13 The imaging device 210 has different units. Figure 14 Imaging device 210 and Figure 13 The difference between the imaging device 210 and the other components is that it includes an optical element unit 241 instead of an optical element unit 211. Figure 13 The imaging device 210 is similarly configured.

[0163] The optical element unit 241 includes, from the subject side, a mesh-shaped optical element 251 and a light-collecting element 252 formed of a repellent material for attached organisms. The optical element unit 241 is mounted at a position corresponding to the light-receiving surface 32a, closer to the subject side than the imaging element 32. Therefore, the light-collecting element 252 is mounted between the imaging element 32 and the optical element 251. The light-collecting element 252 includes a convex lens, a concave mirror, etc.

[0164] exist Figure 14 In the imaging device 210, light from the subject passes through the grid 251a of the optical element 251 and enters the focusing element 252. The focusing element 252 concentrates the light and forms an image on the light-receiving surface 32a of the imaging element 32. Therefore, the captured image is an image. Thus, similar to... Figure 11 The imaging device 11, the distribution generation unit 44 can use the differential image as a masking distribution as is, or the recovery unit 41 can output the captured image input from the imaging element 32 as the final captured image as is.

[0165] <Third Configuration Example of Imaging Device>

[0166] Figure 15 It shows Figure 13 A perspective view of a third configuration example of the imaging device 210 shown.

[0167] exist Figure 15 In the imaging device 210, with Figure 13 The units corresponding to the units of the imaging device 210 are indicated by the same reference numerals. Therefore, the description of the corresponding units will be omitted as appropriate, and the description will focus on those related to the imaging device 210. Figure 13 The imaging device 210 has different units. Figure 15 Imaging device 210 and Figure 13 The difference between the imaging device 210 and the other components is that it includes an optical element unit 261 instead of an optical element unit 211. Figure 13 The imaging device 210 is similarly configured.

[0168] The optical element unit 261 includes a light-concentrating element 271, wherein the optical element 251, which includes a grid 251a, is formed on the surface of the subject side. The light-concentrating element 271 includes a convex lens, a concave mirror, etc.

[0169] exist Figure 15 In the imaging device 210, light from the subject passes through the grid 251a of the optical element 251 and enters the focusing element 271. The focusing element 271 concentrates the light and forms an image on the light-receiving surface 32a of the imaging element 32. Therefore, the captured image is an image. Thus, similar to... Figure 11 The imaging device 11, the distribution generation unit 44 can use the differential image as a masking distribution as is, or the recovery unit 41 can output the captured image input from the imaging element 32 as the final captured image as is.

[0170] Notice, Figure 13 The imaging device 210 may be equipped with an optical element 251 instead of an optical element 221. Figure 14 and Figure 15 The imaging device 210 may be equipped with an optical element 221 instead of an optical element 251.

[0171] As described above, the optical element unit 211 (241 and 261) includes an optical element 221 (251) which is mounted closer to the subject than the imaging element 32 and is formed of a repellent material against attached organisms. Therefore, it is possible to suppress the adhesion of obstructions to the optical element 221 (251), thereby reducing the cost required to remove these obstructions.

[0172] Multiple apertures 222 are arranged non-periodically in the optical element 221, which improves the robustness of the recovery process of the recovery unit 41 compared to the case where multiple apertures 222 are arranged periodically, and enables the captured image to be converted into a recovered image more reliably.

[0173] Furthermore, in the second embodiment, the recovery unit 41 recovers the recovered image from the captured image. Therefore, similar to the first embodiment, the recovery unit 41 is able to suppress the image quality degradation of the captured image caused by the optical element units 211 (241 and 261) and the obstruction.

[0174] <3. Third Implementation Method>

[0175] <Optical Component Configuration Examples>

[0176] The configuration of the optical elements, as well as the recovery and generation functions, in the third embodiment of the imaging system applying this technology differ from those in the first embodiment, but other aspects are similar to those in the first embodiment. Therefore, the description will focus on the optical elements, and descriptions of units other than the optical elements will be omitted. Similar to the second embodiment, in the third embodiment, the imaging device is installed in a ship or the like to prevent the attachment of sessile organisms in the shelter.

[0177] Figure 16 This is a perspective view showing an example of the configuration of optical elements according to a third embodiment of an imaging system applying the present technology.

[0178] Figure 16 The optical element unit 301 includes an optical element 311 arranged as an coded aperture in the optical path from the subject to the imaging element 32. The optical element 311 is formed of an opaque repellent material for sessile organisms. On the subject-side surface 311a of the optical element 311, as another biological repellent structure, a plurality of tiny grooves 312 are arranged with a predetermined directionality and periodicity that some organisms dislike. In the grooves 312, a plurality of fine holes 313 penetrating the optical element 311 are arranged non-periodically. Note that the optical element 311 may not be formed of a repellent material. The plurality of grooves 312 may be arranged to have at least one of a predetermined directionality and periodicity.

[0179] As described above, multiple grooves 312 are arranged in the optical element 311, which enables the adhesion of organisms to the surface 311a. As a result, the cost required to remove organisms can be reduced.

[0180] Multiple apertures 313 are arranged non-periodically in optical element 311, similar to optical element 221, enabling more reliable conversion of captured images into restored images.

[0181] Furthermore, in the third embodiment, the recovery unit 41 recovers the restored image from the captured image. Therefore, similar to the first embodiment, the recovery unit 41 is able to suppress image quality degradation of the captured image caused by the optical element unit 301 and the obstruction.

[0182] Note that in the second and third embodiments, the correlation between the sizes of the fine holes 222 (313) can be low.

[0183] <4. Fourth Implementation Method>

[0184] Figure 17 This is a block diagram illustrating a configuration example of an imaging system according to a fourth embodiment of the present technology.

[0185] exist Figure 17 In the imaging system 410, with Figure 1 The units corresponding to the units of imaging system 10 are indicated by the same reference numerals. Therefore, descriptions of the corresponding units will be omitted as appropriate, and the description will focus on units that are different from those of imaging system 10. Imaging system 410 differs from imaging system 10 in that it includes image processing device 412 and display device 413 instead of image processing device 12 and display device 13, and other components are configured similarly to those of imaging system 10. Imaging system 410 does not display category information, but displays prediction information as occlusion information, which indicates a predicted value of the occlusion rate.

[0186] More specifically, the image processing device 412 differs from the image processing device 12 in that it does not include the classification unit 45, but includes a calculation unit 445 and a prediction unit 447, and includes a storage unit 446 instead of the storage unit 46, and other components are configured similarly to the image processing device 12.

[0187] The calculation unit 445 calculates the proportion of the occupants to the light receiving surface 32a based on the occupant distribution generated by the distribution generation unit 44, as the current occupancy rate. The calculation unit 445 provides the current occupancy rate to the storage unit 446 so that the storage unit 446 stores the current occupancy rate.

[0188] Storage unit 446 stores recovery parameters and generation parameters. Storage unit 446 also stores the history of the current shading rate provided by computing unit 445.

[0189] The prediction unit 447 reads the history of the current occlusion rate from the storage unit 446. Based on this history, the prediction unit 447 performs an extrapolation operation to fit a linear function, exponential function, logarithmic function, etc., with parameters to the change of occlusion rate over time, and extrapolates the future occlusion rate. The prediction unit 447 (output unit) outputs the prediction information indicating the extrapolated future occlusion rate as the predicted value of the occlusion rate to the display device 413.

[0190] The display device 413 displays prediction information provided by the prediction unit 447. When the occlusion rate indicated by the prediction information exceeds a threshold, the user removes the occlusion. Therefore, the user can remove the occlusion before the quality of the captured image deteriorates due to the occlusion. Thus, the user can remove the occlusion at the appropriate time based on the prediction information, and therefore, remove the occlusion more efficiently compared to removing it at specific intervals. As a result, the cost required for occlusion removal can be reduced.

[0191] Note that the image processing device 412 may not output the prediction information to the display device 413, but instead output it to a subsequent occlusion removal device (not shown). In this case, the occlusion removal device determines whether to remove the occlusion based on the prediction information.

[0192] The image processing device 412 may not include the prediction unit 447, and may output occlusion rate information related to the current occlusion rate calculated by the calculation unit 445 (output unit) as occlusion information to the display device 413, so that the display device 413 displays the occlusion information. The occlusion rate information includes information indicating the current occlusion rate, warning information that promotes cleaning, replacement, etc. of the optical element unit 31 based on the current occlusion rate, etc.

[0193] The prediction unit 447 can output information related to the predicted occlusion rate, in addition to the prediction information, as occlusion information. For example, when the occlusion rate indicated by the prediction information exceeds a specified standard range, the prediction unit 447 can output a warning message urging the replacement of the optical element unit 31 as occlusion information. That is, when the occlusion rate indicated by the prediction information exceeds the specified standard range, the degradation of the anti-fouling function of the optical element unit 31 is predicted. Therefore, the prediction unit 447 outputs a warning message urging the user to replace the optical element unit 31. Thus, the user can replace the optical element unit 31 before the anti-fouling function of the optical element unit 31 deteriorates and the image quality of the captured image deteriorates. Note that when the occlusion rate indicated by the prediction information exceeds a threshold, the prediction unit 447 can output a warning message urging the removal of the occlusion. The prediction unit 447 can output both prediction information and warning information.

[0194] <Instructions for Image Processing>

[0195] Figure 18 It is used for explanation Figure 17 The flowchart of image processing by the image processing device 412 is as follows. For example, the image processing begins when the power to the image processing device 412 is turned on and the image processing device 412 is started.

[0196] exist Figure 18 In step S111, the image processing device 412's recovery unit 41 and distribution generation unit 44 load the recovery parameters stored in the storage unit 446. The image generation unit 42 loads the generation parameters stored in the storage unit 446.

[0197] The processing in steps S112 to S115 and Figure 7 The processing in steps S12 to S15 is the same.

[0198] Following the processing in step S115, in step S116, the calculation unit 445 performs an occlusion rate calculation process to calculate the current occlusion rate based on the occlusion distribution generated by the distribution generation process in step S115. (Refer to the description below.) Figure 19 The shading rate calculation process is described in detail.

[0199] Following the processing in step S116, in step S117, prediction unit 447 performs a prediction process to extrapolate the future occlusion rate. (Refer to a description to follow.) Figure 20 Describe the prediction process in detail.

[0200] After the processing in step S117, the process proceeds to step S118. Because the processing in step S118 is related to... Figure 7 The process in step S17 is the same, so its description will be omitted.

[0201] Note that the image processing device 412 can save drive power by performing image processing only during a preset time period or only when an event such as capturing an image is detected.

[0202] The occlusion rate calculation process in step S116 and the prediction process in step S117 can be performed periodically, rather than each time an input captured image is received.

[0203] <Explanation of Shielding Rate Calculation>

[0204] Figure 19 It is used for explanation Figure 18 The flowchart for the shading rate calculation process in step S116.

[0205] exist Figure 19 In step S131, the computing unit 445 obtains the data from the distribution generation unit 44 through... Figure 18 The occlusion distribution is generated by the distribution generation process in step S115. In step S132, the calculation unit 445 calculates the current occlusion rate based on the occlusion distribution obtained in step S131. In step S133, the calculation unit 445 provides the current occlusion rate calculated in step S132 to the storage unit 446 so that the storage unit 446 stores the current occlusion rate. Furthermore, the process returns to... Figure 18 The process proceeds to step S116 and then to step S117.

[0206] <Explanation of Predictive Processing>

[0207] Figure 20 It is used for explanation Figure 18 The flowchart of the prediction process in step S117.

[0208] exist Figure 20In step S151, the prediction unit 447 reads the history of the current occlusion rate from the storage unit 446. In step S152, the prediction unit 447 performs an extrapolation operation based on the history read in step S151 to extrapolate the future occlusion rate. In step S153, the prediction unit 447 outputs prediction information to the display device 413, which indicates the future occlusion rate extrapolated in step S153 as the predicted value of the occlusion rate. As a result, the display device 413 displays the prediction information. Furthermore, the processing returns to... Figure 18 The process in step S117 is continued, and the process in step S118 is continued.

[0209] Imaging system 410 may include optical element units 131, 151, 211, 241, 261, or 301 instead of optical element unit 31. Computation unit 445 may calculate the current occlusion rate based on a scalar such as the average brightness of the occluder rather than the occluder distribution. In this case, computational costs can be reduced compared to calculating costs based on the occluder distribution as a multidimensional quantity.

[0210] Imaging element 32 can be a sensor, such as a microwave sensor that detects other electromagnetic waves. Imaging system 10 (410) can output the captured image taken by imaging element 32 as is, rather than a restored image.

[0211] <5. Computer>

[0212] The series of processes performed by the image processing apparatus 12 (412) described above can be executed by hardware or by software. When the series of processes are executed by software, the software program is installed in a computer. Here, the computer includes, for example, a computer embedded in dedicated hardware or a general-purpose personal computer capable of performing various functions by installing various programs.

[0213] Figure 21 This is a block diagram illustrating an example of a computer hardware configuration that performs a series of processes by the image processing apparatus 12 (412) described above through a program.

[0214] In a computer, the central processing unit (CPU) 901, read-only memory (ROM) 902, and random access memory (RAM) 903 are connected to each other via a bus 904.

[0215] Input / output interface 905 is also connected to bus 904. Input unit 906, output unit 907, storage unit 908, communication unit 909 and driver 910 are connected to input / output interface 905.

[0216] The input unit 906 consists of a keyboard, mouse, microphone, etc. The output unit 907 consists of a speaker, display device 13 (413), etc. The storage unit 908 may be a hard disk, non-volatile memory, etc. The communication unit 909 may be a network interface, etc. The driver 910 drives a removable medium 911 such as a disk, optical disk, magneto-optical disk, or semiconductor memory.

[0217] In a computer configured as described above, for example, CPU 901 loads a program stored in storage unit 908 into RAM 903 via input / output interface 905 and bus 904, and runs the program to perform the series of processes described above.

[0218] For example, a program running by a computer (CPU 901) can be recorded on a removable medium 911 used as a packaging medium for provision. Alternatively, the program can be provided via wired or wireless transmission media such as a local area network, the Internet, or digital satellite broadcasting.

[0219] In a computer, by installing the removable medium 911 on the drive 910, a program can be installed in the storage unit 908 via the input / output interface 905. The program can be received by the communication unit 909 via a wired or wireless transmission medium for installation in the storage unit 908. Alternatively, the program can be pre-installed in the ROM 902 or the storage unit 908.

[0220] A program executed by a computer may be a program that performs processing sequentially in the order described in this specification, or it may be a program that performs processing in parallel or as needed, such as when a call is made.

[0221] A series of processes of the image processing device 12 (412) can be performed by the graphics processing unit (GPU).

[0222] <6. Examples of application to moving objects>

[0223] The technology disclosed herein (the Technology) can be applied to a variety of products. For example, the Technology disclosed herein can be implemented as a device equipped in any type of mobile body, such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility devices, airplanes, drones, ships, and robots.

[0224] Figure 22 This is a block diagram illustrating an example of a schematic configuration of a vehicle control system, which is an example of a mobile body control system to which the technology according to this disclosure can be applied.

[0225] The vehicle control system 12000 includes multiple electronic control units interconnected via a communication network 12001. Figure 22In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an external information detection unit 12030, an internal information detection unit 12040, and an integrated control unit 12050. Furthermore, a microcomputer 12051, an audio / image output unit 12052, and an in-vehicle network interface (I / F) 12053 are shown as functional configurations of the integrated control unit 12050.

[0226] The drive system control unit 12010 controls the operation of devices related to the vehicle's drive system according to various programs. For example, the drive system control unit 12010 serves as a control device, such as a drive force generating device (such as an internal combustion engine or drive motor) for generating the vehicle's driving force, a drive force transmission mechanism for transmitting the driving force to the wheels, a steering mechanism for adjusting the vehicle's steering angle, and a braking device for generating the vehicle's braking force.

[0227] The body system control unit 12020 controls the operation of various devices installed in the vehicle body according to various programs. For example, the body system control unit 12020 is used to control devices such as keyless entry systems, smart key systems, power windows, or various lights such as headlights, reversing lights, brake lights, turn signals, fog lights, etc. In this case, radio waves emitted from the moving device that replaces the key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives the input of radio waves or signals and controls the vehicle's door locking devices, power windows, lights, etc.

[0228] The exterior information detection unit 12030 detects external information of the vehicle on which the vehicle control system 12000 is installed. For example, the exterior information detection unit 12030 is connected to the imaging unit 12031. The exterior information detection unit 12030 causes the imaging unit 12031 to capture images of the exterior of the vehicle and receives the captured images. Based on the received images, the exterior information detection unit 12030 can perform object detection processing or distance detection processing for people, vehicles, obstacles, signs, and symbols on the road surface.

[0229] Imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. Imaging unit 12031 can output the electrical signal as an image or as distance measurement information. Furthermore, the light received by imaging unit 12031 can be visible light or invisible light such as infrared light.

[0230] The in-vehicle information detection unit 12040 detects information about the interior of the vehicle. For example, the in-vehicle information detection unit 12040 is connected to a driver state detection unit 12041 that detects the driver's state. The driver state detection unit 12041 includes, for example, a camera that captures images of the driver, and based on the detection information input from the driver state detection unit 12041, the in-vehicle information detection unit 12040 can calculate the driver's level of fatigue or concentration, or determine whether the driver is dozing off.

[0231] The microcomputer 12051 can calculate control target values ​​for the drive force generation device, steering mechanism, or braking device based on information about the vehicle's interior or exterior obtained by the external information detection unit 12030 or the internal information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform coordinated control aimed at realizing functions of an advanced driver assistance system (ADAS), including collision avoidance or impact mitigation, distance-based following, speed maintenance, collision warning, and lane departure warning.

[0232] Furthermore, the microcomputer 12051 can control the drive force generation device, steering mechanism, braking device, etc., based on information about the vehicle's surroundings obtained by the external information detection unit 12030 or the internal information detection unit 12040, and perform cooperative control intended for use in autonomous driving, etc., in which autonomous driving is performed without relying on the driver's operation.

[0233] Furthermore, the microcomputer 12051 can output control commands to the body system control unit 12020 based on information about the vehicle's exterior obtained by the exterior information detection unit 12030. For example, the microcomputer 12051 can control the headlights according to the position of the vehicle in front or oncoming vehicles detected by the exterior information detection unit 12030, and perform coordinated control aimed at preventing glare, such as switching from high beam to low beam.

[0234] The sound / image output unit 12052 transmits at least one of sound and image output signals to an output device capable of visually or audibly notifying passengers or the outside of the vehicle. Figure 22 In this example, the audio speaker 12061, the display unit 12062, and the instrument panel 12063 are shown as output devices. The display unit 12062 may include, for example, at least one of an in-vehicle display and a head-up display.

[0235] Figure 23 This is a diagram showing an example of the mounting location of the imaging unit 12031.

[0236] exist Figure 23 In the vehicle 12100, imaging units 12101, 12102, 12103, 12104 and 12105 are used as imaging unit 12031.

[0237] For example, imaging units 12101, 12102, 12103, 12104, and 12105 are positioned at locations such as the front nose, side mirrors, rear bumper, rear door, and upper part of the windshield inside the vehicle 12100. Imaging unit 12101 at the front nose and imaging unit 12105 at the upper part of the windshield inside the vehicle primarily capture images of the front of the vehicle 12100. Imaging units 12102 and 12103 at the side mirrors primarily capture images of the sides of the vehicle 12100. Imaging unit 12104 at the rear bumper or rear door primarily captures images of the rear of the vehicle 12100. The frontal images captured by imaging units 12101 and 12105 are mainly used to detect vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0238] Figure 23 Examples of the imaging ranges of imaging units 12101 to 12104 are also shown. Imaging range 12111 indicates the imaging range of imaging unit 12101 located in the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of imaging units 12102 and 12103 located in the side mirrors, respectively, and imaging range 12114 indicates the imaging range of imaging unit 12104 located in the rear bumper or rear door. For example, by overlaying the image data captured by imaging units 12101 to 12104, a bird's-eye view of the vehicle 12100 viewed from above can be obtained.

[0239] At least one of the imaging units 12101 to 12104 may have the function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera including a plurality of imaging elements, or may be an imaging element including pixels for phase difference detection.

[0240] For example, microcomputer 12051 can obtain the distance to each three-dimensional object within the imaging range 12111 to 12114 and the time change of that distance (relative speed to vehicle 12100) based on distance information obtained from imaging units 12101 to 12104, and extract the nearest three-dimensional object as the vehicle ahead. This nearest three-dimensional object is specifically located on the driving path of vehicle 12100 and is traveling in substantially the same direction as vehicle 12100 at a predetermined speed (e.g., above 0 km / h). Furthermore, microcomputer 12051 can pre-set the required vehicle-to-vehicle distance and can perform automatic braking control (including follow-stop control) and automatic acceleration control (including follow-start control). Therefore, cooperative control intended for, for example, autonomous driving can be performed, where autonomous driving is performed without relying on driver operation.

[0241] For example, microcomputer 12051 can classify and extract three-dimensional object data about three-dimensional objects into two-wheeled vehicles, standard-sized vehicles, large vehicles, pedestrians, and other three-dimensional objects such as utility poles, based on distance information obtained from imaging units 12101 to 12104, and can use the three-dimensional object data to perform automatic obstacle avoidance. For example, microcomputer 12051 can distinguish whether obstacles around vehicle 12100 are observable to the driver of vehicle 12100 or difficult to observe. In addition, microcomputer 12051 determines the collision risk, which indicates the degree of risk of collision with each obstacle, and when the collision risk is equal to or higher than a set value and there is a possibility of collision, outputs a warning to the driver via audio speaker 12061 or display unit 12062, or performs forced deceleration or evasive steering via drive system control unit 12010, enabling collision avoidance driving assistance.

[0242] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared light. For example, the microcomputer 12051 can identify a pedestrian by determining whether a pedestrian is present in the captured images of the imaging units 12101 to 12104. This pedestrian identification is performed, for example, by a procedure that extracts feature points from the captured images of the imaging units 12101 to 12104, which are infrared cameras, and a procedure that performs pattern matching processing on a series of feature points indicating the outline of an object and determines whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging units 12101 to 12104 and identifies the pedestrian, the sound / image output unit 12052 controls the display unit 12062 to display a square outline superimposed on the identified pedestrian for emphasis. Furthermore, the sound / image output unit 12052 also controls the display unit 12062 to display an icon or similar indicating the pedestrian at a desired location.

[0243] Examples of vehicle control systems to which the technology according to this disclosure can be applied have been described above. The technology according to this disclosure can be applied to the imaging unit 12031, the external information detection unit 12030, etc., in the aforementioned components. More specifically, for example, the imaging device 11 (210) and the imaging device according to the third embodiment including the optical element unit 301 can be applied to the imaging unit 12031. The image processing device 12 (412) can be applied to the external information detection unit 12030. By applying the technology of this disclosure to the imaging unit 12031, the cost required to remove obstructions attached to the imaging unit 12031 can be reduced. By applying the technology according to this disclosure to the external information detection unit 12030, image quality degradation of the captured image caused by the optical element units 31 (131, 151, 211, 241, 261, and 301) and obstructions can be suppressed. As a result, driver safety and comfort can be improved.

[0244] Furthermore, as used herein, a system refers to a collection of multiple constituent elements (such as devices, modules (components), etc.), and all constituent elements may be located in the same housing or not in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, both constitute a system.

[0245] The implementation of this technology is not limited to the above-described implementation, and various changes can be made without departing from the spirit of this technology.

[0246] For example, aspects obtained by combining all or some of the above-described embodiments can be employed. For example, optical element 61 can be formed of a repellent material for attached organisms to also have a bio-repellent structure. Optical element 221 (311) can also have an anti-fouling structure by arranging a plurality of tiny protrusions on the subject-side surface (surface 311a). The protrusions may or may not be translucent, or may or may not be arranged non-periodically. The occlusion information can be a combination of two or more of category information, warning information, removal information, prediction information, and occlusion rate information. This technology can also be applied to microscope objectives, eyepieces, etc. In this case, the anti-fouling structure (bio-repellent structure) of optical element 61 (301) may not be provided on the surface of the incident light side, but on the surface of the emitted light side, or on both the incident light side and the emitted light side. In the optical element unit 131 (151, 241, and 261), the optical element 61 (251) may not be disposed on the incident light side (surface) of the light-concentrating element 141 (161, 252, and 271), but may be disposed on the emitted light side (surface), or may be disposed on both the incident light side (surface) and the emitted light side (surface).

[0247] For example, this technology can be configured as cloud computing, where individual functions are processed collaboratively in a distributed manner via a network.

[0248] Furthermore, each step described in the flowchart discussed above can be performed by a single device or by multiple devices in a distributed manner.

[0249] Furthermore, when a single step includes multiple types of processing, the multiple types of processing included in the single step can be performed by a single device or by multiple devices in a distributed manner.

[0250] Note that the beneficial effects described in this specification are exemplary and not limiting, and may produce other beneficial effects besides those described in this specification.

[0251] This technology can be configured as follows. (1)

[0253] An optical element is arranged in an optical path from a subject to an imaging element, and is configured to have a plurality of tiny recesses or protrusions on the surface of the optical element to have different light diffusion characteristics depending on their positions in the cross section of the optical path. (2)

[0255] The optical element described in (1) further includes a focusing element, which is mounted in the optical path between the imaging element and the optical element. (3)

[0257] The optical element described in (2) above is formed on the surface of the incident light side of the focusing element. (4)

[0259] The optical element of any one of (1) to (3) above, wherein the optical element is configured to be formed of a repellent material for sessile organisms. (5)

[0261] An imaging device, comprising:

[0262] An optical element is arranged in the optical path from the subject to the imaging element, and is configured to have multiple minute recesses or protrusions on its surface to have different light diffusion characteristics depending on their positions in the cross-section of the optical path; and

[0263] The imaging element. (6)

[0265] The imaging device described in (5) further includes a recovery unit that converts the captured image taken by the imaging element into an image of the subject.

[0266] [List of reference numerals]

[0267] 10 Imaging system, 11 Imaging element, 12 Image processing device, 31 Optical element unit, 32 Imaging element, 32a Light receiving surface, 41 Recovery unit, 44 Distribution generation unit, 45 Output unit, 61 Optical element, 61a Surface, 62 Protrusion, 131 Optical element unit, 141 Condensing element, 151 Optical element unit, 161 Condensing element, 210 Imaging device, 211 Optical element unit, 221 Optical element, 222 Micro-aperture, 241 Optical element unit, 251 Optical element, 252 Condensing element, 261 Optical element, 271 Condensing element, 301 Optical element unit, 311 Optical element, 312 Groove, 313 Micro-aperture, 410 Imaging system, 412 Image processing device, 445 Calculation unit, 447 Prediction unit.

Claims

1. An optical element arranged in an optical path from a subject to an imaging element, and configured to have a plurality of minute recesses or protrusions arranged on the surface of the optical element to have different light diffusion characteristics depending on their positions in the cross section of the optical path.

2. The optical element according to claim 1, further comprising a focusing element, the focusing element being mounted in the optical path between the imaging element and the optical element.

3. The optical element according to claim 2, wherein, The optical element is formed on the surface of the incident light side of the focusing element.

4. The optical element according to claim 1, wherein, The optical element is configured to be formed of a repellent material for sessile organisms.

5. An imaging device, comprising: An optical element is arranged in the optical path from the subject to the imaging element, and is configured to have a plurality of minute recesses or protrusions on its surface to have different light diffusion characteristics depending on their positions in the cross-section of the optical path; and The imaging element.

6. The imaging apparatus according to claim 5, further comprising a recovery unit that converts a captured image taken by the imaging element into an image of the subject.

Images