Optical device and imaging device

Abstract

The present technology relates to an optical device and an imaging device that make it possible to amplify the surface characteristics of a captured image while suppressing any deterioration in the image quality of the captured image in an imaging device using an optical element having no image-forming function. An optical element unit according to the present technology comprises a surface part having a fine structure for amplifying surface characteristics on an optical path leading from a subject to an imaging element, an optical buffer part having a prescribed thickness, and an optical element in which a plurality of fine holes are arranged so as to spatially modulate the light quantity of incident light to a different light quantity according to the position on a cross-section of the optical path. The present technology can be applied to an optical element provided to an installation-type imaging device of an imaging system configured from, for example, an installation-type imaging device installed in a car, drone, utility pole, ship, or the like, an image processing device that performs image processing for converting captured images captured by the imaging device to restored images that are images of subjects, and a display device.

Description

Optical devices and imaging devices

[0001] The present technology relates to an optical device and an imaging device, and in particular to an optical device and an imaging device that are capable of amplifying surface characteristics while suppressing degradation in image quality of an imaged image in an imaging device that uses an optical element that does not have an imaging function.

[0002] Recently, the prices of image sensors have been falling. As a result, a new market has been developed in which cameras equipped with image sensors are placed at multiple locations to continuously capture the surrounding environment, thereby utilizing them in monitoring systems. In the future, it is expected that such monitoring systems will be able to reduce costs by using inexpensive optical elements without imaging functions instead of expensive lenses as the optical elements of the cameras.

[0003] On the other hand, such monitoring systems can be problematic in terms of increased maintenance costs, such as the cost required to remove obstructions. Specifically, conventional expensive cameras are used in close proximity to the user (photographer). Therefore, even if obstructions such as dirt adhere to the lens on the aperture, the user can often easily remove the obstruction, making the cost required to remove the obstruction negligible.

[0004] However, the cameras installed at multiple locations in a monitoring system are used by users to remotely monitor the surrounding environment, and the users are usually not in the vicinity of the cameras. Therefore, if an object gets stuck on the lens, it is difficult for the user to remove the object. Furthermore, since there are many cameras, the cost required to remove the object can be significant.

[0005] As a method for removing an object blocking the camera, for example, a method of removing dirt from the vehicle-mounted camera using a cleaning solution has been devised (see, for example, Patent Documents 1 and 2).

[0006] However, such a method requires significant costs to remove the obstruction, such as the power consumption of the cleaning fluid spray mechanism, the effort required to fill the cleaning fluid, the installation of a cleaning fluid spray nozzle near the vehicle-mounted camera, and the installation of various parts that make up the spray mechanism.

[0007] JP 2022-547672 A JP 2022-131226 A

[0008] For the above reasons, there is a demand for a method of amplifying surface characteristics that have the effect of suppressing adhesion of obstructing objects while suppressing deterioration of image quality in imaging devices such as cameras that use optical elements without an imaging function, but such demands have not yet been adequately met.

[0009] This technology was developed in consideration of such circumstances, and enables an imaging device that uses optical elements without an imaging function to amplify surface characteristics while suppressing degradation in the image quality of the captured image.

[0010] An optical device according to a first aspect of the present technology is an optical device that includes, on an optical path from a subject to an imaging element, a surface portion having a microstructure that amplifies surface characteristics, an optical buffer portion having a predetermined thickness, and an optical element in which a plurality of micropores are arranged so as to spatially modulate the amount of incident light to different light amounts depending on the position on the cross section of the optical path.

[0011] In a first aspect of the present technology, a surface portion having a microstructure that amplifies surface characteristics, an optical buffer portion having a predetermined thickness, and an optical element in which a plurality of micropores are arranged so as to spatially modulate the amount of incident light to different light amounts depending on the position on the cross section of the optical path are provided on the optical path from the subject to the image sensor.

[0012] An imaging device according to a second aspect of the present technology is an imaging device that includes, on an optical path from a subject to an imaging element, an optical element unit that includes a surface portion having a microstructure that amplifies surface characteristics, an optical buffer portion having a predetermined thickness, and an optical element in which a plurality of micropores are arranged so as to spatially modulate the amount of incident light to different light amounts depending on the position on a cross section of the optical path, and the imaging element that captures an optical image of the light incident through the optical element unit.

[0013] In a second aspect of the present technology, an optical element unit is provided on an optical path from a subject to an image sensor, the optical element unit including a surface portion having a microstructure that amplifies surface characteristics, an optical buffer portion having a predetermined thickness, and an optical element in which a plurality of micropores are arranged so as to spatially modulate the amount of incident light to different light amounts depending on the position on a cross section of the optical path, and the image sensor that captures an optical image of the light incident through the optical element unit.

[0014] 7 is a block diagram showing an example configuration of a first embodiment of an imaging system to which the present technology is applied. FIG. 8 is a cross-sectional view showing an example configuration of an imaging device. FIG. 9 is a cross-sectional view of a surface portion showing an example cross-sectional shape of the microstructure of FIG. 2. FIG. 10 is a perspective view showing an example shape of the microstructure of FIG. 2. FIG. 11 is a top view of a surface portion showing an example arrangement of the microstructure of FIG. 2. FIG. 12 is a diagram illustrating a method for determining the thickness of a buffer portion. FIG. 13 is a flowchart illustrating image processing. FIG. 14 is a flowchart illustrating the distribution generation process of FIG. 7. FIG. 15 is a flowchart illustrating the classification process of FIG. 7. FIG. 16 is a cross-sectional view showing an example configuration of a surface buffer portion. FIG. 17 is a block diagram showing an example configuration of computer hardware. FIG. 18 is a block diagram showing an example of the schematic configuration of a vehicle control system. FIG. 19 is an explanatory diagram showing an example installation position of an imaging unit.

[0015] Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. First embodiment (imaging system in which a surface unit and a buffer unit are provided separately) 2. Second embodiment (imaging system in which a surface unit and a buffer unit are provided integrally) 3. Computer 4. Application example to a moving body

[0016] In the drawings referred to in the following description, the same or similar parts are denoted by the same or similar reference numerals. However, the drawings are schematic, and the relationship between thickness and planar dimensions, the thickness ratio of each layer, etc. may differ from the actual ones. Furthermore, the drawings may include parts whose dimensional relationships and ratios differ from each other.

[0017] Furthermore, the definitions of directions such as up and down in the following description are merely for the convenience of explanation and do not limit the technical idea of ​​the present disclosure. For example, if an object is rotated 90 degrees and observed, up and down are converted to left and right and read, and if it is rotated 180 degrees and observed, up and down are read inverted.

[0018] 1. First Embodiment Configuration Example of Imaging System FIG. 1 is a block diagram showing a configuration example of a first embodiment of an imaging system to which the present technology is applied.

[0019] 1 is composed of a stationary imaging device 11, an image processing device 12, and a display device 13. The imaging system 10 captures an image of a subject and outputs the captured image, and also displays category information indicating the category of obstructing objects attached to the imaging device 11, such as water, snow, mud, industrial liquids, oil, manure, dust, attached organisms, and deposits (e.g., salt, sugar, and oxides).

[0020] Specifically, the imaging device 11 is a stationary camera (stationary sensor) that includes an optical element section 31 and an imaging element section 32. The optical element section 31 (optical device) is disposed on an optical path from a subject to the imaging element section 32. The optical element section 31 does not include a light collecting element, and spatially modulates the amount of light, such as visible light, infrared light, or ultraviolet light, incident from the subject into different amounts of light depending on the position on the cross section of the optical path, and irradiates the imaging element section 32 with the modulated light.

[0021] The imaging element unit 32 includes an imaging element, etc. This imaging element is configured by a synchronous image sensor in which a plurality of light receiving elements that receive visible light or invisible light are arranged one-dimensionally or two-dimensionally, an asynchronous vision sensor, a sensor including both a synchronous image sensor and an asynchronous vision sensor, etc. The imaging element unit 32 receives light that is incident from the subject and spatially modulated via the optical element unit 31, and captures (acquires) a non-imaging image that is a non-imaging optical image of that light as a captured image. The imaging element unit 32 supplies the captured image to the image processing device 12.

[0022] The image processing device 12 includes a restoration unit 41, an image generation unit 42, a difference calculation unit 43, a distribution generation unit 44, a classification unit 45, and a storage unit 46.

[0023] The restoration unit 41 performs restoration processing to convert the captured image input from the image sensor unit 32 into a restored image, which is an image of the subject. The principle of this restoration processing is similar to the principle of signal processing in a lensless camera, which simulates the image formation process (inverse transformation of the image capture process) by signal processing without using a lens.

[0024] Specifically, the captured image is a non-imaged image that represents the shadow of the optical element unit 31 cast by light from the subject. This shadow differs depending on the position of the point light source corresponding to the subject. For example, if the imaged image of the subject is f and the function of the shadow of the optical element unit 31 is h, then the captured image g can be expressed by the convolution equation of the following formula (a):

[0025] g=f*h...(a)

[0026] Therefore, the restoration process is a process of determining the imaged image f of the subject as a restored image from the captured image g in equation (a) by deconvolution.

[0027] This restoration process is performed using, for example, a restoration function, which is an image transformation function that transforms a captured image into a restored image. The restoration function is an approximation function designed as a machine learning model that takes a captured image as input and a restored image as output. Examples of this model include a deep learning autoencoder (denoising autoencoder), a generative model of an adversarial learning network, a transformer (particularly an image transformer), a diffusion model, and a sparse model. The restoration parameters, which are parameters of the restoration function, are optimized in advance for the training data by statistically learning using training data consisting of focused images corresponding to restored images of a large number of subjects and non-focused images corresponding to the captured images.

[0028] Here, the restoration parameters differ depending on the structure of the optical element unit 31, and therefore the learning device that learns the restoration parameters needs to prepare learning data and learn the restoration parameters for each structure of the optical element unit 31. However, since the structure of the optical element unit 31 differs depending on the target occluding object, it is expected that many different types of structures exist.

[0029] Therefore, the learning device may be provided with a simulator that generates non-imaged images by simulating the imaging process from an imaged image of a subject via optical element units 31 of various structures. In this case, the learning device can generate non-imaged images corresponding to optical element units 31 of various structures from an imaged image of a single subject, thereby making it possible to generate learning data more efficiently.

[0030] The learning device may improve the efficiency of learning by setting restoration parameters corresponding to the structure of a predetermined optical element unit 31 as initial values ​​for learning restoration parameters corresponding to the structure of another optical element unit 31, and adaptively learning (transfer learning) the restoration parameters. As a method for adaptively learning restoration parameters, learning methods known as transfer learning, domain adaptation, one-shot learning, meta-learning, etc. can be used.

[0031] The restoration parameters may be determined from a physical model such as a ray tracing model, or may be determined by utilizing self-learning, which uses a ray tracing model (a simulator of a captured image) to feed back to the parameters a difference between a captured image predicted from a restored image and an actual captured image.

[0032] The restoration parameters corresponding to the structure of the optical element unit 31 are stored in the storage unit 46. Therefore, the restoration unit 41 reads out the restoration parameters stored in the storage unit 46 and performs restoration processing by calculating a restoration function on the captured image using the restoration parameters.

[0033] The restoration unit 41 supplies the restored image converted by the restoration process to the image generation unit 42 and outputs it to the outside of the imaging system 10 as the final captured image captured by the imaging device 11 .

[0034] The image generation unit 42 reads out generation parameters, which are parameters of a generation function stored in the storage unit 46. The generation function is an image transformation function that converts a restored image into a predicted image of an image of the subject captured when no occluding object is attached to the optical element unit 31. Specifically, the generation function is an approximation function designed as a machine learning model that inputs a restored image and outputs a predicted image. This model may be, for example, an encoder of an autoencoder in deep learning, but this model can also be realized using a generative model of an adversarial learning network, a transformer, a diffusion model, a sparse model, or the like.

[0035] The generation parameters are optimized in advance for the training data by statistically learning using training data consisting of imaged images corresponding to restored images of a large number of subjects and non-imaged images corresponding to predicted images. The generation parameters may be learned efficiently in the same way as the restoration parameters. The generation parameters may be determined from a physical model such as a ray tracing model. Alternatively, the generation parameters may be determined, similar to the restoration parameters, by utilizing self-teaching learning in which the difference between a predicted image predicted using a generation function and an actual predicted image is fed back to the parameters.

[0036] The image generation unit 42 performs a generation process to generate a predicted image from the restored image by calculating a generation function based on the generation parameters on the restored image supplied from the restoration unit 41. The image generation unit 42 supplies the resulting predicted image to the difference calculation unit 43.

[0037] The difference calculation unit 43 acquires, as a real image, the captured image input from the imaging element unit 32. Based on the real image and the predicted image supplied from the image generation unit 42, the difference calculation unit 43 calculates the difference between each pixel value of the real image and the predicted image, and generates a difference image made up of the differences between each pixel value.

[0038] Here, the predicted image is an image captured when no obstruction is attached to the optical element unit 31, and the actual image is an image actually captured through the optical element unit 31 to which an obstruction may be attached. Therefore, it can be assumed that the difference image is a non-imaged image of an obstruction attached to the optical element unit 31. The difference calculation unit 43 supplies the difference image to the distribution generation unit 44.

[0039] The distribution generation unit 44 reads out the restoration parameters from the storage unit 46. Based on the restoration parameters, the distribution generation unit 44 performs a difference conversion process to convert the difference image supplied from the difference calculation unit 43 into an obstruction distribution by calculating a restoration function. The obstruction distribution is an estimated value of the image of the obstruction on the optical element unit 31. The distribution generation unit 44 supplies the obstruction distribution generated by the difference conversion process to the classification unit 45.

[0040] The classification unit 45 (output unit) reads out the obstructing object parameters for each obstructing object category stored in the storage unit 46. The obstructing object parameters are parameters of an obstructing object distribution function designed as a deep learning model or the like representing the obstructing object distribution. These obstructing object parameters are intended to be obstructing object parameters of an obstructing object attached to the optical element unit 31, rather than obstructing object parameters when the optical element unit 31 itself, which is intended to prevent the attachment of obstructing objects, is used as the obstructing object. The obstructing object parameters for each obstructing object category are determined, for example, by performing deep learning fitting for each category using learning data consisting of the obstructing object distributions of a large number of obstructing objects and the categories of those obstructing objects.

[0041] The classification unit 45 generates a shielding object distribution model by calculating a shielding object distribution function using the shielding object parameters for each shielding object category. The classification unit 45 selects, as the shielding object category, a category corresponding to the shielding object distribution model that is most similar to the shielding object distribution from among all categories of shielding object distribution models, based on the shielding object distribution supplied from the distribution generation unit 44. The classification unit 45 outputs the category information to the display device 13 as shielding object information related to the shielding object on the optical element unit 31.

[0042] The storage unit 46 stores restoration parameters, generation parameters, and obstruction parameters.

[0043] The display device 13 displays the category information supplied from the classification unit 45. If the category represented by the category information is a category of obstructions that are unlikely to disappear naturally, the user removes the obstructions by, for example, cleaning the optical element unit 31. For example, if the category represented by the category information is mud or feces and urine that are unlikely to disappear naturally, the user removes the obstructions. On the other hand, if the category represented by the category information is snow that is likely to disappear naturally by melting, the user does not remove the obstructions. Note that at this time, the user may also remove the obstructions if the snow that is an obstruction is unlikely to disappear naturally by melting, based on the weather conditions at the location where the imaging device 11 is installed.

[0044] As described above, the user can remove the obstruction at an appropriate time based on the category information, which allows the user to remove the obstruction more efficiently than when removing the obstruction regardless of the category of the obstruction, thereby reducing the cost required for removing the obstruction.

[0045] The image processing device 12 may output the category information not to the display device 13 but to an obstruction removal device (not shown) that removes the obstruction by cleaning or the like in a subsequent stage. In this case, the obstruction removal device determines whether or not to remove the obstruction based on the category information.

[0046] The image processing device 12 may be configured to include a restoration unit 41 and a storage unit 46, and to output only the restored image to the outside.

[0047] The image processing device 12 is configured with a restoration unit 41, an image generation unit 42, a difference calculation unit 43, a distribution generation unit 44, and a storage unit 46, and may output a restored image and an obstructing object distribution to the display device 13. In this case, the user can know the presence and amount of obstructing objects from the obstructing object distribution displayed on the display device 13. Therefore, the user can remove the obstructing objects at an appropriate time based on the presence and amount of obstructing objects. As a result, the cost required for removing the obstructing objects can be reduced.

[0048] The pixel values ​​of each pixel in the captured image, restored image, predicted image, and difference image may be luminance values ​​such as RGB values ​​or grayscale values, or may be values ​​equivalent to luminance values ​​such as time differences of luminance values.

[0049] The processing of each unit of the image processing device 12 may be performed for each captured image, or may be performed collectively for a plurality of captured images as parallel processing or batch processing.

[0050] The classification unit 45 may output, as the obstructing object information, alert information that prompts cleaning or replacement of the optical element unit 31 based on the category of the obstructing object instead of the category information. In this case, for example, if the category of the obstructing object is mud or feces and urine, which are unlikely to disappear, the classification unit 45 outputs the alert information. On the other hand, if the category of the obstructing object is snow, which is likely to disappear naturally by melting, the classification unit 45 does not output the alert information. Note that in this case, the classification unit 45 may also output the alert information based on the weather conditions of the installation location of the imaging device 11, if the snow, which is the obstructing object, is unlikely to disappear naturally by melting.

[0051] The classification unit 45 may output, instead of category information, removal information such as adhesion strength of the obstructing object and an optimal remover based on the category of the obstructing object as the obstructing object information. The classification unit 45 may output two or more of the category information, alert information, and removal information.

[0052] The restoration unit 41 may supply the restored image to the display device 13 for display.

[0053] <Structural Example of Imaging Device> FIG. 2 is a cross-sectional view showing a structural example of the imaging device 11 of FIG.

[0054] As shown in FIG. 2, the imaging device 11 is configured by arranging an optical element section 31 and an imaging element section 32 in a holder 60 .

[0055] The optical element section 31 includes, in order from the subject side, which is the upper side in FIG. 2, a surface section 61, a buffer section 62, and an optical element 63.

[0056] The surface portion 61 is fixed by the holder 60 so as to be positioned at an upper part within the holder 60. The surface portion 61 is configured by arranging, at a predetermined surface density, microstructures 61b, which are fine three-dimensional structures that amplify the water repellency and biological repellency as surface properties, on a plane 61a facing the subject, which is formed by one surface of a highly transparent polyhedral medium that transmits light incident from the subject. Methods for forming the surface portion 61 include, for example, attaching or pressing the microstructures 61b to the plane 61a.

[0057] 2, the microstructure 61b is composed of one minute convex portion (protrusion) having a trapezoidal cross section, and is arranged in a dot pattern on the plane 61a. This allows the microstructure 61b to have a water-repellent effect, a so-called lotus effect, that prevents droplets of an obstructing object from adhering to the upper surface of the surface portion 61, and amplifies the water-repellent property as a surface characteristic.

[0058] The buffer section 62 is an optical buffer section having a predetermined thickness, and is formed so that its upper surface contacts the surface section 61 and its lower surface contacts the optical element 63. This buffer section 62 serves to prevent the area shaded by the microstructure 61b of the surface section 61 from being concentrated on a specific pixel of the imaging element section 32. The buffer section 62 may be formed of a single layer such as a resin layer or an air layer, or may be formed of a multilayer structure in which resin layers, air layers, etc. are laminated. When the buffer section 62 is formed of a single layer of resin, the buffer section 62 is bonded to the surface section 61 and the optical element 63 or fixed by the holder 60. When the buffer section 62 is formed of a single layer of air, the buffer section 62 is formed by fixing the surface section 61 and the optical element 63 with a predetermined thickness of separation between them by the holder 60. The buffer section 62 allows light emitted from the surface section 61 to enter the optical element 63. The buffer section 62 does not have to be transparent as long as it can transmit light. In the case where the surface portion 61 also serves as the buffer portion 62, the buffer portion 62 may be omitted.

[0059] The optical element 63 is formed on the lower surface of the buffer section 62 and fixed by the holder 60. The optical element 63 functions as a coded aperture that spatially modulates the amount of light incident from the buffer section 62. Specifically, the optical element 63 is an optical element in which a plurality of micropores 63a are arranged so as to spatially modulate the amount of incident light to different light amounts depending on the position on the cross section of the optical path. Therefore, the arrangement pattern of the plurality of micropores 63a is, for example, a low autocorrelation pattern in which the correlation between the positions of the micropores 63a is low.

[0060] The imaging element unit 32 includes an imaging element 71 formed on the bottom surface of the holder 60 at a predetermined distance from the bottom surface of the optical element 63. The imaging element 71 captures an optical image of light incident from a subject through the surface unit 61, the buffer unit 62, and the optical element 63.

[0061] As described above, in the optical element 63, the multiple micropores 63a are arranged so as to spatially modulate the amount of incident light to different amounts depending on the position on the cross section of the optical path, so that the captured image corresponding to the light incident on the image sensor 71 via the optical element 63 has the characteristics of the image formed on the subject. Therefore, the robustness of the restoration process by the restoration unit 41 is improved, and the restoration unit 41 can generate a restored image more reliably.

[0062] An air layer or a layer other than an air layer may be formed between the imaging element 71 and the optical element 63. The optical element 63 may be adhered to the imaging element 71 in the form of a sticker or the like.

[0063] <Examples of Cross-Sectional Shape of Microstructure> FIG. 3 is a cross-sectional view of the surface portion 61 showing examples of the cross-sectional shape of the microstructure 61b.

[0064] In FIG. 3, for the sake of simplicity, only the area around one microstructure 61b of the surface portion 61 is shown.

[0065] In the example of Fig. 3A, the cross-sectional shape of the microstructure 61b is triangular. In the example of Fig. 3B, the cross-sectional shape of the microstructure 61b is trapezoidal, similar to the example of Fig. 2. Note that the cross-sectional shape of the microstructure 61b is not limited to the shape shown in Fig. 3.

[0066] As shown in Fig. 3A, when the cross-sectional shape of the microstructure 61b is triangular, the microstructure 61b is preferably opaque. When the microstructure 61b is opaque, scattering of light incident on the surface portion 61 by the microstructure 61b can be suppressed. When the cross-sectional shape of the microstructure 61b is trapezoidal, as shown in Fig. 3B, the microstructure 61b is transparent.

[0067] <Examples of Shape of Microstructure> FIG. 4 is a perspective view showing examples of the shape of the microstructure 61b.

[0068] In the example of FIG. 4A, the microstructure 61b is a convex portion having a substantially elliptical cylindrical shape with another convex portion having a substantially elliptical cylindrical shape at the center. In the example of FIG. 4B, the microstructure 61b is a hemispherical convex portion. In the example of FIG. 4C, the microstructure 61b is a three-dimensional convex portion having the same rounded rectangular surface as the upper and lower surfaces. In the example of FIG. 4D, the microstructure 61b is a spiral-shaped convex portion. In the example of FIG. 4E, the microstructure 61b is composed of a cluster of multiple spherical convex portions.

[0069] The shape of the microstructure 61b is not limited to the shape shown in Fig. 4, as long as the microstructure 61b is composed of one or more minute protrusions, recesses, or a combination of protrusions and recesses. For example, the microstructure 61b may be composed of protrusions having a substantially elliptical cylindrical shape with a recess at the center. The microstructure 61b may have a shape such as a ridge line or a groove line on the upper surface.

[0070] <Example of Arrangement of Microstructures> FIG. 5 is a top view of the surface portion 61 showing an example of arrangement of the microstructures 61b.

[0071] In FIG. 5, in order to simplify the drawing, only a part of the surface portion 61 is shown.

[0072] In the example of Fig. 5A, microstructures 61b whose upper surfaces are elliptical are arranged in a dot pattern on the plane 61a, similar to the example of Fig. 2. In the example of Fig. 5B, microstructures 61b whose upper surfaces are ridge-shaped are arranged periodically in two predetermined directions on the plane 61a. That is, the microstructures 61b are arranged in a mesh pattern.

[0073] The microstructures 61b may be arranged in a manner other than that shown in Fig. 5. For example, the microstructures 61b may be arranged periodically in one or more predetermined directions. By arranging the microstructures 61b in a predetermined direction or period that is disliked by attached organisms, the organism repellency as a surface characteristic can be enhanced.

[0074] From the viewpoint of surface characteristics, it is desirable that the microstructures 61b of the surface portion 61 are arranged at a high density to obtain effects such as amplifying water repellency (amplifying surface tension), etc. On the other hand, from the viewpoint of optics, it is desirable that the microstructures 61b are arranged at as low a density as possible to prevent light from the subject from being blocked and the amount of light from being reduced.

[0075] <Method of Determining Thickness of Buffer Section> FIG. 6 is a diagram for explaining a method of determining the thickness of the buffer section 62 in FIG.

[0076] In Fig. 6, the horizontal direction of the imaging element 71 is the x-direction, the vertical direction is the y-direction, and the thickness direction of the imaging device 11 is the z-direction. The position on the surface of the optical element 63 corresponding to the center of the bottom surface of a certain microstructure 61b is the origin, and the thickness of the buffer section 62 is d. Fig. 6A is a plan view of the imaging device 11 at z=d, i.e., a top view, and Fig. 6B is a plan view of the imaging device 11 at y=0, i.e., a cross-sectional view. In Fig. 6, to simplify the drawing, only the areas around seven microstructures 61b of the surface section 61 are shown.

[0077] In order to prevent a specific region of the optical element 63 or the imaging element 71 from always being in the shadow of the microstructure 61b, it is necessary to provide a transparent buffer portion 62 between the surface portion 61 and the optical element 63. The thickness d of this buffer portion 62 is determined based on, for example, the density of the microstructure 61b and the maximum incident angle.

[0078] Specifically, in the example of FIG. 6, as shown in FIG. 6A and FIG. 6B, the microstructure 61b is composed of one cone with a circular bottom surface having a radius of a. As shown in FIG. 6A, the microstructure 61b is arranged such that the center of the bottom surface is at the vertex of an equilateral triangle 90 with one side being p (2a < p) on the plane 61a where z = d. As shown in FIG. 6B, the thickness of the plane 61a is negligibly thin.

[0079] In this case, for the light that can reach the origin among the light incident from the subject, the radius r of the circular region 91 on the plane 61a of the surface portion 61 is expressed as r = dtanθ using the maximum incident angle θ of the light incident from the subject and the thickness d of the buffer portion 62. Here, since the maximum incident angle θ is a fixed value determined by other elements, the radius r changes according to the thickness d. On the other hand, the most efficient buffer portion 62 is the buffer portion 62 in which the ratio ρ(r) = B(r) / A(r) of the area B(r) of the region of the microstructure 61b (hereinafter referred to as the microstructure region) within the region 91 to the area A(r) of the region 91 is minimized. Therefore, the thickness d of the most efficient buffer portion 62 is determined based on, for example, the radius r at which the ratio ρ(r) is minimized.

[0080] Here, when the radius r of the region 91 is 0.5p, the microstructure region is composed of a circular bottom surface with a radius a of the microstructure 61b corresponding to the origin (hereinafter referred to as the central microstructure). Therefore, the ratio ρ(r) is obtained by the following formula (1).

[0081]

[0082] When the radius r of the region 91 is 1.0p, the microstructure region is composed of a circular bottom surface with a radius a of the central microstructure and a sector region smaller than a semi - circle of each circular bottom surface with a radius a of the six microstructures 61b arranged closest to the outside of the central microstructure. Therefore, the ratio ρ(r) is obtained by the following formula (2). Hereinafter, these six microstructures 61b are referred to as the outer microstructure group.

[0083]

[0084] When the radius r of the region 91 is 1.5p, the fine structure region is composed of the circular bottom surfaces of the origin fine structure and the outer fine structure group with radius a. Therefore, the ratio ρ(r) is obtained by the following formula (3).

[0085]

[0086] When the radius r of the region 91 is √3p, the fine structure region is composed of the circular bottom surfaces of the origin fine structure and the outer fine structure group with radius a, and six sector regions smaller than a semi-circle among the bottom surfaces of the six fine structures 61b arranged closest to the outside of the outer fine structure group. Therefore, the ratio ρ(r) is obtained by the following formula (4).

[0087]

[0088] From the above, it can be seen from the mean value theorem that there is a minimum value of the ratio ρ(r) in the range of 1.0p < r < √3p. Note that this minimum value is smaller than the average value of the ratio ρ(r) obtained by the following formula (5).

[0089]

[0090] As shown in formula (5), the average value of the ratio ρ(r) is 0.5πa, which is the area of three sectors where the area A(r) is 1 / 6 of the circle, ２ and is equal to the ratio ρ(r) when the area B(r) is the area of the equilateral triangle 90, which is 0.25√3p ２

[0091] As described above, the radius r at which the ratio ρ(r) is minimized exists in the range of 1.0p < r < √3p. Therefore, the thickness d of the buffer portion 62 is determined to be about 1.5p / tanθ based on p representing the density of the fine structure 61b and the maximum incident angle θ so that the efficiency of the buffer portion 62 is improved. Note that the thickness d is not limited to about 1.5p / tanθ as long as it is determined so that the ratio ρ(r) becomes small.

[0092] <Description of Image Processing> Fig. 7 is a flowchart for explaining the image processing by the image processing apparatus 12 in Fig. 1. This image processing is started, for example, when the power of the image processing apparatus 12 is turned on and the image processing apparatus 12 is activated. ​

[0093] 7 , the restoration unit 41 and distribution generation unit 44 of the image processing device 12 load (read) the restoration parameters stored in the storage unit 46. The image generation unit 42 loads the generation parameters stored in the storage unit 46. The classification unit 45 loads the obstructing object parameters stored in the storage unit 46.

[0094] In step S12, the restoration unit 41 determines whether or not a captured image has been input from the image sensor 71. If it is determined in step S12 that a captured image has not yet been input, the restoration unit 41 waits until a captured image is input. On the other hand, if it is determined in step S12 that a captured image has been input, the process proceeds to step S13.

[0095] In step S13, the restoration unit 41 performs restoration processing on the captured image input from the image sensor 71 based on the restoration parameters read out in step S11, and generates a restored image.

[0096] In step S14, the restoration unit 41 outputs the restored image generated in step S13 to the image generation unit 42, and also outputs it to the outside of the imaging system 10 as a final captured image.

[0097] In step S15, the image processing device 12 performs a distribution generation process to generate a distribution of obstructing objects. Details of this distribution generation process will be described later with reference to FIG.

[0098] After the process of step S15, in step S16, the classification unit 45 performs a classification process to classify the obstructing objects into categories based on the obstructing object distribution generated by the distribution generation process of step S15. Details of this classification process will be described later with reference to FIG.

[0099] After the process of step S16, in step S17, the image processing device 12 determines whether or not to end the image processing, for example, whether or not the power supply of the image processing device 12 has been turned off. If it is determined in step S17 that the image processing should not be ended, for example, if the power supply of the image processing device 12 has not been turned off, the process returns to step S12, and the subsequent processes are repeated.

[0100] On the other hand, if it is determined in step S17 that the image processing is to be ended, for example, if the power supply of the image processing device 12 is turned off, the processing ends.

[0101] The image processing device 12 may also save drive power by performing image processing only during a pre-scheduled period or by performing image processing only when an event such as the input of a captured image is detected.

[0102] <Description of Distribution Generation Process> FIG. 8 is a flowchart illustrating the distribution generation process in step S15 of FIG.

[0103] In step S31 of Fig. 8, the image generation unit 42 acquires the restored image output in step S14 of Fig. 7. In step S32, the image generation unit 42 performs a generation process on the restored image acquired in step S31 based on the generation parameters read in step S11 of Fig. 7, to generate a predicted image. In step S33, the image generation unit 42 outputs the predicted image generated by the process of step S32 to the difference calculation unit 43.

[0104] In step S34, the difference calculation unit 43 acquires the predicted image output in step S33. In step S35, the difference calculation unit 43 generates a difference image based on the predicted image acquired in step S34 and the real image, which is the captured image input from the image sensor 71. In step S36, the difference calculation unit 43 outputs the difference image generated in step S35 to the distribution generation unit 44.

[0105] In step S37, the distribution generation unit 44 performs difference conversion processing on the difference image output in step S36 based on the restoration parameters read in step S11, to generate a distribution of obstructing objects. The distribution generation unit 44 supplies the obtained distribution of obstructing objects to the classification unit 45. Then, the process returns to step S15 in FIG. 7 and proceeds to step S16.

[0106] <Explanation of Classification Processing> FIG. 9 is a flowchart illustrating the classification processing in step S16 of FIG.

[0107] In step S51 of Fig. 9, the classification unit 45 acquires the obstructing object distribution generated in step S37 of Fig. 8. In step S52, the classification unit 45 selects a category of obstructing objects based on the obstructing object parameters read in step S11 of Fig. 7 and the obstructing object distribution acquired in step S51. In step S53, the classification unit 45 outputs category information of the category selected in step S52 to the display device 13. As a result, the category information is displayed on the display device 13. Then, the process returns to step S16 of Fig. 7 and proceeds to step S17.

[0108] As described above, the optical element section 31 includes, in order from the subject side, the surface section 61, the optical buffer section 62 having a thickness R, and the optical element 63 on the optical path from the subject to the image sensor 71. The surface section 61 has a microstructure 61b that amplifies the surface characteristics. In the optical element 63, a plurality of micropores 63a are arranged so as to spatially modulate the amount of incident light to different light amounts depending on the position on the cross section of the optical path.

[0109] Therefore, in an imaging device 11 that uses an optical element unit 31 that does not have an imaging function, it is possible to enhance the surface characteristics while suppressing degradation of the image quality of the captured image. As a result, for example, if water repellency is enhanced as a surface characteristic, adhesion of obstructing substances can be suppressed and the cost required for removing the obstructing substances can be reduced. If bio-repellency is enhanced as a surface characteristic, adhesion of adherent organisms such as bacteria, barnacles, and algae can be prevented and the cost required for removing the adherent organisms can be reduced. This is particularly effective when the imaging device 11 is installed on a ship or the like.

[0110] The image generating unit 42 may perform the generation process by simulating the image capturing process using a physical model such as a ray tracing model.

[0111] The surface properties amplified by the microstructure 61b may be physicochemical properties other than water repellency, such as hydrophilicity, adhesiveness, and adsorptive oil repellency, or biochemical properties other than biorepellency. For example, if the microstructure 61b amplifies hydrophilicity as a surface property, it can guide droplets of the obstructing material to the outside and prevent the droplets from adhering to the flat surface 61a. The number of surface properties amplified by the microstructure 61b may be multiple. For example, the microstructure 61b may amplify both water repellency and biorepellency.

[0112] The imaging system 10 may display, as the obstructing object information, prediction information that indicates a predicted value of the obstructing object rate, which is the proportion of the obstructing object occupying the light receiving surface of the imaging element 71, instead of category information.

[0113] In this case, the current shielding rate is calculated based on the distribution of shielding objects, and a parameterized linear function, exponential function, logarithmic function, etc. is fitted to the time transition of the shielding rate based on the shielding rate history, and an extrapolation calculation is performed to extrapolate the future shielding rate. Then, prediction information representing the extrapolated future shielding rate as a predicted value of the shielding rate is output to the display device 13 and displayed on the display device 13.

[0114] The user removes the obstruction when the obstruction rate indicated by the prediction information exceeds a threshold. This allows the user to remove the obstruction before the obstruction degrades the image quality of the captured image. In this way, the user can remove the obstruction at an appropriate time based on the prediction information, which allows the user to remove the obstruction more efficiently than if the obstruction were removed at regular intervals. As a result, the cost required for removing the obstruction can be reduced.

[0115] The predicted information may be output to a downstream obstruction removal device (not shown) instead of the display device 13. In this case, the obstruction removal device determines whether or not to remove the obstruction based on the predicted information. Instead of the predicted information, obstruction rate information regarding the current obstruction rate may be displayed as obstruction information on the display device 13. The obstruction rate information includes information indicating the current obstruction rate, alert information that prompts cleaning or replacement of the optical element unit 31 based on the current obstruction rate, and the like.

[0116] Information regarding the predicted value of the occlusion rate other than the prediction information may be output as the occlusion information. For example, alert information prompting the user to replace the optical element unit 31 may be output as the occlusion information when the occlusion rate indicated by the prediction information exceeds the range of the specification standard. That is, when the occlusion rate indicated by the prediction information exceeds the range of the specification standard, deterioration of the anti-fouling function of the optical element unit 31 is predicted. Therefore, by outputting alert information prompting the user to replace the optical element unit 31, the user can replace the optical element unit 31 before the anti-fouling function of the optical element unit 31 deteriorates and the image quality of the captured image deteriorates. Note that alert information prompting the user to remove the occlusion rate may be output when the occlusion rate indicated by the prediction information exceeds a threshold. Both the prediction information and the alert information may be output.

[0117] The current occlusion rate may be calculated based on a scalar quantity such as the average luminance of the occlusion object, rather than the occlusion object distribution. In this case, the calculation cost can be reduced compared to when the calculation is based on the occlusion object distribution, which is a multidimensional quantity.

[0118] The imaging element 71 may be a sensor that detects other electromagnetic waves, such as a microwave sensor. The imaging system 10 may output the image captured by the imaging element 71 as is, instead of a restored image.

[0119] The surface portion 61 may be formed on a lens (not shown). In this case, the captured image is an image-formed image. Therefore, the distribution generation unit 44 may convert the differential image directly into an obstruction distribution without performing a differential conversion process. The restoration unit 41 may output the captured image input from the image sensor unit 32 as the final captured image without any processing.

[0120] 2. Second Embodiment Structural Example of Surface Buffer Unit A second embodiment of an imaging system to which the present technology is applied differs from the imaging system 10 in Fig. 1 in that a surface unit 61 and a buffer unit 62 are integrated as a surface buffer unit, but is otherwise configured in the same manner as the imaging system 10. Therefore, only the surface buffer unit will be described below.

[0121] FIG. 10 is a cross-sectional view showing an example of the structure of the surface buffer section.

[0122] In Fig. 10, the same reference numerals are used to designate parts corresponding to those in Fig. 2. In order to simplify the drawing, Fig. 10 shows only the area surrounding one microstructure 61b of the surface buffer portion.

[0123] In the example of Fig. 10, the microstructure 61b is composed of a single minute recess with a triangular cross-sectional shape. The surface buffer 161 is an optical buffer having a predetermined thickness R that transmits light incident from the subject. The surface buffer 161 has a plane 161a on the subject side, which is the upper side in Fig. 10, formed by one surface of a highly transparent polyhedral medium, and the microstructure 61b is formed on this plane 161a. Methods for forming the surface buffer 161 include, for example, forming the microstructure 61b on the plane 161a by etching, or forming the microstructure 61b by vapor deposition, sputtering, coating, spraying, printing, or the like.

[0124] As described above, in the second embodiment, the surface portion 61 and the buffer portion 62 are integrated as the surface buffer portion 161, so that the manufacturing cost of the optical element portion 31 can be reduced.

[0125] 3. Computer The series of processes performed by the image processing device 12 described above can be executed by hardware or software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, the term "computer" includes computers built into dedicated hardware and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0126] FIG. 11 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes by the image processing device 12 described above using a program.

[0127] In the computer, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are interconnected by a bus 904.

[0128] An input / output interface 905 is also connected to the bus 904. An input unit 906, an output unit 907, a storage unit 908, a communication unit 909, and a drive 910 are connected to the input / output interface 905.

[0129] The input unit 906 includes a keyboard, a mouse, a microphone, etc. The output unit 907 includes a speaker, a display device 13 (413), etc. The storage unit 908 includes a hard disk, a non-volatile memory, etc. The communication unit 909 includes a network interface, etc. The drive 910 drives removable media 911 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0130] In a computer configured as described above, the CPU 901 performs the above-described series of processes by, for example, loading a program stored in the memory unit 908 into the RAM 903 via the input / output interface 905 and the bus 904 and executing it.

[0131] The program executed by the computer (CPU 901) can be provided by being recorded on a removable medium 911 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0132] In a computer, the program can be installed in the storage unit 908 via the input / output interface 905 by inserting the removable medium 911 into the drive 910. The program can also be received by the communication unit 909 via a wired or wireless transmission medium and installed in the storage unit 908. Alternatively, the program can be installed in the ROM 902 or the storage unit 908 in advance.

[0133] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0134] The series of processes performed by the image processing device 12 may be executed by a GPU (Graphics Processing Unit).

[0135] 4. Application Examples to Mobile Bodies The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of mobile body, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, or a robot.

[0136] FIG. 12 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied.

[0137] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 12, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also shown as functional components of the integrated control unit 12050 are a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (interface) 12053.

[0138] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0139] The body system control unit 12020 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0140] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture images outside the vehicle and receives the captured images. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, characters on the road surface, etc. based on the received images.

[0141] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0142] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.

[0143] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, steering mechanism, or braking device based on the information inside and outside the vehicle acquired by the outside-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc.

[0144] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle obtained by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0145] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching from high beams to low beams.

[0146] The audio / video output unit 12052 transmits at least one of audio and video output signals to an output device capable of visually or audibly notifying the passengers of the vehicle or the outside of the vehicle of information. In the example of Fig. 12, the output devices are exemplified by an audio speaker 12061, a display unit 12062, and an instrument panel 12063. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

[0147] FIG. 13 is a diagram showing an example of the installation position of the imaging unit 12031.

[0148] In FIG. 13, the vehicle 12100 has imaging units 12101, 12102, 12103, 12104, and 12105 as the imaging unit 12031.

[0149] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at positions such as the front nose, side mirrors, rear bumper, back door, and the top of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the top of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided on the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The forward images acquired by the imaging units 12101 and 12105 are mainly used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0150] 13 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 12101 to 12104, an overhead image of the vehicle 12100 viewed from above can be obtained.

[0151] At least one of the image capturing units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image capturing units 12101 to 12104 may be a stereo camera made up of multiple image capturing elements, or may be an image capturing element having pixels for phase difference detection.

[0152] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 can calculate the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100), thereby extracting as a preceding vehicle, in particular, the three-dimensional object that is the closest three-dimensional object on the path of the vehicle 12100 and traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km / h or higher). Furthermore, the microcomputer 12051 can set a vehicle-to-vehicle distance to be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. In this way, cooperative control can be performed for the purpose of autonomous driving, which runs autonomously without relying on driver operation.

[0153] For example, the microcomputer 12051 classifies and extracts three-dimensional object data regarding three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 then determines a collision risk that indicates the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0154] At least one of the image capturing units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the images captured by the image capturing units 12101 to 12104. Such pedestrian recognition is performed, for example, by extracting feature points from the images captured by the image capturing units 12101 to 12104 as infrared cameras and performing pattern matching on a series of feature points that indicate the outline of an object to determine whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the images captured by the image capturing units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular outline on the recognized pedestrian for emphasis. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating the pedestrian at a desired position.

[0155] The above describes an example of a vehicle control system to which the technology according to the present disclosure can be applied. The technology according to the present disclosure can be applied to the imaging unit 12031, the outside vehicle information detection unit 12030, and other components of the above-described configuration. Specifically, for example, the imaging device 11 can be applied to the imaging unit 12031. By applying the technology according to the present disclosure to the imaging unit 12031, it is possible to amplify surface characteristics while suppressing degradation in the image quality of the captured image. As a result, it is possible to reduce the cost required to remove an object blocking the imaging unit 12031 while suppressing degradation in the image quality of the captured image, for example.

[0156] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0157] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of the present technology.

[0158] For example, the obstructing object information may be a combination of two or more of category information, alert information, removal information, prediction information, and obstruction rate information.

[0159] Each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0160] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0161] The effects described in this specification are merely examples and are not limiting, and there may be effects other than those described in this specification.

[0162] The present technology can have the following configurations: (1) An optical device comprising, on an optical path from a subject to an imaging element, a surface portion having a microstructure that amplifies surface characteristics, an optical buffer portion having a predetermined thickness, and an optical element having a plurality of micropores arranged so as to spatially modulate the amount of incident light to a different amount depending on the position on a cross section of the optical path. (2) The optical device described in (1), wherein the surface portion is configured by forming the microstructure on a plane that transmits light incident from the subject, and the predetermined thickness is determined so that a ratio of an area of ​​the microstructure in the incident area to an area of ​​an incident area on the surface portion through which the light can reach a predetermined position on the optical element is small. (3) The optical device described in (2), wherein the predetermined thickness is determined based on a density of the microstructure and a maximum incident angle of the light. (4) The optical device according to (2) or (3), wherein the plane is formed by one surface of a highly transparent polyhedral medium, and the microstructure is configured by one or more minute convex portions, concave portions, or a combination of convex portions and concave portions, and is arranged on the plane at a predetermined surface density. (5) The optical device according to (2) to (4), wherein the microstructure is arranged on the plane in a predetermined direction or period. (6) The optical device according to any of (1) to (5), wherein the surface property is water repellency, hydrophilicity, adhesiveness, adsorptiveness, oil repellency, or bio-repellency. (7) The optical device according to any of (1) to (6), wherein the surface portion and the buffer portion are integrated. (8) An imaging device comprising: an optical element section on an optical path from a subject to an imaging element, the optical element section comprising: a surface section having a microstructure that amplifies surface characteristics; an optical buffer section having a predetermined thickness; and an optical element having a plurality of micropores arranged so as to spatially modulate the amount of incident light to different light amounts depending on the position on the cross section of the optical path; and the imaging element that captures an optical image of the light incident through the optical element section.

[0163] REFERENCE SIGNS LIST 11 imaging device, 31 optical element portion, 32 imaging element portion, 61 surface portion, 61a plane, 61b microstructure, 62 buffer portion, 63 optical element, 63a micropore, 71 imaging element, 161 surface buffer portion, 161a plane

Claims

1. An optical device comprising, on an optical path from a subject to an image sensor, a surface portion having a microstructure that amplifies surface characteristics, an optical buffer portion having a predetermined thickness, and an optical element having a plurality of micropores arranged so as to spatially modulate the amount of incident light into different amounts of light depending on the position on the cross section of the optical path.

2. The optical device according to claim 1, wherein the surface portion is configured by forming the microstructure on a plane that transmits the light incident from the subject, and the predetermined thickness is determined so that the ratio of the area of ​​the region of the microstructure within the incident region, which is the area on the surface portion of the light that can reach a predetermined position on the optical element, is small.

3. The optical device according to claim 2, wherein the predetermined thickness is determined based on the density of the microstructure and the maximum angle of incidence of the light.

4. The optical device according to claim 2, wherein the plane is formed by one surface of a highly transparent polyhedral medium, and the microstructure is composed of one or more minute convex portions, concave portions, or a combination of convex portions and concave portions, and is arranged on the plane at a predetermined surface density.

5. The optical device according to claim 2, wherein the microstructures are arranged on the plane in a predetermined direction or with a predetermined period.

6. The optical device according to claim 1, wherein the surface characteristics are water repellency, hydrophilicity, adhesiveness, adsorptiveness, oleophobicity, or bio-repellency.

7. The optical device according to claim 1, wherein the surface portion and the buffer portion are integrated.

8. An imaging device comprising: an optical element section on an optical path from a subject to an imaging element, the optical element section comprising: a surface section having a microstructure that amplifies surface characteristics; an optical buffer section having a predetermined thickness; and an optical element having a plurality of micropores arranged so as to spatially modulate the amount of incident light to different light amounts depending on the position on a cross section of the optical path; and the imaging element that captures an optical image of the light incident through the optical element section.

Images