Imaging system and image alignment method

Abstract

This imaging system includes: an imaging element; an optical member that disperses light from a subject into a plurality of optical components, and forms a plurality of images, each of which is an image of the subject based on a corresponding light component among the plurality of light components, so as to be spatially shifted on an imaging surface of the imaging element; a processing circuit that performs alignment between the plurality of images on the basis of a feature point in each of the plurality of images; and a light source that emits illumination light for illuminating a part of the subject. Each of the plurality of light components has a wavelength included in a corresponding wavelength region among a plurality of wavelength regions, or has a polarization angle included in a corresponding polarization angle region among a plurality of polarization angle regions. The optical member disperses the light from the subject on the basis of the plurality of wavelength regions or the plurality of polarization angle regions. Each of the plurality of images includes an illumination region corresponding to the part of the subject illuminated by the light source. The plurality of images include a first image and a second image, and the illumination region in the first image does not overlap the illumination region in the second image. The feature point is extracted from the illumination region in each of the plurality of images.

Description

Imaging system and image alignment method

[0001] The present disclosure relates to an imaging system and the like.

[0002] There is known an imaging system that obtains images of multiple wavelength regions and characteristics obtained from the images of multiple wavelength regions by splitting light from a subject using an optical element (see, for example, Patent Document 1). Patent Document 1 discloses an imaging system that forms images of multiple wavelength regions on a single imaging element. In such an imaging system, when obtaining characteristics such as the temperature of the subject from multiple images, it is important to determine which points on the subject correspond to which points on each image. In other words, the accuracy of alignment between each image is important.

[0003] JP 2005-031558 A Japanese Patent No. 7209398 A

[0004] The imaging system described in Patent Document 1 discloses an imaging system that forms images in multiple wavelength regions on a single sensor. Patent Document 2 discloses a method for extracting feature points in images in order to calibrate the correspondence between distorted images. However, when multiple images are formed on a single sensor as in the imaging system of Patent Document 1, it is not possible to distinguish which wavelength the feature points in the formed images belong to, which can reduce the accuracy of alignment between images. A similar problem can also occur when forming multiple images with different polarization characteristics on a single sensor.

[0005] Therefore, an object of the present disclosure is to provide an imaging system and the like that improves the accuracy of alignment between images.

[0006] An imaging system according to one embodiment of the present disclosure includes an imaging element, an optical element that splits light from a subject into multiple light components and forms multiple wavelength images, each of which is an image of the subject based on a corresponding light component among the multiple light components, at spatially shifted positions on an imaging surface of the imaging element, a processing circuit that aligns the multiple images based on feature points in each of the multiple images, and a light source that emits illumination light to illuminate a portion of the subject. Each of the multiple light components has a wavelength included in a corresponding wavelength range among a plurality of wavelength ranges or a polarization angle included in a corresponding polarization angle range among a plurality of polarization angle ranges. The optical element splits the light from the subject based on the multiple wavelength ranges or the multiple polarization angle ranges. Each of the multiple images includes an illumination area corresponding to the portion of the subject illuminated by the light source. The multiple images include a first image and a second image, and the illumination area in the first image does not overlap with the illumination area in the second image. The feature points are extracted from the illumination area in each of the multiple images.

[0007] An imaging system according to one embodiment of the present disclosure includes an imaging element, an optical element that splits light from a subject into multiple light components and forms multiple images, each of which is an image of the subject based on a corresponding light component among the multiple light components, at spatially offset positions on an imaging surface of the imaging element, and a processing circuit that aligns the multiple images based on feature points in each of the multiple images. Each of the multiple light components has a wavelength included in a corresponding wavelength range among a plurality of wavelength ranges or a polarization angle included in a corresponding polarization angle range among a plurality of polarization angle ranges. The optical element splits the light from the subject based on the multiple wavelength ranges or the multiple polarization angle ranges. The subject includes a target marker and a surrounding area, and the reflectance of the target marker is higher than the reflectance of the surrounding area. The multiple images include a first image and a second image, and the target marker in the first image does not overlap with the target marker in the second image. The feature points are extracted from the target marker in each of the multiple images.

[0008] An image alignment method according to one embodiment of the present disclosure includes: splitting light from a subject into a plurality of light components; forming a plurality of images, each of which is an image of the subject based on a corresponding light component among the plurality of light components, at spatially offset positions on an imaging surface of an image sensor; aligning the plurality of images based on feature points in each of the plurality of images; and emitting illumination light to illuminate a portion of the subject. Each of the plurality of light components has a wavelength included in a corresponding wavelength range among a plurality of wavelength ranges or a polarization angle included in a corresponding polarization angle range among a plurality of polarization angle ranges. The light from the subject is split based on the plurality of wavelength ranges or the plurality of polarization angles. Each of the plurality of images includes an illumination area corresponding to the portion of the subject illuminated by the light source. The plurality of images includes a first image and a second image, and the illumination area in the first image does not overlap with the illumination area in the second image. The feature points are extracted from the illumination area in each of the plurality of images.

[0009] An image alignment method according to one embodiment of the present disclosure includes: splitting light from a subject into a plurality of light components; forming a plurality of images, each of which is an image of the subject based on a corresponding light component among the plurality of light components, on an imaging surface of an image sensor at spatially shifted positions; and aligning the plurality of images based on feature points in each of the plurality of images. Each of the plurality of light components has a wavelength included in a corresponding wavelength range among a plurality of wavelength ranges, or a polarization angle included in a corresponding polarization angle range among a plurality of polarization angle ranges. The light from the subject is split based on the plurality of wavelength ranges or the plurality of polarization angle ranges. The subject includes a target marker and a surrounding area, and the reflectance of the target marker is higher than the reflectance of the surrounding area. The plurality of images include a first image and a second image, and the target marker in the first image does not overlap with the target marker in the second image. The feature points are extracted from the target marker in each of the plurality of images.

[0010] The present disclosure provides an imaging system and the like that improves the accuracy of alignment between images.

[0011] FIG. 1 is a block diagram showing a configuration of an imaging system according to a first embodiment. FIG. 2 is a flowchart showing an operation of the imaging system according to the first embodiment when performing alignment. FIG. 3 is a schematic diagram showing the operation of the imaging system according to the first embodiment. FIG. 4 is an enlarged view of an image shown in FIG. 3. FIG. 5 is a schematic diagram showing a configuration of a light source. FIG. 6 is a diagram showing a first example of an optical element in the imaging system according to the first embodiment, which separates light reflected from a subject based on wavelength regions. FIG. 7 is a diagram showing a second example of an optical element in the imaging system according to the first embodiment, which separates light reflected from a subject based on wavelength regions. FIG. 8 is a diagram showing a third example of an optical element in the imaging system according to the first embodiment, which separates light reflected from a subject based on wavelength regions. FIG. 9 is a diagram showing a fourth example of an optical element in the imaging system according to the first embodiment, which separates light reflected from a subject based on wavelength regions. FIG. 10 is a diagram showing wavelength regions. FIG. 11 is a diagram showing a method for calculating the luminance of an image in multiple wavelength regions. FIG. 12 is a diagram showing an example of an image captured by an image sensor. FIG. 13 is a block diagram showing a configuration of an imaging system according to a second embodiment. FIG. 14 is a flowchart showing the operation of the imaging system according to the second embodiment when performing alignment. FIG. 15 is a schematic diagram showing the operation of the imaging system according to the second embodiment. FIG. 16 is a diagram showing an example in which an optical member forms images by spatially shifting two images based on light components of different wavelengths on the imaging surface of an imaging element so that target markers in the two images do not overlap. FIG. 17 is a diagram showing an example in which an optical member forms images by spatially shifting two images based on light components of different wavelengths on the imaging surface of an imaging element so that target markers in the two images overlap. FIG. 18 is a diagram showing wavelength regions. FIG. 19 is a diagram showing example markers. FIG. 20 is a schematic diagram showing the operation of an imaging system of a comparative example. FIG. 21 is an enlarged view of the image shown in FIG. 20. FIG. 22 is a diagram showing an example of an optical member that separates reflected light from a subject into two polarized light in an imaging system according to the third embodiment. FIG. 23 is a diagram showing regions of each polarization angle.FIG. 24 is a diagram showing a method for calculating the brightness of an image of multiple polarizations.

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. The numerical values, components, component placement and connection configurations, steps, step order, display examples, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Therefore, among the components in the following embodiments, components that are not recited in the independent claims of the present disclosure will be described as optional components. Furthermore, each figure is not necessarily an exact illustration. In each figure, substantially identical components are assigned the same reference numerals, and redundant explanations are omitted or simplified.

[0013] First Embodiment [Configuration] FIG. 1 is a block diagram showing the configuration of an imaging system 1 according to a first embodiment.

[0014] The imaging system 1 is a system that captures images in multiple wavelength regions and aligns the images in the multiple wavelength regions. As shown in Fig. 1, the imaging system 1 includes an optical member 11, an imaging element 12, a light source 13, and a processing circuit 14.

[0015] The optical member 11 is a device that separates the reflected light from the subject 2 into a plurality of light components based on a plurality of wavelength regions, and forms a plurality of images, each of which is an image of the subject 2 based on a corresponding light component among the plurality of light components, at spatially shifted positions on the imaging surface of the imaging element 12. The specific configuration of the optical member 11 will be described later. In this specification, each of the plurality of wavelength regions to be separated has an appropriate wavelength width (i.e., wavelength band).

[0016] The image sensor 12 is an electronic component that captures an image of the subject 2 based on the multiple light components dispersed by the optical member 11. The images captured by the image sensor 12 are images of the peripheral area and the illumination area of ​​each of the images in the multiple wavelength regions. The peripheral area and the illumination area will be described later.

[0017] The light source 13 is a device that irradiates light onto the subject 2. The light source 13 also emits illumination light that illuminates a part of the subject 2. The specific configuration of the light source 13 will be described later.

[0018] The processing circuit 14 aligns the images in the multiple wavelength regions based on feature points in each of the images in the multiple wavelength regions captured by the imaging element 12. Note that aligning the images in the multiple wavelength regions means correcting misalignment of the multiple images. Also, a feature point means a point that can be prominently detected on an image. For example, a corner of a figure is a feature point. The processing circuit 14 outputs information or an image obtained in aligning the images in the multiple wavelength regions. The processing circuit 14 aligns the images in the multiple wavelength regions using the following procedure.

[0019] First, the processing circuit 14 extracts coordinates from feature points in each image of each wavelength region. Coordinate extraction is a method of detecting the coordinates of an object from an edge or surrounding image, which are important for identifying the shape of the object.

[0020] Next, the processing circuit 14 aligns the images of the multiple wavelength regions using a method of calculating a transformation matrix based on the coordinates obtained by the coordinate extraction. The method of calculating a transformation matrix is ​​effective when the coordinates of the active markers in the images of the wavelength regions are different due to image distortion or the like, making it impossible to align the images. The processing circuit 14 can enlarge or reduce, shift, or rotate the image by performing pixel correction using, for example, projective transformation. Furthermore, when the coordinates of four active markers are detected, the processing circuit 14 can correct the shape of the image from a parallelogram or trapezoid to a rectangle, square, or the like. The processing circuit 14 may also use decimal pixel accuracy for pixel correction. Furthermore, the processing circuitry 14 may use a correlation coefficient between images in multiple wavelength regions or a correlation coefficient such as a normalized correlation coefficient to determine the transformation matrix, or may use a program such as Scale-invariant feature transformation (SIFT) or Speeded Up Robust Features (SURF) to determine the transformation matrix.

[0021] [Alignment Operation] FIG. 2 is a flowchart showing the alignment operation of the imaging system 1 according to the first embodiment.

[0022] First, the light source 13 emits illumination light that illuminates a part of the subject 2 (step S1).

[0023] Next, the optical member 11 separates the reflected light from the subject 2 into a plurality of light components based on wavelength ranges, and forms a plurality of images of the subject 2, each of which is based on a corresponding light component among the plurality of light components, on the image sensor 12 so that they do not overlap (step S2). Note that "so that they do not overlap on the image sensor 12" means that the illumination areas of the respective images do not overlap on the image sensor 12.

[0024] Next, the image sensor 12 captures each of the multiple images (step S3).

[0025] Then, the processing circuitry 14 aligns the images based on the feature points in each of the multiple images (step S4).

[0026] Fig. 3 is a schematic diagram illustrating the operation of the imaging system 1 according to the first embodiment. Fig. 3 also illustrates the operations from step S1 to step S3 in Fig. 2. Fig. 3 is a schematic diagram illustrating the optical element 11 separating the light reflected from the subject 2 into two light components based on wavelength, and the imaging element 12 capturing an image of the two light components having different wavelengths. Note that image 121 is an image captured by the imaging element 12. Furthermore, lens 15 is a lens that converges the light separated by the optical element 11 and forms an image on the imaging element 12.

[0027] As shown in FIG. 3, the light source 13 adjusts the area to which the illumination light is irradiated so as to illuminate the illumination area, which is the area of ​​the subject 2 depicted in white, and not illuminate the peripheral area, which is the area of ​​the subject 2 depicted in gray.

[0028] Fig. 4 is an enlarged view of the image 121 shown in Fig. 3. The arrows indicate characteristic points, each pointing to a corner of a figure captured in an image of each wavelength.

[0029] As shown in Fig. 4, image 121 is an image obtained by combining a peripheral region 122 and an illumination region 123 in an image in the first wavelength region captured by image sensor 12 with a peripheral region 122 and an illumination region 123 in an image in the second wavelength region. In other words, peripheral region 122 is the grayed-out region of subject 2 shown in Fig. 3 captured on image sensor 12, and illumination region 123 is the white region of subject 2 in Fig. 3 captured on image sensor 12. Note that in Fig. 4, peripheral region 122 is the region painted black in image 121. Furthermore, illumination region 123 is the white region in image 121 that includes feature points.

[0030] For example, if two illumination areas 123 are formed on the image sensor 12 so as to overlap, the processing circuit 14 cannot distinguish which wavelength a feature point in the image corresponds to, resulting in a decrease in alignment accuracy. Therefore, the light source 13 adjusts the area irradiated with illumination light so that the two illumination areas 123 are formed on the image sensor 12 without overlapping.

[0031] Because the light source 13 emits illumination light so as to illuminate only a portion of the subject 2, it is possible to reduce the illumination area 123 imaged on the image sensor 12. Furthermore, because the optical member 11 separates the reflected light from the subject 2 into multiple light components based on wavelength ranges so that the illumination areas 123 do not overlap on the image sensor 12, the image sensor 12 can capture multiple images without overlapping the illumination areas 123 in each image. Therefore, the processing circuit 14 can align the images based on feature points in each of the multiple images.

[0032] [Configuration of Light Source] Next, a specific configuration of the light source 13 will be described with reference to Fig. 5. Fig. 5 is a schematic diagram showing the configuration of the light source 13. Note that the arrows indicate the irradiation direction.

[0033] As shown in FIG. 5, the light source 13 includes an illumination device 131, a diaphragm 132, a mask 133, and an illumination lens 134.

[0034] The lighting device 131 is, for example, a white light bulb or an LED light bulb.

[0035] The diaphragm 132 is a movable diaphragm for adjusting the amount of illumination light emitted by the lighting device 131. The white area of ​​the diaphragm 132 is an area that can transmit the illumination light. The diaphragm 132 may adjust the amount of light by changing the size of the white area, or may adjust the amount of light by moving between the lighting device 131 and the mask 133.

[0036] The mask 133 is a mask on which a pattern is drawn to limit the area illuminated on the subject 2. The white areas of the mask 133 are areas that allow the illumination light to pass through. Note that by changing the pattern drawn on the mask 133, the shape of the illumination area illuminated on the subject 2 can be changed.

[0037] The projection lens 134 is a lens for forming an image of an irradiation area, which is an enlarged or reduced version of the pattern drawn on the mask 133, on the subject 2.

[0038] This allows the light source 13 to partially illuminate the subject 2. The light source 13 that partially illuminates the subject 2 may be, for example, a spotlight or a collimator light source, or may be a device that can spatially limit the luminance distribution of the illumination.

[0039] [Configuration of Optical Element] Next, a specific configuration of the optical element 11 that realizes step S2 in Fig. 2 will be described with reference to Figs. 6 to 9. The optical element 11 shown in Figs. 6 to 9 is an example of an optical element that separates reflected light from the subject 2 into two wavelengths. In Figs. 6 to 9, dashed arrows indicate the path of light in the long wavelength region, and dotted arrows indicate the path of light in the short wavelength region. Figs. 6 and 7 are cross-sectional views of the optical element 11.

[0040] 6 is a diagram showing a first example of an optical element 11 that separates reflected light from a subject 2 based on wavelength regions in the imaging system 1 according to the first embodiment. Part (a) of Fig. 6 shows that the optical element 11 separates the reflected light into light components having different wavelengths, and the imaging element 12 captures images at two wavelengths. Part (b) of Fig. 6 shows a dielectric multilayer film wavelength prism 111 included in the optical element 11.

[0041] The dielectric multilayer film wavelength prism 111 is an optical element having a dielectric multilayer film wavelength filter 111a. As shown in parts (a) and (b) of Figure 6, the dielectric multilayer film wavelength prism 111 has the dielectric multilayer film wavelength filter 111a arranged perpendicular to the surface facing the lens 15. As shown in part (b) of Figure 6, the dielectric multilayer film wavelength prism 111 may be, for example, a regular triangular prism whose cross section has an equilateral triangle shape, or may be a prism having a shape other than a regular triangular prism shape.

[0042] The dielectric multilayer film wavelength filter 111a is a filter that reflects light in a specific wavelength range and transmits light in other wavelength ranges. In the first embodiment, the dielectric multilayer film wavelength filter 111a is a filter that reflects light in a long wavelength range (for example, a wavelength range of 650 nm or more), but it may also be a filter that reflects light in a short wavelength range.

[0043] 6A, the optical member 11 separates the light reflected from the subject 2 into light in the long wavelength region and light in the short wavelength region using a dielectric multilayer film wavelength prism 111. The lens 15 then converges the light in the long wavelength region and the light in the short wavelength region, respectively, to form an image 121 on the image sensor 12.

[0044] This allows the optical member 11 to separate the light reflected from the subject 2 into light in the long wavelength region and light in the short wavelength region.

[0045] 6A, light in the long wavelength region is reflected twice within the dielectric multilayer film wavelength prism 111 before passing through, whereas light in the short wavelength region is reflected once within the dielectric multilayer film wavelength prism 111 before passing through. As a result, the phase of the wavelength of the light in the long wavelength region is opposite to the phase of the wavelength of the light in the short wavelength region. As a result, the images of the two wavelengths captured by the image sensor 12 are mirror images.

[0046] Fig. 7 is a diagram showing a second example of the optical member 11 that separates the light reflected from the subject 2 based on wavelength regions in the imaging system 1 according to the first embodiment. Fig. 7 differs from Fig. 6 in that the optical member 11 includes a prism 112, but is otherwise the same as Fig. 6.

[0047] The prism 112 is an optical element for dispersing, refracting, or totally reflecting light.

[0048] 7, the reflected light from the subject 2 is incident at a shallow angle on the incident surface of the dielectric multilayer wavelength prism 111. Therefore, the optical member 11 allows the reflected light to pass through the prism 112, so that the reflected light is incident at a deep angle on the incident surface.

[0049] Furthermore, the reflected light incident on the prism 112 is reflected twice within the prism 112 before passing through it. Therefore, the phase of the reflected light before it enters the prism 112 and the phase of the reflected light after it passes through the prism 112 are the same.

[0050] FIG. 8 is a diagram showing a third example of the optical member 11 that separates the light reflected from the subject 2 based on wavelength regions in the imaging system 1 according to the first embodiment.

[0051] As shown in FIG. 8, the optical member 11 has two total reflection mirrors 113 , two half mirrors 114 , and two band-pass filters 115 .

[0052] The total reflection mirror 113 is a mirror that reflects all incident light.

[0053] The half mirror 114 is a mirror that disperses incident light so that the intensity of the reflected light is almost the same as the intensity of the transmitted light when the incident light is incident on one surface, and is a mirror that transmits all of the incident light when the incident light is incident on the other surface.

[0054] The bandpass filters 115 are filters that transmit light of a specific wavelength and attenuate light of other wavelengths. In Fig. 8, one bandpass filter 115 transmits light in the long wavelength range and attenuates light in the short wavelength range, while the other bandpass filter 115 transmits light in the short wavelength range and attenuates light in the long wavelength range.

[0055] First, the first half mirror 114 separates the reflected light from the subject 2. One total reflection mirror 113 totally reflects the reflected light separated by the first half mirror 114, and the other total reflection mirror 113 totally reflects the transmitted light separated by the first half mirror 114. The light totally reflected by the two total reflection mirrors 113 is incident on different bandpass filters 115. One bandpass filter 115 transmits light in the long wavelength region, and the other bandpass filter 115 transmits light in the short wavelength region. Finally, the light in the long wavelength region and the light in the short wavelength region are incident on the second half mirror 114. Then, the lens 15 converges the light in the long wavelength region and the light in the short wavelength region, respectively, to form an image 121 on the image sensor 12. Note that in this figure, the surface onto which the light in the long wavelength region is incident is a surface that transmits all of the incident light.

[0056] This allows the optical member 11 to separate the light reflected from the subject 2 into light in the long wavelength region and light in the short wavelength region.

[0057] Note that the light in the long wavelength region and the light in the short wavelength region are reflected twice within the optical member 11 before being transmitted through the optical member 11. Therefore, the phase of the wavelength of the light in the long wavelength region is the same as the phase of the wavelength of the light in the short wavelength region.

[0058] FIG. 9 is a diagram showing a fourth example of the optical member 11 that separates the light reflected from the subject 2 based on wavelength regions in the imaging system 1 according to the first embodiment.

[0059] As shown in FIG. 9, the optical member 11 has two total reflection mirrors 113 and two dichroic mirrors 116 .

[0060] The dichroic mirror 116 is a mirror that reflects light of a specific wavelength among the incident light and transmits light of other wavelengths. In Fig. 9, one dichroic mirror 116 is a mirror that reflects light in the long wavelength region and transmits light in the short wavelength region, and the other dichroic mirror 116 is a mirror that reflects light in the short wavelength region and transmits light in the long wavelength region.

[0061] First, one dichroic mirror 116 reflects light in the long wavelength region and transmits light in the short wavelength region out of the light reflected from the subject 2. One total reflection mirror 113 totally reflects light in the long wavelength region, and the other total reflection mirror 113 totally reflects light in the short wavelength region. The light totally reflected by the two total reflection mirrors 113 is incident on the other dichroic mirror 116. The other dichroic mirror 116 reflects light in the short wavelength region and transmits light in the long wavelength region. Then, the lens 15 converges the light in the long wavelength region and the light in the short wavelength region, respectively, to form an image 121 on the image sensor 12.

[0062] This allows the optical member 11 to separate the light reflected from the subject 2 into light in the long wavelength region and light in the short wavelength region.

[0063] As in FIG. 8, the phase of the wavelength of light in the long wavelength region is the same as the phase of the wavelength of light in the short wavelength region.

[0064] [Wavelength Range of Light Source] The above describes the light split by the optical element 11. However, for example, if the wavelength of light split by the optical element 11 is not included in the wavelength range to which the image sensor 12 is sensitive, the image sensor 12 cannot capture an image of light of that wavelength. Therefore, the relationship between the wavelength range to which the image sensor 12 is sensitive and the wavelength range of light emitted by the light source will be described with reference to FIG. 10 . FIG. 10 is a diagram showing each wavelength range. Here, a wavelength range refers to at least one wavelength band. The wavelength range to which the image sensor 12 is sensitive refers to the wavelength range of light that the image sensor 12 can receive.

[0065] Part (a) of FIG. 10 shows the sensitive wavelength range of the image sensor 12. Part (b) of FIG. 10 shows two wavelength ranges into which the optical member 11 separates the reflected light. Part (c) of FIG. 10 shows a first example of the wavelength range of the light irradiated by the light source 13. Part (d) of FIG. 10 shows a second example of the wavelength range of the light irradiated by the light source 13. Part (e) of FIG. 10 shows a third example of the wavelength range of the light irradiated by the light source 13. Part (f) of FIG. 10 shows a fourth example of the wavelength range of the light irradiated by the light source 13. Note that, from part (a) to part (f) of FIG. 10, the light indicates a longer wavelength as it moves to the right of the figure, and the light indicates a shorter wavelength as it moves to the left of the figure. In other words, the light in region 1 shown in part (b) of FIG. 10 is light in a short wavelength range, and the light in region 2 is light in a long wavelength range. 10(c) to 10(f) show the wavelength range of light emitted by one light source 13. In FIG.

[0066] 10A and 10B, the regions 1 and 2 are included in the sensitive wavelength range of the image sensor 12. This allows the image sensor 12 to capture images of the regions 1 and 2.

[0067] In order for regions 1 and 2 to be included in the sensitive wavelength range of the image sensor 12, the wavelength range of light before it is dispersed by the optical element 11, i.e., the wavelength range of light with which the light source 13 irradiates the subject 2, must be a wavelength range of light that includes regions 1 and 2.

[0068] As shown in parts (c) to (f) of Fig. 10, the region of wavelengths of light irradiated by light source 13 is a region that includes region 1 and region 2. As shown in parts (c) to (e) of Fig. 10, the region of wavelengths of light irradiated by light source 13 may be a continuous region, or as shown in part (f) of Fig. 10, the region of wavelengths of light irradiated by light source 13 may be a discrete region.

[0069] The light source 13 irradiates light including a plurality of wavelength regions included in the sensitive wavelength range of the image sensor 12, thereby enabling the image sensor 12 to capture an image in a plurality of wavelength regions. Furthermore, the image sensor 12 can efficiently capture images in a plurality of wavelength regions dispersed by the optical member 11 with a single illumination source, and can capture the images without being subject to temporal fluctuations.

[0070] [Method for Calculating the Luminance of a Captured Image] FIG. 11 is a diagram showing a method for calculating the luminance of an image in multiple wavelength regions. Part (a) of FIG. 11 shows the wavelength region of light that can be captured by the image sensor 12 (i.e., the sensitive wavelength region). Wavelength components that overlap with the wavelength region of the illumination light in part (c) of FIG. 11 , which will be described later, are indicated by horizontal stripes. Part (b) of FIG. 11 shows Regions 1 and 2, which are wavelength regions dispersed by the optical member 11. Wavelength components that overlap with the wavelength region of the illumination light in part (c) of FIG. 11 , which will be described later, are indicated by diagonal stripes. In part (b) of FIG. 11 , the wavelength components of the illumination light in Region 1 are indicated by diagonal stripes that slope downward to the right, and the wavelength components of the illumination light in Region 2 are indicated by diagonal stripes that slope upward to the right. Part (c) of FIG. 11 shows the wavelength region of the illumination light irradiated by the light source 13 (i.e., the wavelength components of the illumination light). Part (d) of FIG. 11 shows the luminance of an image captured in Region 1 and the luminance of an image captured in Region 2. In part (d) of Figure 11, the brightness of the image captured in area 1 is represented by a figure with a diagonal stripe pattern sloping downward to the right, and the brightness of the image captured in area 2 is represented by a figure with a diagonal stripe pattern sloping upward to the right.

[0071] 11A is a line indicating the sensitivity of the image sensor 12. The dotted line also represents the wavelength function of the sensitivity (i.e., spectral sensitivity) of the image sensor 12 as F1(λ).

[0072] The dashed lines shown in part (b) of Figure 11 are lines indicating the wavelength region of region 1 dispersed by the optical element 11, and the solid lines are lines indicating the wavelength region of region 2 dispersed by the optical element 11. The dashed and solid lines are expressed as a function F2i(λ) that describes the wavelength dependency of the spectral intensity (in other words, the spectral characteristics) in each of the multiple wavelength regions dispersed by the optical element 11. Note that i is a number assigned to each of the multiple wavelength regions and is expressed as i = 1, 2, ..., n. Furthermore, n is an integer equal to or greater than 2. In other words, the dotted line indicating region 1 is expressed as F21(λ), ​​and the dashed line indicating region 2 is expressed as F22(λ).

[0073] 11(c) is a line indicating the wavelength region of the illumination light emitted from the light source 13. The solid line is also expressed as a function F3(λ) that describes the wavelength dependency of the intensity of the illumination light (i.e., the illumination intensity).

[0074] The figure showing the brightness of the image in the multiple wavelength ranges shown in part (d) of Figure 11 is calculated by the following formula (1) written using the above-mentioned functions F1(λ), ​​F2i(λ), and F3(λ). Note that the range of each integral formula is the range in which the pattern of each function is drawn.

[0075]

[0076] The figure with diagonal stripes sloping downward to the right, which indicates the brightness of the image captured of region 1, is calculated by an integral formula F2i(λ) where i has a value of 1. Also, the figure with diagonal stripes sloping upward to the right, which indicates the brightness of the image captured of region 2, is calculated by an integral formula F2i(λ) where i has a value of 2.

[0077] The optical element 11 separates the reflected light from the subject 2 based on wavelength regions so that the above-described formula (1) holds. This allows the imaging system 1 to reduce the difference in image brightness between images of each wavelength. This improves the accuracy of image alignment in the imaging system 1.

[0078] The luminance values ​​of the images of each wavelength calculated by the above formula (1) do not necessarily have to be the same value. For example, the imaging system 1 may consider that the above formula (1) holds when the error calculated by the luminance values ​​of the images of each wavelength is within 10%.

[0079] [Image Example] Next, an image 121 captured by the image sensor 12 will be described with reference to Fig. 12. Fig. 12 is a diagram showing an example of an image 121 captured by the image sensor 12. Fig. 12 is a diagram showing an example of an image 121 that can be aligned by the processing circuitry 14.

[0080] Part (a) of FIG. 12 shows a first example in which the image sensor 12 captures an image at one wavelength. Part (b) of FIG. 12 shows a second example in which the image sensor 12 captures an image at one wavelength. Part (c) of FIG. 12 shows a third example in which the image sensor 12 captures an image at one wavelength. Part (d) of FIG. 12 shows a first example in which the image sensor 12 captures images at two wavelengths. Part (e) of FIG. 12 shows a second example in which the image sensor 12 captures images at two wavelengths. Part (f) of FIG. 12 shows a third example in which the image sensor 12 captures images at two wavelengths. Note that in parts (a) and (d) of FIG. 12, no flare or blurring occurs in the image 121, and the boundary between the peripheral region 122 and the illuminated region 123 is clear. In parts (b) and (e) of Figure 12, slight flare or blurring occurs in the image 121, and the boundary between the peripheral region 122 and the illuminated region 123 is not very clear. In parts (c) and (f) of Figure 12, flare or blurring occurs in the image 121, and the boundary between the peripheral region 122 and the illuminated region 123 is even less clear. Note that flare refers to a part or the entire peripheral region 122 that is blurred white. Furthermore, blurring refers to the image 121 being out of focus.

[0081] As shown in parts (d) and (f) of Figure 12 , in order to enhance the independence of the images of each wavelength when the boundary between the peripheral region 122 and the illumination region 123 is unclear, the image sensor 12 must capture images of each wavelength with the illumination region 123 spatially separated. Independence of the images of each wavelength means that the processing circuit 14 can distinguish between the images of each wavelength. This limits the range of images of each wavelength that can be used to extract feature points on the image sensor 12. Note that when the boundary between the peripheral region 122 and the illumination region 123 is unclear as in part (f) of Figure 12 , bringing the images of each wavelength closer together on the image sensor 12 as in part (d) of Figure 12 reduces the independence of the images of each wavelength. As a result, the processing circuit 14 cannot determine which wavelength image each feature point belongs to, resulting in reduced alignment accuracy.

[0082] The image 121 to be aligned by the processing circuit 14 may be an image 121 in which flare or blurring occurs, such as in part (c) of Figure 12 and part (f) of Figure 12, but is preferably an image 121 in which no flare or blurring occurs, such as in part (a) of Figure 12 and part (d) of Figure 12.

[0083] [Effects, etc.] As described above, the light source 13 emits illumination light so as to illuminate only a portion of the subject 2, thereby reducing the illumination area 123 formed on the image sensor 12. Furthermore, the optical element 11 separates the reflected light from the subject 2 based on wavelength regions so that the illumination areas 123 do not overlap on the image sensor 12. Therefore, the image sensor 12 can capture multiple images, each corresponding to a different wavelength region among multiple wavelength regions, without overlapping the illumination areas 123. Therefore, the processing circuit 14 can align images based on feature points in each of the multiple images. Therefore, the image sensor 1 captures independent images for each wavelength, thereby obtaining feature points in the image for each wavelength. This improves the accuracy of image alignment in the image sensor 1. Furthermore, for the same reason, the image alignment method also improves the accuracy of image alignment.

[0084] (Embodiment 2) An imaging system according to embodiment 2 differs from imaging system 1 according to embodiment 1 in that a light source illuminates the entire subject, and that the subject has a peripheral area and a target marker. The following description will focus on the differences from imaging system 1 according to embodiment 1, and will omit a description of the same points.

[0085] 13 is a block diagram showing the configuration of an imaging system 1a according to embodiment 2. The imaging system 1a is the same as the imaging system 1 according to embodiment 1 in that it includes an optical member 11, an imaging element 12, and a processing circuit 14. The imaging system 1a differs from the imaging system 1 according to embodiment 1 in that it does not include a light source 13a. Note that the light source 13a may be the same as or different from the light source 13 described in embodiment 1.

[0086] As shown in FIG. 13, a subject 2 a has a peripheral region 21 and a target marker 22 .

[0087] The peripheral region 21 is a region that covers the subject 2a in order to limit the illumination region 123 of the image 121, and has the characteristic of having low reflectance that hardly reflects the illumination light from the light source 13a. Note that "the reflectance of the peripheral region 21 is low" means that the peripheral region 21 reflects less illumination light than the target marker 22.

[0088] The reflectance of the peripheral region 21 is, for example, 50% or less. The reflectance of the peripheral region 21 is preferably 30% or less. The reflectance of the peripheral region 21 is more preferably 10% or less.

[0089] The target marker 22 is a region of the subject 2a that is not covered by the peripheral region 21, and has the characteristic of having high reflectivity for reflecting the illumination light from the light source 13a. Note that "the target marker 22 has high reflectivity" means that the target marker 22 reflects the illumination light more than the peripheral region 21.

[0090] The reflectance of the target marker 22 is, for example, 50% or more. The reflectance of the target marker 22 is preferably 70% or more. The reflectance of the target marker 22 is more preferably 90% or more.

[0091] [Alignment Operation] FIG. 14 is a flowchart showing the alignment operation of the imaging system 1a according to the second embodiment.

[0092] First, the light source 13a emits illumination light that illuminates the subject 2a (step S11).

[0093] Next, the optical member 11 separates the reflected light from the subject 2a into a plurality of light components based on wavelength regions, and forms a plurality of images, each of which is an image of the subject 2a based on a corresponding light component among the plurality of light components, on the image sensor 12 so that they do not overlap (step S12). Note that "so that they do not overlap on the image sensor 12" means that the target markers 22 in the plurality of images do not overlap on the image sensor 12.

[0094] Next, the image sensor 12 captures each of the multiple images (step S13).

[0095] Then, the processing circuitry 14 aligns the images based on feature points extracted from the target markers 22 in each of the multiple images (step S14).

[0096] Note that step S14 is the same as step S4 in the alignment operation performed by the imaging system 1 according to the first embodiment, and therefore a description thereof will be omitted.

[0097] Fig. 15 is a schematic diagram illustrating the operation of the imaging system 1a according to the second embodiment. Fig. 15 also illustrates the operations from step S11 to step S13 in Fig. 14. Fig. 15 is a schematic diagram illustrating the optical member 11 separating the light reflected from the subject 2a into two light components based on wavelength, and the imaging element 12 capturing an image of the two light components having different wavelengths. Note that an image 121 is an image captured by the imaging element 12. Furthermore, the lens 15 is a lens that converges the light separated by the optical member 11 and forms the image 121 on the imaging element 12.

[0098] As shown in FIG. 15 , the optical element 11 spatially shifts the images of the two wavelengths on the imaging surface of the imaging element 12 to form an image 121 so that the target marker 22 in each of the images of the two wavelengths does not overlap. In other words, the optical element 11 spatially shifts the images of the two wavelengths on the imaging surface of the imaging element 12 to form an image 121 so that the target marker 22 in one image of the wavelength overlaps the peripheral region 21 in the image of the other wavelength. This allows the imaging system 1a to suppress overlapping of the images of each wavelength and to enhance the independence of the images of each wavelength and the independence of the feature points in each image. Note that in FIG. 15 , the peripheral region 21 is the area of ​​the subject 2a that is painted black and surrounds the target marker 22, and the target marker 22 is the area of ​​the subject 2a that is painted white.

[0099] Because the subject 2a has a peripheral region 21 and a target marker 22, it is possible to make the target marker 22 small when it is imaged on the image sensor 12. Furthermore, because the optical member 11 disperses the light so that the target markers 22 do not overlap on the image sensor 12, the image sensor 12 can capture images of the target markers 22 in the multiple wavelength regions without overlapping each other. Therefore, the processing circuit 14 can align the images based on feature points in each of the images in the multiple wavelength regions.

[0100] [Image Example] Next, an example in which the optical member 11 forms an image 121 by spatially shifting two images based on light components of different wavelengths on the imaging surface of the image sensor 12 will be described with reference to FIGS. 16 and 17 . FIG. 16 is a diagram illustrating an example in which the optical member 11 forms an image 121 by spatially shifting two images based on light components of different wavelengths on the imaging surface of the image sensor 12 so that the target markers 22 in the two images do not overlap. FIG. 17 is a diagram illustrating an example in which the optical member 11 forms an image 121 by spatially shifting two images based on light components of different wavelengths on the imaging surface of the image sensor 12 so that the target markers 22 in the two images overlap. Note that FIG. 17 is a diagram illustrating an image 121 captured by a comparative example of the imaging system 1a according to the second embodiment.

[0101] Part (a) of Fig. 16 shows an example in which the imaging element 12 captures an image of only one wavelength. Part (b) of Fig. 16 shows an example in which the imaging element 12 captures an image of only the other wavelength. Part (c) of Fig. 16 shows an example in which the imaging element 12 captures images of two wavelengths. Note that in parts (a) to (c) of Fig. 16, the area surrounded by dashed lines indicates the range that the imaging element 12 can capture.

[0102] The optical element 11 spatially shifts each of the multiple images on the imaging surface of the image sensor 12 to form an image 121 so that the target markers 22 in the two images based on light components of different wavelengths do not overlap. For example, as shown in part (a) of FIG. 16 , the image sensor 12 captures the target marker 22 in one image at one wavelength to the left of the area surrounded by the dashed line. Also, as shown in part (b) of FIG. 16 , the image sensor 12 captures the target marker 22 in the image at the other wavelength to the right of the area surrounded by the dashed line. Because the target markers 22 in the images at each wavelength are spatially shifted to avoid overlapping, the image sensor 12 can obtain images at each wavelength that are highly independent, as shown in part (c) of FIG. 16 . This allows the processing circuit 14 to determine which wavelength image contains the feature point included in the target marker 22.

[0103] Part (a) of Fig. 17 shows an example in which the imaging element 12 captures an image of only one wavelength. Part (b) of Fig. 17 shows an example in which the imaging element 12 captures an image of only the other wavelength. Part (c) of Fig. 17 is a diagram showing an example in which the imaging element 12 captures images of two wavelengths. Note that in parts (a) to (c) of Fig. 17, the area surrounded by a dashed line indicates the range that the imaging element 12 can capture.

[0104] The optical element 11 spatially shifts each of the multiple images on the imaging surface of the image sensor 12 so that the target markers 22 in each of the two images based on light components of different wavelengths overlap, forming an image 121. For example, as shown in part (a) of FIG. 17 , the image sensor 12 captures the target marker 22 in one image at one wavelength to the left of the area surrounded by the dashed line. Also, as shown in part (b) of FIG. 17 , the image sensor 12 captures the target marker 22 in the image at the other wavelength to the right of the area surrounded by the dashed line. Because the target markers 22 are spatially shifted and focused to overlap, the image sensor 12 cannot obtain images of each wavelength that are highly independent, as shown in part (c) of FIG. 17 . As a result, the processing circuit 14 cannot determine which wavelength the feature points included in the target marker 22 are included in. In such a case, the accuracy of image alignment between the images decreases.

[0105] 17A and 17B, each of the two images based on light components of different wavelengths is an image in which the blank space within the region surrounded by the dashed line by the peripheral region 21 is not filled in with respect to the direction in which the optical member 11 spatially shifts the other image. In such a case, when the imaging element 12 captures two images based on light components of different wavelengths, the contrast of the image 121 deteriorates in the region where the two images do not overlap, as shown in 17C.

[0106] [Wavelength Range] Next, the relationship between the sensitive wavelength range of the image sensor 12 and the wavelength range of light reflected by the peripheral region 21 or the wavelength range of light reflected by the target marker 22 will be described with reference to Fig. 18. Fig. 18 is a diagram showing wavelength ranges.

[0107] Part (a) of Figure 18 shows the sensitive wavelength range of the image sensor 12. Part (b) of Figure 18 shows two wavelength ranges into which the optical member 11 disperses reflected light. Part (c) of Figure 18 shows the intersection of the wavelength ranges shown in parts (a) and (b) of Figure 18. Part (d) of Figure 18 shows the wavelength range of light reflected by the peripheral region 21 with low reflectivity. Part (e) of Figure 18 shows the wavelength range of light reflected by the target marker 22 with high reflectivity. Note that parts (a) to (e) of Figure 18 indicate light with longer wavelengths toward the right and light with shorter wavelengths toward the left. In other words, light in region 1 shown in part (b) of Figure 18 is light in the short wavelength range, and light in region 2 is light in the long wavelength range.

[0108] 18A and 18B, the regions 1 and 2 are included in the sensitive wavelength range of the image sensor 12. This allows the image sensor 12 to capture images of the regions 1 and 2.

[0109] The region shown in part (c) of Fig. 18 is a region that represents the intersection of the wavelength regions shown in parts (a) and (b) of Fig. 18. That is, wavelength region A is a region that represents the intersection of the sensitive wavelength region of the image sensor 12 and region 1, and wavelength region B is a region that represents the intersection of the sensitive wavelength region of the image sensor 12 and region 2.

[0110] 18, the range of wavelengths of light reflected by peripheral region 21 with low reflectance has a wider range of wavelengths than wavelength region A and wavelength region B. In other words, peripheral region 21 reflects illumination light with low reflectance at all wavelengths in wavelength region A and wavelength region B. This allows peripheral region 21 to limit the illuminated area of ​​subject 2a.

[0111] As shown in part (c) and part (e) of Figure 18, the wavelength region of light reflected by the target marker 22 with high reflectivity is included in both wavelength region A and wavelength region B. In other words, the target marker 22 reflects illumination light with high reflectivity in a portion of wavelength bands in wavelength region A and wavelength region B. This allows the imaging element 12 to capture images of the target marker 22 in multiple images corresponding to mutually different wavelength regions. Note that the wavelength region of light reflected by the target marker 22 with high reflectivity may be a region wider than wavelength region A and wavelength region B.

[0112] This allows the imaging system 1a to increase the independence of images of each wavelength and the independence of feature points in each image.

[0113] [Modifications of Target Markers] In the above, an example has been described in which the processing circuitry 14 performs alignment based on feature points extracted from the target marker 22. Below, an example will be described in which the processing circuitry 14 performs alignment based on markers included in the target marker 22, rather than feature points extracted from the target marker 22. Fig. 19 is a diagram showing examples of markers. Part (a) of Fig. 19 shows a first example of a marker. Part (b) of Fig. 19 shows a second example of a marker. Part (c) of Fig. 19 shows a third example of a marker.

[0114] The marker shown in part (a) of Fig. 19 is a grid-shaped marker. The marker shown in part (b) of Fig. 19 is a checkerboard marker. The marker shown in part (c) of Fig. 19 is a ChArUco marker that combines an ArUco marker with an embedded code used in AR (Augmented Reality) and a checkerboard marker. Note that the type of marker may be other than the example markers shown in parts (a) to (c) of Fig. 19.

[0115] The processing circuitry 14 performs alignment based on the markers included in the target markers 22. The method by which the processing circuitry 14 performs alignment based on the markers is the same as the method by which the processing circuitry 14 performs alignment based on feature points in the first embodiment, and therefore a description thereof will be omitted.

[0116] [Comparative Example] Next, a comparative example will be described. Fig. 20 is a schematic diagram showing the operation of an imaging system of the comparative example.

[0117] 20 , the imaging system of the comparative example does not include a light source 13 that partially illuminates the subject 2, and the subject 2 does not include a target marker 22 with high reflectivity and a peripheral area 21 with low reflectivity. As a result, the imaging system of the comparative example cannot limit the illuminated area of ​​the subject 2.

[0118] Fig. 21 is an enlarged view of the image 121 shown in Fig. 20. The image 121 is an image obtained by imaging two images corresponding to different wavelengths. The arrows indicate characteristic points. Each arrow indicates a corner of a figure.

[0119] 21 , the image 121 does not have the peripheral regions 21 of the two images corresponding to different wavelengths, and therefore the target markers 22 of the two images corresponding to different wavelengths are superimposed and imaged on the imaging surface of the image sensor 12. As a result, the processing circuit 14 cannot determine which image of which wavelength the feature point included in the target marker 22 is included. In such a case, the accuracy of alignment between the images decreases.

[0120] [Effects, etc.] As described above, since the subject 2a has a peripheral region 21 and a target marker 22, the target marker 22 formed on the image sensor 12 can be made smaller. Furthermore, since the optical member 11 disperses light so that the target markers 22 do not overlap on the image sensor 12, the image sensor 12 can capture multiple images corresponding to different wavelength regions without the target markers 22 overlapping each other. This allows the processing circuit 14 to align the images based on feature points in each of the multiple images. Therefore, since the image sensor 1a captures independent images for each wavelength, feature points in the images for each wavelength can be obtained. This improves the accuracy of image alignment in the image sensor 1a. Furthermore, for the same reason, the accuracy of image alignment is also improved in the image alignment method.

[0121] (Embodiment 3) An imaging system according to embodiment 3 differs from imaging system 1 according to embodiment 1 and imaging system 1a according to embodiment 2 in that the imaging system disperses reflected light from a subject based on the region of the polarization angle. The following description will focus on the differences from the imaging systems according to embodiments 1 and 2, and will omit a description of the same points.

[0122] The configuration of the imaging system according to the third embodiment is basically the same as the configuration of the imaging system 1 described using Fig. 1 or the configuration of the imaging system 1a described using Fig. 13. However, the difference is that while the optical member 11 in the imaging systems 1 and 1a splits the light reflected from the subject into two wavelengths, the optical member 11 in the imaging system according to the third embodiment splits the light reflected from the subject into two polarized light beams.

[0123] 22 is a diagram showing an example of an optical element that separates reflected light from a subject into two polarized light beams in an imaging system according to embodiment 3. In the optical element 11 according to embodiment 3, the two dichroic mirrors 116 in the configuration shown in FIG. 9 are replaced with a polarizing beam splitter 136. The polarizing beam splitter 136 separates incident light into linearly polarized light beams that are orthogonal to each other, and transmits one polarized light beam and reflects the other polarized light beam.

[0124] First, the first polarizing beam splitter 136 splits the light reflected from the subject into two linearly polarized light beams that are orthogonal to each other. One total reflection mirror 113 totally reflects the linearly polarized light reflected by the first polarizing beam splitter 136, and the other total reflection mirror 113 totally reflects the linearly polarized light beam that is transmitted through the first polarizing beam splitter 136. The linearly polarized light beams that are orthogonal to each other and that are reflected by the two total reflection mirrors 113 are incident on the second polarizing beam splitter 136. Then, the lens 15 converges each of the linearly polarized light beams that are orthogonal to each other to form an image on the image sensor 12.

[0125] This allows the optical member 11 to separate the reflected light from the subject 2 into two polarized lights.

[0126] [Polarization Angle Region of the Light Source] The above describes the splitting of light by the optical element 11. However, if the image sensor 12 is not sensitive to the polarized light split by the optical element 11, the image sensor 12 cannot capture an image of the polarized light. Therefore, the relationship between the sensitivity of the image sensor 12 to polarized light and the polarization angle region of the light emitted by the light source will be described using FIG. 23 . FIG. 23 is a diagram showing the polarization angle region. Here, the polarization angle region refers to a polarization angle within at least one acceptable angle range. In the case of an image sensor that has high sensitivity to linearly polarized light in a certain direction, the component of the incident linearly polarized light in that direction is captured. In other words, parallel linearly polarized light (relative angle 0 degrees) has the greatest sensitivity, and orthogonal linearly polarized light (relative angle 90 degrees) has the least sensitivity. The sensitivity to linearly polarized light between relative angles 0 degrees and 90 degrees depends on a cosine function.

[0127] Part (a) of Figure 23 shows the polarization sensitivity dependence of the image sensor 12. In this case, the sensitivity of the image sensor 12 is independent of polarization, and the image sensor 12 is sensitive to all polarization directions. Part (b) of Figure 23 shows the polarization angle range of the two polarized lights obtained by separating the reflected light by the optical member 11. Part (c) of Figure 23 shows a first example of the polarization angle range of the light emitted by the light source 13. Part (d) of Figure 23 shows a second example of the polarization angle range of the light emitted by the light source 13. Part (e) of Figure 23 shows a third example of the polarization angle range of the light emitted by the light source 13. Note that the arrows in parts (a) to (e) of Figure 23 indicate light whose main component is linearly polarized light in the direction indicated by the arrow, and symbols containing multiple arrows within a circle indicate light that includes all polarization components or is sensitive to all polarization components.

[0128] 23A and 23B, when the sensitivity of the image sensor 12 is independent of polarization, the image sensor 12 has sensitivity to the polarization regions 1 and 2. This allows the image sensor 12 to capture images of the polarization regions 1 and 2.

[0129] In order for polarization region 1 and polarization region 2 to be in a relationship that produces a significant signal for image sensor 12, the region of polarization angle of the light before it is dispersed by optical element 11, i.e., the region of polarization angle of the light with which light source 13 illuminates subject 2, must include components of polarization region 1 and polarization region 2.

[0130] As shown in parts (c) to (e) of Fig. 23 , the polarization component of the light irradiated by light source 13 needs to have components in the polarization directions of polarization regions 1 and 2. That is, the light irradiated by light source 13 may be unpolarized light as shown in part (c) of Fig. 23 , or may be linearly polarized light that is not orthogonal to either polarization region 1 or 2 as shown in part (d) of Fig. 23 . Alternatively, when polarization regions 1 and 2 are polarized light that is orthogonal to each other as shown in part (e) of Fig. 23 , the light irradiated by light source 13 may be illumination light whose main component is linearly polarized light that is at a relative angle of 45 degrees to both polarization regions 1 and 2.

[0131] In the above example, the image sensor 12 has sensitivity that is independent of polarization. On the other hand, if the sensitivity of the image sensor 12 depends on polarization, the polarized light dispersed by the optical element 11 is in a polarization angle range to which the image sensor 12 is sensitive, allowing the image sensor 12 to capture images of multiple polarized lights. If all of the polarized light dispersed by the optical element 11 is linearly polarized, the relative angles between the polarization angle of the light source 13 and polarization region 1 and polarization region 2 dispersed by the optical element 11 are not orthogonal, and the polarization directions of polarization regions 1 and 2 are not orthogonal to the polarization direction on which the image sensor 12 has polarization sensitivity dependence.

[0132] [Method for calculating the luminance of a captured image] Fig. 24 is a diagram showing a method for calculating the luminance of an image of multiple polarized lights. Part (a) of Fig. 24 shows the region of polarization angles of light that can be captured by the image sensor 12 (i.e., the polarization dependency of spectral sensitivity). Part (b) of Fig. 24 shows Region 1 and Region 2, which are the regions of polarization angles of polarized light dispersed by the optical member 11. Part (c) of Fig. 24 shows the region of polarization angles of illumination light emitted by the light source 13 (i.e., the polarization angle components of the illumination light).

[0133] 24A is a line indicating the sensitivity of the image sensor 12. The dotted line also represents the polarization angle function of the sensitivity of the image sensor 12 as F1(ξ).

[0134] The dashed lines shown in part (b) of Figure 24 are lines indicating the polarization angle region of region 1 where the optical element 11 disperses light, and the solid lines are lines indicating the polarization angle region of region 2 where the optical element 11 disperses light. The dashed and solid lines are expressed as a function F2i(ξ) that describes the polarization dependence of the spectral intensity (in other words, the spectral characteristics) in each of the multiple polarization angle regions where the optical element 11 disperses light. Note that i is a number assigned to each of the multiple polarization angle regions and is expressed as i = 1, 2, ..., n. Furthermore, n is an integer equal to or greater than 2. In other words, the dotted line indicating region 1 is expressed as F21(ξ), and the dashed line indicating region 2 is expressed as F22(ξ).

[0135] The solid line shown in part (c) of Fig. 24 is a line indicating the region of the polarization angle of the light source 13. The solid line is also expressed as a function F3(ξ) that describes the polarization dependency of the intensity of the light source 13 that emits illumination light (i.e., the illumination intensity).

[0136] The expression showing the brightness of the images of the multiple polarizations in the image sensor 12 is calculated by the following expression (2) written using the above-mentioned functions F1(ξ), F2i(ξ), and F3(ξ).

[0137] The brightness of the image captured in region 1 is calculated by the integral formula F2i(ξ) where the value of i is 1. The brightness of the image captured in region 2 is calculated by the integral formula F2i(ξ) where the value of i is 2.

[0138] The light source 13 and the optical member 11 split the reflected light from the subject 2 into a plurality of polarized lights so that the above-mentioned formula (2) is satisfied. This allows the imaging system 1 to reduce the difference in image brightness between images of each polarized light. Therefore, the imaging system 1 improves the accuracy of alignment between images.

[0139] The brightness values ​​of the images of each polarization calculated by the above formula (2) do not necessarily have to be the same value. For example, the imaging system 1 may consider that the above formula (2) holds when the error calculated by the brightness values ​​of the images of each polarization is within 10%.

[0140] Furthermore, the target marker 22 in the imaging system 1a may have high reflectivity in the range of the intersection of the polarization angle region of the light source 13, regions 1 and 2 of polarized light dispersed by the optical element 11, and the polarization angle region to which the imaging element 12 is sensitive, and the surrounding region may have low reflectivity throughout the intersection of the polarization angle region of the light source 13, regions 1 and 2 of polarized light dispersed by the optical element 11, and the polarization angle region to which the imaging element 12 is sensitive.

[0141] (Modifications) While imaging systems according to one or more aspects have been described above based on the above-described embodiments, the present disclosure is not limited to the above-described embodiments. As long as they do not deviate from the spirit of the present disclosure, various modifications conceivable by those skilled in the art to the above-described embodiments and configurations constructed by combining components of different embodiments may also be included within the scope of one or more aspects.

[0142] In the above embodiments, the processing circuit may be configured with dedicated hardware, or may be realized by executing a software program suitable for the processing circuit. The processing circuit may also be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0143] Furthermore, some or all of the functions of the processing circuitry according to the above embodiments may be realized by a processor such as a CPU executing a program.

[0144] Some or all of the components constituting each of the above devices may be configured as an IC card or a standalone module that can be attached to or detached from each device. The IC card or module is a computer system composed of a microprocessor, ROM, RAM, etc. The IC card or module may include a super multi-function LSI. The IC card or module achieves its functions when the microprocessor operates in accordance with a computer program. The IC card or module may be tamper-resistant.

[0145] [Additional Notes] The above description of the embodiments discloses the following techniques.

[0146] (Technology 1) An imaging system comprising: an image sensor; an optical element that splits light from a subject into a plurality of light components and forms a plurality of images, each of which is an image of the subject based on a corresponding light component of the plurality of light components, on an imaging surface of the image sensor at spatially shifted positions; a processing circuit that aligns the plurality of images based on a feature point in each of the plurality of images; and a light source that emits illumination light to illuminate a portion of the subject, wherein each of the plurality of light components has a wavelength included in a corresponding wavelength range of a plurality of wavelengths, or has a polarization angle included in a corresponding polarization angle range of a plurality of polarization angles, the optical element splits the light from the subject based on the plurality of wavelength ranges or the plurality of polarization angle ranges, and each of the plurality of images includes an illumination area corresponding to the portion of the subject illuminated by the light source, the plurality of images including a first image and a second image, the illumination area in the first image does not overlap with the illumination area in the second image, and the feature point is extracted from the illumination area in each of the plurality of images.

[0147] With this configuration, the light source emits illumination light so as to illuminate only a portion of the subject, thereby reducing the illumination area formed on the image sensor. Furthermore, the optical element separates the reflected light from the subject based on multiple wavelength regions or polarization angle regions so that the illumination areas on the image sensor do not overlap. Therefore, the image sensor can capture multiple images corresponding to different wavelength regions or polarization angle regions without overlapping the illumination areas. Therefore, the processing circuit can align the images based on feature points in each of the multiple images. Therefore, the imaging system captures independent images for each wavelength, thereby obtaining feature points in the images for each wavelength. This improves the accuracy of the image alignment in the imaging system.

[0148] (Technology 2) The imaging system according to Technology 1, wherein the illumination light includes wavelength components included in each of the plurality of wavelength regions.

[0149] With this configuration, the imaging system efficiently acquires images corresponding to the multiple wavelength regions dispersed by the optical element using a single illumination source, and can acquire the images without being affected by temporal fluctuations, thereby improving the accuracy of image alignment in the imaging system.

[0150] (Technology 3) The imaging system according to Technology 1 or 2, wherein the above formula (1) holds, where F1(λ) represents a wavelength function of the sensitivity of the imaging element, F2i(λ) represents a function describing the wavelength dependency of the spectral intensity in each of the plurality of wavelength regions, the plurality of wavelength regions including n wavelength regions, i is an integer from 1 to n, and F3(λ) represents a function describing the wavelength dependency of the intensity of the illumination light.

[0151] This configuration allows the imaging system to reduce the difference in brightness between images of each wavelength, thereby improving the accuracy of alignment between images.

[0152] (Technology 4) An imaging system comprising: an image sensor; an optical element that splits light from a subject into a plurality of light components and forms a plurality of images, each of which is an image of the subject based on a corresponding light component among the plurality of light components, on an imaging surface of the image sensor at spatially shifted positions; and a processing circuit that aligns the plurality of images based on feature points in each of the plurality of images, wherein each of the plurality of light components has a wavelength included in a corresponding wavelength range among a plurality of wavelength ranges, or has a polarization angle included in a corresponding polarization angle range among a plurality of polarization angle ranges; the optical element splits the light from the subject based on the plurality of wavelength ranges or the plurality of polarization angle ranges; the subject includes a target marker and a surrounding area, wherein a reflectance of the target marker is higher than a reflectance of the surrounding area; the plurality of images include a first image and a second image; the target marker in the first image does not overlap with the target marker in the second image; and the feature points are extracted from the target marker in each of the plurality of images.

[0153] With this configuration, since the subject has a peripheral region and a target marker, the target marker imaged on the imaging element can be made smaller. Furthermore, since the optical element separates the light so that the target markers do not overlap on the imaging element, the imaging element can capture multiple images without the target markers overlapping each other. This allows the processing circuit to align the images based on feature points in each of the multiple images. Therefore, since the imaging system captures independent images for each wavelength or polarization angle, feature points in each image can be obtained. This improves the accuracy of the imaging system's alignment between images.

[0154] (Technology 5) An imaging system described in Technology 4, wherein the reflectance of the surrounding area is lower than the reflectance of the target marker in all wavelength ranges of the intersection of the multiple wavelength ranges and a sensitive wavelength range, which is the range of wavelengths of light that can be received by the imaging element, and the reflectance of the target marker is higher than the reflectance of the surrounding area in at least a portion of the area where each of the multiple wavelength ranges overlaps with the sensitive wavelength range.

[0155] This configuration allows the imaging system to suppress overlapping of images of each wavelength and to increase the independence of the images of each wavelength and the independence of feature points in each image, thereby improving the accuracy of image registration in the imaging system.

[0156] (Technology 6) A method for aligning images, comprising: splitting light from a subject into a plurality of light components; forming a plurality of images, each of which is an image of the subject based on a corresponding light component among the plurality of light components, on an imaging surface of an image sensor at spatially shifted positions; aligning the plurality of images based on a feature point in each of the plurality of images; and emitting illumination light to illuminate a portion of the subject, wherein each of the plurality of light components has a wavelength included in a corresponding wavelength range among a plurality of wavelength ranges, or has a polarization angle included in a corresponding polarization angle range among a plurality of polarization angle ranges, the light from the subject is split based on the plurality of wavelength ranges or the plurality of polarization angle ranges, each of the plurality of images includes an illumination area corresponding to the portion of the subject illuminated by the light source, the plurality of images including a first image and a second image, the illumination area in the first image does not overlap with the illumination area in the second image, and the feature point is extracted from the illumination area in each of the plurality of images.

[0157] With this configuration, the light source emits illumination light so as to illuminate only a portion of the subject, thereby reducing the illumination area formed on the image sensor. Furthermore, the optical element separates the reflected light from the subject based on multiple wavelength regions or polarization angle regions so that the illumination areas on the image sensor do not overlap. This allows the image sensor to capture multiple images without overlapping the illumination areas in each of the multiple images. Therefore, the processing circuit can align the images based on feature points in each of the multiple images. Therefore, since the image alignment method captures independent images for each wavelength or polarization angle, feature points in each image can be obtained. This improves the accuracy of image alignment.

[0158] (Technology 7) A method for aligning images, comprising: splitting light from a subject into a plurality of light components; forming a plurality of images, each of which is an image of the subject based on a corresponding light component among the plurality of light components, on an imaging surface of an image sensor while spatially shifting the plurality of images; and aligning the plurality of images based on a feature point in each of the plurality of images, wherein each of the plurality of light components has a wavelength included in a corresponding wavelength range among a plurality of wavelength ranges, or has a polarization angle included in a corresponding polarization angle range among a plurality of polarization angle ranges; the light from the subject is split based on the plurality of wavelength ranges or the plurality of polarization angle ranges; the subject includes a target marker and a surrounding area, the reflectance of the target marker being higher than the reflectance of the surrounding area; the plurality of images include a first image and a second image; the target marker in the first image does not overlap with the target marker in the second image; and the feature point is extracted from the target marker in each of the plurality of images.

[0159] With this configuration, since the subject has a peripheral region and a target marker, the target marker imaged on the image sensor can be made smaller. Furthermore, since the optical element separates the light so that the target markers do not overlap on the image sensor, the image sensor can capture multiple images without overlapping the target markers in each of the multiple images. This allows the processing circuit to align the images based on feature points in each of the multiple images. Therefore, the alignment method can suppress image overlap and increase the independence of each image and the independence of feature points in each image. This improves the accuracy of alignment between images.

[0160] (Technology 8) The imaging system according to Technology 1, wherein the illumination light includes polarization components included in each of the regions of the plurality of polarization angles.

[0161] (Technology 9) The imaging system according to Technology 1 or 8, wherein the above formula (2) holds, F1(ξ) represents a function of polarization dependency of the sensitivity of the imaging element, F2i(ξ) represents a function describing the polarization dependency of the spectral intensity in each of the multiple polarization angle regions, the multiple polarization angle regions including n polarization angle regions, i is an integer from 1 to n, and F3(ξ) represents a function describing the polarization dependency of the intensity of the illumination light.

[0162] (Technology 10) An imaging system described in Technology 4, wherein the reflectivity of the peripheral region is lower than the reflectivity of the target marker in all polarization angle regions of the intersection of the region of the multiple polarization angles and a sensitive polarization region, which is a region of polarization of light that the imaging element can receive, and the reflectivity of the target marker is higher than the reflectivity of the peripheral region in at least a portion of the area where each of the region of the multiple polarization angles overlaps with the sensitive polarization region.

[0163] An imaging system according to the present disclosure is useful, for example, as a system for aligning images in multiple wavelength regions.

[0164] REFERENCE SIGNS LIST 1, 1a Imaging system 2, 2a Subject 11 Optical member 12 Imaging element 13, 13a Light source 14 Processing circuit 15 Lens 21 Peripheral area 22 Target marker 111 Dielectric multilayer wavelength prism 111a Dielectric multilayer wavelength filter 112 Prism 113 Total reflection mirror 114 Half mirror 115 Bandpass filter 116 Dichroic mirror 121 Image 122 Peripheral area 123 Illumination area 131 Illumination device 132 Aperture 133 Mask 134 Illumination lens 136 Polarizing beam splitter

Claims

1. An imaging system comprising: an image sensor; an optical element that disperses light from a subject into a plurality of light components and forms a plurality of images, each of which is an image of the subject based on a corresponding light component among the plurality of light components, on an imaging surface of the image sensor with the images spatially shifted; a processing circuit that aligns the plurality of images based on feature points in each of the plurality of images; and a light source that emits illumination light to illuminate a portion of the subject, wherein each of the plurality of light components has a wavelength included in a corresponding wavelength range among a plurality of wavelength ranges, or has a polarization angle included in a corresponding polarization angle range among a plurality of polarization angle ranges, the optical element disperses the light from the subject based on the plurality of wavelength ranges or the plurality of polarization angle ranges, each of the plurality of images includes an illumination area corresponding to the portion of the subject illuminated by the light source, the plurality of images include a first image and a second image, the illumination area in the first image does not overlap with the illumination area in the second image, and the feature points are extracted from the illumination area in each of the plurality of images.

2. The imaging system according to claim 1, wherein the illumination light includes wavelength components included in each of the multiple wavelength ranges.

3. The imaging system according to claim 1 or 2, wherein the following formula (1) is established, F1(λ) represents a wavelength function of the sensitivity of the imaging element, F2i(λ) represents a function describing the wavelength dependence of the spectral intensity in each of the multiple wavelength regions, the multiple wavelength regions including n wavelength regions, i being an integer from 1 to n, and F3(λ) represents a function describing the wavelength dependence of the intensity of the illumination light.

4. An imaging system comprising: an imaging element; an optical element that disperses light from a subject into a plurality of light components and forms a plurality of images, each of which is an image of the subject based on a corresponding light component among the plurality of light components, on an imaging surface of the imaging element with the images spatially shifted; and a processing circuit that aligns the plurality of images based on feature points in each of the plurality of images, wherein each of the plurality of light components has a wavelength included in a corresponding wavelength range among a plurality of wavelength ranges, or has a polarization angle included in a corresponding polarization angle range among a plurality of polarization angle ranges, the optical element disperses the light from the subject based on the plurality of wavelength ranges or the plurality of polarization angle ranges, the subject includes a target marker and a surrounding area, the reflectance of the target marker is higher than the reflectance of the surrounding area, the plurality of images include a first image and a second image, the target marker in the first image does not overlap with the target marker in the second image, and the feature points are extracted from the target marker in each of the plurality of images.

5. The imaging system described in claim 4, wherein the reflectance of the surrounding region is lower than the reflectance of the target marker in all wavelength regions of the intersection of the multiple wavelength regions and a sensitive wavelength region, which is a region of wavelengths of light that can be received by the imaging element, and the reflectance of the target marker is higher than the reflectance of the surrounding region in at least a portion of the overlapping area between each of the multiple wavelength regions and the sensitive wavelength region.

6. A method for aligning images, comprising: splitting light from a subject into a plurality of light components, and forming a plurality of images, each of which is an image of the subject based on a corresponding one of the plurality of light components, on an imaging surface of an image sensor with the images spatially shifted; aligning the plurality of images based on feature points in each of the plurality of images; and emitting illumination light to illuminate a portion of the subject, wherein each of the plurality of light components has a wavelength included in a corresponding wavelength range of a plurality of wavelength ranges, or has a polarization angle included in a corresponding polarization angle range of a plurality of polarization angle ranges, the light from the subject is split based on the plurality of wavelength ranges or the plurality of polarization angle ranges, each of the plurality of images includes an illumination area corresponding to the portion of the subject illuminated by the light source, the plurality of images include a first image and a second image, the illumination area in the first image does not overlap with the illumination area in the second image, and the feature points are extracted from the illumination area in each of the plurality of images.

7. A method for aligning images, comprising: splitting light from a subject into a plurality of light components; forming a plurality of images, each of which is an image of the subject based on a corresponding one of the plurality of light components, on an imaging surface of an image sensor with the images spatially shifted; and aligning the plurality of images based on feature points in each of the plurality of images, wherein each of the plurality of light components has a wavelength included in a corresponding wavelength range of a plurality of wavelength ranges, or has a polarization angle included in a corresponding polarization angle range of a plurality of polarization angle ranges, the light from the subject is split based on the plurality of wavelength ranges or the plurality of polarization angle ranges, the subject includes a target marker and a surrounding area, the reflectance of the target marker is higher than the reflectance of the surrounding area, the plurality of images include a first image and a second image, the target marker in the first image does not overlap with the target marker in the second image, and the feature points are extracted from the target marker in each of the plurality of images.

8. The imaging system according to claim 1, wherein the illumination light includes polarization components that fall within each of the multiple polarization angle regions.

9. The imaging system according to claim 1 or 8, wherein the following equation (2) holds, F1(ξ) represents a function of the polarization dependence of the sensitivity of the imaging element, F2i(ξ) represents a function describing the polarization dependence of the spectral intensity in each of the multiple polarization angle regions, the multiple polarization angle regions including n polarization angle regions, i being an integer from 1 to n, and F3(ξ) represents a function describing the polarization dependence of the intensity of the illumination light.

10. The imaging system of claim 4, wherein the reflectance of the surrounding region is lower than the reflectance of the target marker in all polarization angle regions of the intersection of the region of the multiple polarization angles and a sensitive polarization region, which is a region of polarization of light that can be received by the imaging element, and the reflectance of the target marker is higher than the reflectance of the surrounding region in at least a portion of the overlapping area between each of the regions of the multiple polarization angles and the sensitive polarization region.

Images