Imaging system and image alignment method

Abstract

To provide an imaging system etc. that enables highly accurate positioning.SOLUTION: An imaging system 1 includes an optical element 11 that separates light from a subject 2 into multiple wavelengths, at least one image sensor 12 that captures each of the multiple wavelength images 121, a light source 13 that illuminates the subject 2 with an active marker, and a processing circuit 14 that aligns the multiple wavelength images 121 on the basis of the active markers in the multiple wavelength images 121.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to an imaging system and the like.

Background Art

[0002] There is known an imaging system that acquires images in a plurality of wavelength regions and characteristics obtained from the images in the plurality of wavelength regions by splitting light from a subject with an optical member (see, for example, Patent Documents 1 and 2). Patent Document 1 discloses an imaging system that forms images in a plurality of wavelength regions on different imaging elements. Further, Patent Document 2 discloses an imaging system that forms images of different wavelengths on one imaging element, so that variations caused by mechanical vibrations or the like are the same among the respective images. In such imaging systems, when obtaining characteristics such as the temperature of a subject from a plurality of images, it is important to determine which point of each image corresponds to the same point of the subject. That is, the accuracy of alignment between the respective images becomes important.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the imaging system described in Patent Document 1, the correspondence relationship of images in regions of a plurality of wavelengths is liable to be affected by temporal variations due to mechanical vibrations or the like. Further, in the imaging system described in Patent Document 2, the deviation from the ideal optical state caused by tolerances due to processing or tolerances due to assembly causes a deviation from the correspondence between ideal images. And the deviation from the correspondence between these ideal images varies due to mechanical vibrations or imaging conditions, etc., and as the imaging device has higher resolution, it has come to have a greater impact. For these reasons, when aligning images in regions of a plurality of wavelengths, the alignment accuracy of the above-described imaging system may decrease.

[0005] Therefore, an object of the present disclosure is to provide an imaging system or the like that enables highly accurate alignment.

Means for Solving the Problems

[0006] To achieve the above object, an imaging system according to one embodiment of the present disclosure includes an optical member that splits light from a subject into regions of a plurality of wavelengths, at least one imaging device that images each of the images in the regions of the plurality of wavelengths, a light source that irradiates an active marker onto the subject, and a processing circuit that performs alignment between the images in the regions of the plurality of wavelengths based on the active marker in each of the images in the regions of the plurality of wavelengths.

[0007] To achieve the above object, an image alignment method according to one embodiment of the present disclosure is an image alignment method that splits light from a subject into regions of a plurality of wavelengths, images the images in the regions of the plurality of wavelengths with at least one imaging device, irradiates an active marker onto the subject, and performs alignment between the images in the regions of the plurality of wavelengths based on the active marker in the images in the regions of the plurality of wavelengths.

Advantages of the Invention

[0008] According to the present disclosure, an imaging system or the like that enables highly accurate alignment is provided.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11A

Figure 11B

Figure 12A

Figure 12B

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that each of the embodiments described below shows a specific example of the present disclosure. The numerical values, components, arrangement positions and connection forms of the components, steps, order of steps, display examples, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Therefore, among the components in the following embodiments, the components not described in the independent claims of the present disclosure are described as optional components. Also, each drawing is not necessarily drawn precisely. In each drawing, substantially the same configuration is denoted by the same reference numeral, and overlapping descriptions are omitted or simplified.

[0011] (Embodiment 1) [Configuration] FIG. 1 is a block diagram showing the configuration of an imaging system 1 according to Embodiment 1.

[0012] The imaging system 1 is a system that captures images in a plurality of wavelength regions and performs alignment between the images in the plurality of wavelength regions. As shown in FIG. 1, the imaging system 1 includes an optical member 11, a plurality of imaging elements 12, a light source 13, and a processing circuit 14.

[0013] The optical member 11 is a device that splits the reflected light from the subject 2 into a plurality of wavelength regions. Note that the specific configuration of the optical member 11 will be described later. Also, in this specification, each of the plurality of wavelength regions to be split may appropriately have a wavelength width (that is, a wavelength band).

[0014] The imaging element 12 is an electronic component that captures images in a plurality of wavelength regions split by the optical member 11. An active marker is captured in each image captured by the imaging element 12. An active marker is an image that can be distinguished from other feature points captured in each image. Also, in this figure, the imaging system 1 includes two imaging elements 12, but may include two or more imaging elements 12.

[0015] The light source 13 is a device that irradiates the subject 2 with an active marker. For example, a device having a high-brightness light source such as a laser light or an LED light source and a narrow wavelength width is used as a device that irradiates the active marker. For example, in Embodiment 1, the wavelength width of the active marker is 5 nm with respect to the central wavelength.

[0016] The processing circuit 14 is a system that performs alignment between images in a plurality of wavelength regions based on the active marker in each of the images in the plurality of wavelength regions captured by the imaging device 12. Note that alignment between images in a plurality of wavelength regions means correcting the misalignment of a plurality of images. The processing circuit 14 outputs information or an image obtained in the alignment between images in a plurality of wavelength regions. The processing circuit 14 performs alignment between images in a plurality of wavelength regions according to the following procedure.

[0017] First, the processing circuit 14 performs pattern matching on the images in the plurality of wavelength regions. Pattern matching is a method of detecting a pattern having the same shape as a previously stored pattern from a target image. By previously storing the shape of the active marker in the processing circuit 14, the processing circuit 14 can detect the active marker imaged in the images in the plurality of wavelength regions.

[0018] Next, the processing circuit 14 extracts the coordinates of the active marker detected by pattern matching. Coordinate extraction is a method of detecting the coordinates of an object from an edge or surrounding image that is important for specifying the shape of the object.

[0019] Then, the processing circuit 14 performs alignment between images in regions of a plurality of wavelengths using a method for obtaining a transformation matrix based on the coordinates obtained by coordinate extraction. The method for obtaining the transformation matrix is effective when the coordinates of the active markers in the images of each wavelength are different due to image distortion or the like, and alignment between the images cannot be performed. The processing circuit 14 can, for example, enlarge and reduce an image, shift an image, or rotate an image by performing pixel correction by projective transformation or the like. When the processing circuit 14 further detects the coordinates of four active markers, the shape of the image can be corrected from a parallelogram or trapezoid to a rectangle or square. Note that the processing circuit 14 may use sub-pixel accuracy for pixel correction. Further, the processing circuit 14 may use, for a method for obtaining a transformation matrix, a correlation coefficient such as a correlation coefficient or a normalized correlation coefficient between images in regions of a plurality of wavelengths, instead of coordinate extraction by pattern matching, or may use a program such as Scale-invariant feature transformation (SIFT) or Speeded Up Robust Features (SURF) for the method for obtaining a transformation matrix.

[0020] [Alignment operation] FIG. 2 is a flowchart showing the operation when the imaging system 1 according to Embodiment 1 performs alignment.

[0021] First, the light source 13 irradiates the subject 2 with active markers (step S1).

[0022] Next, the optical member 11 splits the reflected light from the subject 2 into regions of a plurality of wavelengths (step S2).

[0023] Next, the imaging device 12 images each of the images in regions of a plurality of wavelengths (step S3).

[0024] Then, the processing circuit 14 performs alignment between the images based on the active markers in each of the images in regions of a plurality of wavelengths (step S4).

[0025] FIG. 3 is a schematic diagram related to the operation of the imaging system 1 according to Embodiment 1. Further, FIG. 3 is a schematic diagram related to the operations from step S1 to step S3 in FIG. 2. FIG. 3 is a schematic diagram in which the optical member 11 splits the reflected light from the subject 2 into three wavelengths, and the three imaging elements 12 image each of the images 121 of the three wavelengths. Each image 121 is an image captured by the imaging element 12 located on the left of each image 121. The lens 15 is a lens for converging the light split by the optical member 11 and forming the image 121 on the imaging element 12.

[0026] As shown in FIG. 3, the light source 13 irradiates the subject 2 with an active marker, which is a cross-shaped marker. Further, the light source 13 irradiates four of the active markers at the corners of the subject 2 so as not to overlap the subject 2. The active marker may be any marker that can be distinguished from other feature points imaged in each image 121. For example, it may be another marker such as a circle. Further, since the light source 13 only needs to irradiate the active marker at a position that does not overlap the subject 2, it may irradiate a position other than the corners of the subject. Furthermore, as long as the processing circuit 14 can perform alignment based on the active markers in each of the images 121 in the regions of multiple wavelengths, the number of markers irradiated by the light source 13 is not limited to four.

[0027] Also, as shown in FIG. 3, each image 121 captured by the imaging element 12 has an active marker imaged therein. Since the active marker is imaged in each image 121, the processing circuit 14 can perform alignment between the images 121 in the regions of multiple wavelengths based on the active markers in each of the images 121 in the regions of multiple wavelengths.

[0028] [Configuration of the optical member] Next, the specific configuration of the optical member 11 that realizes step S2 in FIG. 2 will be described with reference to FIGS. 4 to 7.

[0029] FIG. 4 is a diagram showing a first example of an optical member 11 that splits reflected light from a subject 2 into regions of a plurality of wavelengths in the imaging system 1 according to Embodiment 1.

[0030] As shown in FIG. 4, the optical member 11 uses one total reflection mirror 111 and two dichroic mirrors 112 to split the reflected light from the subject 2 into three wavelengths (strictly speaking, three wavelength bands). Here, the total reflection mirror 111 is a mirror that reflects all incident light. The dichroic mirror 112 is a mirror that reflects light of a specific wavelength among the incident light and transmits light of other wavelengths.

[0031] The optical member 11 having the dichroic mirror 112 reflects light of a specific wavelength among the reflected light from the subject 2 and transmits light of other wavelengths. Then, the lens 15 converges the light reflected by the dichroic mirror 112 and forms an image 121 on the imaging element 12. Also, the optical member 11 having another dichroic mirror 112 reflects light of a specific wavelength among the light transmitted through the dichroic mirror 112 and transmits light of other wavelengths. Then, another lens 15 converges the light reflected by the other dichroic mirror 112 and forms an image 121 on another imaging element 12. Further, the optical member 11 having the total reflection mirror 111 reflects all the light transmitted through the other optical member 11. Then, yet another lens 15 converges the light reflected by the total reflection mirror 111 and forms an image 121 on yet another imaging element 12. In this way, the optical member 11 splits the reflected light from the subject 2 into three wavelengths.

[0032] FIG. 5 is a diagram showing a second example of the optical member 11 that splits the reflected light from the subject 2 into regions of a plurality of wavelengths in the imaging system 1 according to Embodiment 1.

[0033] As shown in FIG. 5, the optical member 11 uses one total reflection mirror 111, two partial reflection mirrors 113, and three band-pass filters 114 to split the reflected light from the subject 2 into three wavelengths (strictly speaking, three wavelength bands). Here, the partial reflection mirror 113 is a mirror that reflects part of the incident light and transmits the other light for the entire wavelength band. The band-pass filter 114 is a filter that transmits light of a specific wavelength and attenuates light of other wavelengths.

[0034] The optical member 11 having the partial reflection mirror 113 and the band-pass filter 114 reflects part of the reflected light from the subject 2 and transmits the other light. The band-pass filter 114 transmits light of a specific wavelength and attenuates light of other wavelengths among part of the light reflected by the partial reflection mirror 113. Then, the lens 15 converges the light transmitted through the band-pass filter 114 and forms an image 121 on the imaging element 12. Also, the optical member 11 having another partial reflection mirror 113 and another band-pass filter 114 reflects part of the light transmitted through the partial reflection mirror 113 and transmits the other light. Another band-pass filter 114 transmits light of a specific wavelength and attenuates light of other wavelengths among the light reflected by the other partial reflection mirror 113. Then, another lens 15 converges the light transmitted through the other band-pass filter 114 and forms an image 121 on another imaging element 12. Also, the optical member 11 having the total reflection mirror 111 and yet another band-pass filter 114 reflects all the light transmitted through the other partial reflection mirror 113. Yet another band-pass filter 114 transmits light of a specific wavelength and attenuates light of other wavelengths among the light reflected by the total reflection mirror 111. Then, yet another lens 15 converges the light transmitted through yet another band-pass filter 114 and forms an image 121 on yet another imaging element 12. Note that the light of the specific wavelengths transmitted through the three band-pass filters 114 is light of different wavelengths. In this way, the optical member 11 splits the reflected light from the subject 2 into three wavelengths.

[0035] FIG. 6 is a diagram showing a third example of the optical member 11 that splits the reflected light from the subject 2 into regions of a plurality of wavelengths in the imaging system 1 according to the first embodiment.

[0036] As shown in FIG. 6, the optical member 11 uses one dichroic mirror 112 to split the reflected light from the subject 2 into two wavelengths (strictly speaking, two wavelength bands). Note that, unlike FIGS. 4 and 5, FIG. 6 is a diagram showing an example in which an image 121 is formed on two imaging elements 12 using one optical member 11.

[0037] The optical member 11 having the dichroic mirror 112 reflects light of a specific wavelength among the reflected light from the subject 2 and transmits light of other wavelengths. One lens 15 converges the light transmitted by the dichroic mirror 112 and forms an image 121 on the imaging element 12. The other lens 15 converges the light reflected by the dichroic mirror 112 and forms an image 121 on another imaging element 12. In this way, the optical member 11 splits the reflected light from the subject 2 into two wavelengths.

[0038] FIG. 7 is a diagram showing a fourth example of the optical member 11 that splits the reflected light from the subject 2 into regions of a plurality of wavelengths in the imaging system 1 according to the first embodiment.

[0039] As shown in FIG. 7, the optical member 11 uses two dichroic mirrors 112 to split the reflected light from the subject 2 into three wavelengths (strictly speaking, three wavelength bands). Note that, unlike FIGS. 4 and 5, FIG. 7 is a diagram showing an example in which an image 121 is formed on three imaging elements 12 using one optical member 11 having two dichroic mirrors 112. The light of a specific wavelength transmitted by each dichroic mirror 112 has a different wavelength.

[0040] One dichroic mirror 112 reflects light of a specific wavelength, and the other dichroic mirror 112 reflects light of another specific wavelength. Also, light of other wavelengths that the two dichroic mirrors 112 did not reflect passes through the two dichroic mirrors 112. The three lenses 15 converge the respective light and form an image 121 on each imaging element 12. In this way, the optical member 11 splits the reflected light from the subject 2 into three wavelengths.

[0041] [Wavelength region] Next, the regions of a plurality of wavelengths split by the optical member 11 will be described. Here, the wavelength region is at least one wavelength band. For example, when the wavelength of the active marker is not included in at least one wavelength region, the image 121 obtained by imaging the at least one wavelength region is an image 121 that does not include the active marker. When such an image 121 is included, the processing circuit 14 cannot perform highly accurate alignment between the images 121 based on the active marker. Therefore, the imaging system 1 determines whether the wavelength of the active marker is present in the regions of a plurality of wavelengths split by the optical member 11, and when the wavelength of the active marker is not present in the region, it may be configured to expand by additionally providing a region of the wavelength of the active marker (that is, a region including the wavelength of the active marker). Below, the specifications of the imaging system 1 will be described with reference to FIG. 8.

[0042] FIG. 8 is a flowchart showing the flow of adding a region of the wavelength of the active marker to the regions of a plurality of wavelengths after step S2 in FIG. 2. Note that the flowchart shown in FIG. 8 may be performed between step S2 and step S3 in FIG. 2.

[0043] When the wavelength of the active marker is included in each of a plurality of wavelength regions (Yes in step S11), the imaging system 1 may be configured not to add the wavelength region of the active marker to each wavelength region. The above configuration is a configuration when the optical member 11 is split into wavelength regions from region 1, region 2, ···, and region n (n is a natural number of 2 or more), and the wavelength of the active marker is in the intersection wavelength region of region 1, region 2, ···, and region n.

[0044] Further, when the wavelength of the active marker is not included in each of the plurality of wavelength regions (No in step S11), the imaging system 1 may be configured to add the wavelength region of the active marker to the region where the wavelength of the active marker is not included (step S12). The above configuration is a configuration when the optical member 11 is split into wavelength regions from region 1, region 2, ···, and region n (n is a natural number of 2 or more), and the wavelength of the active marker is not in the intersection wavelength region of region 1, region 2, ···, and region n.

[0045] The flowchart of FIG. 8 will be described using FIGS. 9 to 12B which are specific examples. Note that FIGS. 9 to 12B are all diagrams when the optical member 11 is split into two wavelength regions of region 1 and region 2. Further, the marker wavelength shown in FIGS. 9 to 12B is the center wavelength of the active marker.

[0046] First, an example when the wavelength of the active marker is included in two regions will be described.

[0047] FIG. 9 is a diagram when the optical member splits the reflected light into two wavelength regions and the wavelength of the active marker is in the intersection region of the two regions.

[0048] As shown in FIG. 9, region 1 and region 2 have regions of the same wavelength in some regions. Also, the marker wavelength is included within the region of the same wavelength. As a result, the active marker is imaged in both the image 121 of region 1 and the image 121 of region 2. In the case of FIG. 9, the imaging system 1 does not need to add the wavelength region of the active marker to regions 1 and 2.

[0049] As a result, the imaging system 1 can project the active marker onto all images 121 by using one precise active marker that is a high-intensity and narrow-wavelength-width light source such as a laser light or an LED light source.

[0050] FIG. 10 is a diagram when the reflected light is split into regions of two wavelengths adjacent to the optical member and the wavelength of the active marker includes a wavelength sandwiched between the two wavelength regions.

[0051] As shown in FIG. 10, the marker wavelength is the wavelength at which regions 1 and 2 switch. However, since the light source 13 that irradiates the active marker is a device having a narrow wavelength width, the wavelength of the active marker is included in both regions 1 and 2. As a result, the active marker is imaged in the image 121 of region 1 and the image 121 of region 2. In the case of FIG. 10, the imaging system 1 does not need to add the wavelength region of the active marker to regions 1 and 2.

[0052] As a result, the imaging system 1 can project the active marker onto images 121 of two different wavelengths by using one precise active marker that is a high-intensity and narrow-wavelength-width light source such as a laser light or an LED light source.

[0053] Next, an example when the wavelength of the active marker is not included in at least one of the regions will be described.

[0054] FIG. 11A is a diagram when the optical member 11 splits the reflected light into two wavelength regions where the active marker wavelength is in region 1. FIG. 11B is a diagram when, after FIG. 11A, the imaging system 1 is extended by additionally providing the region of the active marker wavelength in region 2.

[0055] As shown in FIG. 11A, region 1 and region 2 do not have a common wavelength region. Also, the marker wavelength is included in region 1. In such a case, the active marker is imaged only in the image 121 of region 1 and not in the image 121 of region 2. The processing circuit 14 cannot align the images 121 of the two wavelengths based on the active marker.

[0056] Therefore, as shown in FIG. 11B, the imaging system 1 adds the region of the active marker wavelength to region 2. As a result, the active marker is imaged in the images 121 of the two wavelengths. Accordingly, the processing circuit 14 can align the images 121 of the two wavelengths based on the active marker.

[0057] Thereby, the imaging system 1 can project the active marker onto all the images 121 by using one precise active marker that is a high-brightness and narrow-wavelength-width light source such as a laser light or an LED light source.

[0058] FIG. 12A is a diagram when the optical member 11 splits the reflected light into two wavelength regions and there is no active marker wavelength in the two regions. FIG. 12B is a diagram when, after FIG. 12A, the imaging system 1 is extended by additionally providing the region of the active marker wavelength in the two regions.

[0059] As shown in FIG. 12A, Region 1 and Region 2 do not have a common wavelength region. Also, the marker wavelength is not included in either Region 1 or Region 2. In such a case, the active marker is not imaged in Image 121 of Region 1 and Image 121 of Region 2. The processing circuit 14 cannot align the images 121 of the two wavelengths based on the active marker.

[0060] Therefore, as shown in FIG. 12B, the imaging system 1 adds the wavelength region of the active marker to Region 1 and Region 2. As a result, the active marker is imaged in the images 121 of the two wavelengths. Accordingly, the processing circuit 14 can align the images 121 of the two wavelengths based on the active marker.

[0061] Thereby, the imaging system 1 can project the active marker onto all the images 121 by using one precise active marker that is a high-brightness and narrow-wavelength-width light source such as a laser light or an LED light source.

[0062] [Method for Calculating Luminance of Active Marker] FIG. 13 is a diagram showing a method for calculating the luminance of the active marker in each of the images 121 in a plurality of wavelength regions. (a) of FIG. 13 is a diagram showing the wavelength region of light (i.e., the sensitive wavelength region) that the imaging device 12 can image. The component overlapping with the wavelength region of the light source in (c) of FIG. 13 to be described later is shown in a horizontal stripe pattern. (b) of FIG. 13 is a diagram showing region 1 and region 2 which are the wavelength regions spectrally separated by the optical member 11. The component overlapping with the wavelength region of the light source in (c) of FIG. 13 to be described later is shown in a diagonal stripe pattern. In (b) of FIG. 13, the wavelength component of the light source in region 1 is shown in a downward diagonal stripe pattern, and the wavelength component of the light source in region 2 is shown in an upward diagonal stripe pattern. (c) of FIG. 13 is a diagram showing the wavelength region of the active marker (i.e., the wavelength component of the light source). (d) of FIG. 13 is a diagram showing the luminance of the active marker in the image 121 obtained by imaging region 1 and the luminance of the active marker in the image 121 obtained by imaging region 2. In (d) of FIG. 13, the luminance of the active marker in the image 121 obtained by imaging region 1 is represented by a figure with a downward diagonal stripe pattern drawn, and the luminance of the active marker in the image 121 obtained by imaging region 2 is represented by a figure with an upward diagonal stripe pattern drawn.

[0063] The dotted line shown in (a) of FIG. 13 is a line indicating the sensitivity of the imaging device 12. Also, the dotted line is expressed as a wavelength function F1(λ) of the sensitivity (i.e., spectral sensitivity) of the imaging device 12.

[0064] The broken line shown in (b) of FIG. 13 is a line indicating the wavelength region of region 1 spectrally separated by the optical member 11, and the solid line is a line indicating the wavelength region of region 2 spectrally separated by the optical member 11. Also, the broken line and the solid line are expressed as a function F2i(λ) that describes the wavelength dependence of the spectral intensity (spectral characteristics) in each of the plurality of wavelength regions spectrally separated by the optical member 11. Here, i is a number assigned to the plurality of wavelength regions, and is expressed as i = 1, 2, ···, n. Also, n is an integer of 2 or more. That is, the dotted line indicating region 1 is expressed as F21(λ), and the broken line indicating region 2 is expressed as F22(λ).

[0065] The solid line shown in Fig. 13(c) is a line indicating the wavelength region of the active marker. Also, the solid line is expressed as a function F3(λ) that describes the wavelength dependence of the intensity of the light source 13 that irradiates the active marker (i.e., the light source intensity).

[0066] The figure showing the luminance of the active marker shown in Fig. 13(d) is calculated by the following formula (1) described using the above functions F1(λ), F2n(λ), and F3(λ). Note that the range of each integral formula is the range in which the pattern of each function is drawn.

[0067] [Equation]

[0068] The figure with a downward-sloping diagonal stripe pattern showing the luminance of the active marker in the image 121 obtained by imaging region 1 is calculated by the integral formula where the value of i in F2i(λ) is 1. Also, the figure with an upward-sloping diagonal stripe pattern showing the luminance of the active marker in the image 121 obtained by imaging region 2 is calculated by the integral formula where the value of i in F2i(λ) is 2.

[0069] The optical member 11 disperses the reflected light from the subject 2 into regions of a plurality of wavelengths so that the above formula (1) holds. Thereby, the imaging system 1 can reduce the difference in the luminance of the active marker imaged in the images 121 of each wavelength. Therefore, the imaging system 1 enables more accurate alignment.

[0070] Note that the luminance values of the images of each wavelength calculated by the above formula (1) do not necessarily have to be the same value. For example, when the error calculated from the luminance values of the images of each wavelength is within 10%, the imaging system 1 may consider that the above formula (1) holds.

[0071] [Comparative Example 1] Next, a comparative example will be described. FIG. 14 is a schematic diagram showing a comparative example of FIG. 3. Similar to FIG. 3, FIG. 14 is a diagram when the light source 13 irradiates the active marker and the optical member 11 splits the reflected light from the subject 2 into three wavelengths. Note that FIG. 14 is a diagram when the wavelength of the active marker is included in two wavelength regions and not included in one wavelength region among the three wavelength regions split by the optical member 11.

[0072] As shown in FIG. 14, the lens 15 converges each light split by the optical member 11 and forms an image 121 on the imaging device 12. Note that the image 121 drawn to the right of each imaging device 12 is the image formed on each imaging device 12.

[0073] Among the three images 121, the upper and middle images 121 capture the active marker. On the other hand, the lower image 121 among the three images 121 does not capture the active marker. In such a case, the processing circuit 14 cannot perform alignment of the three images based on the active marker.

[0074] [Comparative Example 2] Next, another comparative example will be described. FIG. 15 is a schematic diagram showing another comparative example of FIG. 3. Similar to FIG. 3, FIG. 15 is a diagram when the light source 13 irradiates the active marker and the optical member 11 splits the reflected light from the subject 2 into three wavelengths. Note that FIG. 15 is a diagram when the light source 13 irradiates two active markers with different center wavelengths.

[0075] As shown in FIG. 15, the lens 15 converges each light split by the optical member 11 and forms an image 121 on the imaging device 12. Note that the image 121 drawn to the right of each imaging device 12 is the image formed on each imaging device 12.

[0076] In the upper and middle images 121 among the three images 121, one of the two active markers is imaged. Also, in the lower image 121 among the three images 121, the other of the two active markers is imaged. With active markers having different center wavelengths, a shift occurs between the irradiation light sources. Due to this shift, the positions of the active markers in the upper and middle images 121 are shifted from the positions of the active markers in the lower image 121. In such a case, the accuracy of alignment performed by the processing circuit 14 decreases.

[0077] [Effects, etc.] As described above, since the imaging system 1 performs alignment based on the active markers in each of the images in a plurality of wavelength regions, it is less affected by changes in the optical state and temporal variations due to mechanical vibration or distortion. Thereby, the imaging system 1 enables highly accurate alignment. Also, in the method of image alignment, for the same reason, the accuracy of alignment between images is improved.

[0078] (Embodiment 2) The imaging system 1a according to Embodiment 2 differs from the imaging system 1 according to Embodiment 1 in that the images in a plurality of wavelength regions spectrally separated by the optical member 11 are formed on one imaging device 12. Hereinafter, the description will focus on the points that differ from the imaging system 1 according to Embodiment 1, and the description of the same points will be omitted.

[0079] [Configuration] FIG. 16 is a block diagram showing the configuration of the imaging system 1a according to Embodiment 2. The imaging system 1a is the same as the imaging system 1 according to Embodiment 1 in that it includes an optical member 11, a light source 13, and a processing circuit 14, but the imaging system 1a differs from the imaging system 1 according to Embodiment 1 in that it includes one imaging device 12.

[0080] [Alignment operation] The operation of the imaging system 1a for alignment is the same as the operation described in FIG. 2, so the description thereof is omitted. FIG. 17 is a schematic diagram related to the operation of the imaging system according to the second embodiment. Further, FIG. 17 is a schematic diagram related to the operations from step S1 to step S3 in FIG. 2. FIG. 17 is a schematic diagram in which the optical member 11 splits the reflected light into two wavelengths, and one imaging element 12 images each of the images of the two wavelengths. Note that the image 121 is an image in which images of two wavelengths are captured. Also, the lens 15 shown in FIG. 17 is a lens for converging the light split by the optical member 11 and forming the image 121 on the imaging element 12. The cross-shaped markers depicted on the subject 2 and the image 121 are active markers as in the first embodiment.

[0081] As shown in FIG. 17, the lens 15 forms images of two wavelengths on one imaging element 12. The processing circuit 14 performs alignment between the images of the two wavelengths based on the active markers depicted in the image 121.

[0082] [Configuration of Optical Member] Next, the specific configuration of the optical member 11 that realizes step S2 in FIG. 2 will be described with reference to FIGS. 18 to 21. The dashed arrows used in FIGS. 18 to 21 indicate the path of light in the long-wavelength region, and the dotted arrows indicate the path of light in the short-wavelength region. FIGS. 18 and 19 are cross-sectional views of the optical member 11.

[0083] FIG. 18 is a diagram showing a first example of the optical member 11 that splits the reflected light from the subject 2 into two wavelengths in the imaging system 1a according to the second embodiment. FIG. 18(a) is a schematic diagram in which the optical member 11 splits into two wavelengths and the imaging element 12 images the images of the two wavelengths. FIG. 18(b) is an enlarged view of the dielectric multilayer film wavelength prism 115 included in the optical member 11.

[0084] The dielectric multilayer film wavelength prism 115 is an optical element having a dielectric multilayer film wavelength filter 115a. As shown in FIGS. 18(a) and 18(b), the dielectric multilayer film wavelength prism 115 has a dielectric multilayer film wavelength filter 115a provided perpendicular to the surface facing the lens 15. Note that, as shown in FIG. 18(b), the dielectric multilayer film wavelength prism 115 may be, for example, a prism having a regular triangular prism shape in which the cross-sectional shape is a regular triangle, or a prism other than the regular triangular prism shape.

[0085] The dielectric multilayer film wavelength filter 115a is a filter that reflects light in a specific wavelength region and transmits light in other wavelength regions. In Embodiment 2, the dielectric multilayer film wavelength filter 115a is a filter that reflects light in the long wavelength region (for example, a wavelength region of 650 nm or more), but may be a filter that reflects light in the short wavelength region.

[0086] As shown in FIG. 18(a), the optical member 11 uses the dielectric multilayer film wavelength prism 115 to split the reflected light from the subject 2 into light in the long wavelength region and light in the short wavelength region. Then, the lens 15 converges the light in the long wavelength region and the light in the short wavelength region, respectively, and forms an image 121 on one imaging element 12.

[0087] Thereby, the optical member 11 can split the reflected light from the subject 2 into two wavelength regions.

[0088] Note that, as shown in FIG. 18(a), the light in the long wavelength region is transmitted after being reflected twice within the dielectric multilayer film wavelength prism 115, whereas the light in the short wavelength region is transmitted after being reflected once within the dielectric multilayer film wavelength prism 115. Thereby, the phase of the wavelength in the light in the long wavelength region is in the opposite phase to the phase of the wavelength in the light in the short wavelength region. Thereby, the two wavelength images captured by the imaging element 12 are mirror images.

[0089] FIG. 19 is a diagram showing a second example of the optical member 11 that splits the reflected light from the subject 2 into two wavelengths in the imaging system 1a according to the second embodiment. Note that FIG. 19 is different from FIG. 18 in that the optical member 11 has a prism 116, but is the same as FIG. 18 in other respects.

[0090] The prism 116 is an optical element for causing light to be dispersed, refracted, totally reflected, or the like.

[0091] As shown in FIG. 19, the reflected light from the subject 2 is incident on the incident surface of the dielectric multilayer film wavelength prism 115 at a shallow angle. Therefore, the optical member 11 can make the reflected light incident on the incident surface at a deep angle by passing it through the prism 116.

[0092] In addition, the reflected light incident on the prism 116 is reflected twice inside the prism 116 and then transmitted. Therefore, the phase of the reflected light before incidence on the prism 116 and the phase of the reflected light after transmission from the prism 116 are in the same phase.

[0093] FIG. 20 is a diagram showing a third example of the optical member 11 that splits the reflected light from the subject 2 into two wavelengths in the imaging system 1a according to the second embodiment.

[0094] As shown in FIG. 20, the optical member 11 includes two total reflection mirrors 111, two band-pass filters 114, and two half mirrors 117. One of the band-pass filters 114 is a filter that transmits light in the long wavelength region and attenuates light in the short wavelength region. The other band-pass filter 114 is a filter that transmits light in the short wavelength region and attenuates light in the long wavelength region.

[0095] The half mirror 117 is a mirror that splits light such that the intensity of the reflected light is almost the same as the intensity of the transmitted light when the incident light is incident on one surface, and is a mirror that transmits all of the incident light when the incident light is incident on the other surface.

[0096] First, the first half mirror 117 splits the reflected light from the subject 2. One total reflection mirror 111 totally reflects the reflected light split by the first half mirror 117, and the other total reflection mirror 111 totally reflects the transmitted light split by the first half mirror 117. The respective lights totally reflected by the two total reflection mirrors 111 enter different band - pass filters 114. One band - pass filter 114 transmits light in the long - wavelength region, and the other band - pass filter 114 transmits light in the short - wavelength region. Finally, the light in the long - wavelength region and the light in the short - wavelength region enter the second half mirror 117. Then, the lens 15 converges the light in the long - wavelength region and the light in the short - wavelength region respectively, and forms an image 121 on the imaging device 12. In this figure, the surface where the light in the long - wavelength region enters is a surface that transmits all the incident light.

[0097] Thus, the optical member 11 can split the reflected light from the subject 2 into two wavelength regions.

[0098] Note that the light in the long - wavelength region and the light in the short - wavelength region are transmitted after reflecting twice within the optical member 11. Therefore, the phase of the wavelength in the light in the long - wavelength region is in the same phase as the phase of the wavelength in the light in the short - wavelength region.

[0099] FIG. 21 is a diagram showing a fourth example of the optical member 11 that splits the reflected light from the subject 2 into two wavelengths in the imaging system 1a according to the second embodiment.

[0100] As shown in FIG. 21, the optical member 11 has two total reflection mirrors 111 and two dichroic mirrors 112. Note that one dichroic mirror 112 is a mirror that reflects light in the long - wavelength region and transmits light in the short - wavelength region. The other dichroic mirror 112 is a mirror that reflects light in the short - wavelength region and transmits light in the long - wavelength region.

[0101] First, one dichroic mirror 112 reflects light in the long-wavelength region among the reflected light from the subject 2 and transmits light in the short-wavelength region. One total reflection mirror 111 totally reflects light in the long-wavelength region, and the other total reflection mirror 111 totally reflects light in the short-wavelength region. Each of the lights totally reflected by the two total reflection mirrors 111 enters the other dichroic mirror 112. The other dichroic mirror 112 reflects light in the short-wavelength region and transmits light in the long-wavelength region. Then, the lens 15 converges the light in the long-wavelength region and the light in the short-wavelength region respectively, and forms an image 121 on the imaging element 12.

[0102] Thus, the optical member 11 can split the reflected light from the subject 2 into two wavelength regions.

[0103] Note that, similar to FIG. 20, the phase of the wavelength in the light in the long-wavelength region is in the same phase as the phase of the wavelength in the light in the short-wavelength region.

[0104] [Effects, etc.] As described above, the imaging system 1a performs alignment based on the active markers in each of the images in a plurality of wavelength regions, so it is less affected by changes in the optical state and temporal variations due to mechanical vibration or distortion. Thereby, the imaging system 1a enables highly accurate alignment. Also, in the method of image alignment, for the same reason, the accuracy of alignment between images is improved.

[0105] [Modification Example] As described above, the imaging system according to one or more aspects has been described based on the above-described embodiments, but the present disclosure is not limited to the above-described embodiments. As long as the gist of the present disclosure is not deviated from, various modifications conceived by those skilled in the art applied to the above-described embodiments, or forms constructed by combining components in different embodiments may also be included within the scope of one or more aspects.

[0106] In the above embodiment, the processing circuit may be configured by dedicated hardware or may be realized by executing a software program suitable for the processing circuit. The processing circuit may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0107] Also, part or all of the functions of the processing circuit according to the above embodiment may be realized by a processor such as a CPU executing a program.

[0108] Part or all of the above devices may be configured from an IC card or a single module detachable from each device. The above IC card or module is a computer system composed of a microprocessor, ROM, RAM, etc. The above IC card or module may include a super multifunctional LSI. When the microprocessor operates according to a computer program, the above IC card or module achieves its function. This IC card or module may have tamper resistance.

[0109] [Appendix] From the description of the above embodiments, the following technologies are disclosed.

[0110] (Technology 1) An imaging system including: an optical member that splits light from a subject into regions of a plurality of wavelengths; at least one imaging device that images each of the images in the regions of the plurality of wavelengths; a light source that irradiates an active marker on the subject; and a processing circuit that performs alignment between the images in the regions of the plurality of wavelengths based on the active marker in each of the images in the regions of the plurality of wavelengths.

[0111] With this configuration, the imaging system performs alignment based on active markers in each of the images in the regions of multiple wavelengths, so it is less affected by changes in the optical state and temporal variations due to mechanical vibration or strain. As a result, the imaging system enables highly accurate alignment.

[0112] (Technology 2) The imaging system according to Technology 1, wherein when the regions of the multiple wavelengths into which the optical member disperses light are Region 1, Region 2, ···, and Region n, the wavelength of the active marker is within the wavelength region of the intersection of Region 1, Region 2, ···, and Region n.

[0113] With this configuration, the imaging system can project an active marker onto all images by using one precise active marker that is a high-intensity and narrow-wavelength-width light source such as a laser light or an LED light source. As a result, the imaging system enables highly accurate alignment.

[0114] (Technology 3) The imaging system according to Technology 1, wherein the optical member disperses the light into two adjacent wavelength regions, and the wavelength of the active marker includes a wavelength sandwiched between the two wavelength regions.

[0115] With this configuration, the imaging system can project an active marker onto images of two different wavelengths by using one precise active marker that is a high-intensity and narrow-wavelength-width light source such as a laser light or an LED light source. As a result, the imaging system enables highly accurate alignment.

[0116] (Technology 4) The imaging system according to Technology 1, wherein when the regions of the multiple wavelengths into which the optical member disperses light are Region 1, Region 2, ···, and Region n, and the wavelength of the active marker is a wavelength within Region X which is one of Region 1, Region 2, ···, and Region n, the regions other than Region X among Region 1, Region 2, ···, and Region n are extended by additionally providing a region of the wavelength of the active marker.

[0117] With this configuration, the imaging system can project an active marker onto all images by using one precise active marker that is a high-brightness and narrow-wavelength-width light source such as a laser light or an LED light source. Thereby, the imaging system enables highly accurate alignment.

[0118] (Technology 5) The regions of the plurality of wavelengths spectrally separated by the optical member are defined as region 1, region 2, ···, and region n. When the wavelength of the active marker is not in any of the regions of region 1, region 2, ···, and region n, each of region 1, region 2, ···, and region n is extended by additionally providing a region of the wavelength of the active marker. The imaging system according to Technology 1 or 4.

[0119] With this configuration, the imaging system can project an active marker onto all images by using one precise active marker that is a high-brightness and narrow-wavelength-width light source such as a laser light or an LED light source. Thereby, the imaging system enables highly accurate alignment.

[0120] (Technology 6) When the wavelength function of the sensitivity of the imaging element is F1(λ), the function describing the wavelength dependence of the spectral intensity in each of the regions of the plurality of wavelengths spectrally separated by the optical member is F2i(λ) (where the plurality of wavelength regions are sequentially i = 1, 2, ···, n), and the function describing the wavelength dependence of the intensity of the active marker is F3(λ), the imaging system according to any one of Technologies 1 to 5, for which the above formula (1) holds.

[0121] With this configuration, the imaging system can reduce the difference in the brightness of the active marker imaged in the images of each wavelength. Thereby, the imaging system enables even more highly accurate alignment.

[0122] (Technique 7) An image alignment method that splits light from a subject into regions of a plurality of wavelengths, captures images of the regions of the plurality of wavelengths with at least one image sensor, irradiates an active marker on the subject, and performs alignment between the images of the regions of the plurality of wavelengths based on the active marker in the images of the regions of the plurality of wavelengths.

[0123] With this configuration, the alignment method performs alignment based on the active marker in each of the images of the regions of the plurality of wavelengths, so it is less affected by changes in the optical state and temporal variations due to mechanical vibration or distortion. As a result, the alignment method enables highly accurate alignment.

Industrial Applicability

[0124] The imaging system according to the present disclosure is useful, for example, as a system for performing alignment between images of regions of a plurality of wavelengths.

Explanation of Reference Numerals

[0125] 1, 1a Imaging system 2 Subject 11 Optical member 12 Image sensor 13 Light source 14 Processing circuit 15 Lens 111 Total reflection mirror 112 Dichroic mirror 113 Partial reflection mirror 114 Bandpass filter 115 Dielectric multilayer wavelength prism 115a Dielectric multilayer wavelength filter 116 Prism 117 Half mirror 121 Image

Claims

1. An optical member that splits light from a subject into regions of a plurality of wavelengths, At least one imaging device that images each of the images in the regions of the plurality of wavelengths, A light source that irradiates an active marker onto the subject, A processing circuit that performs alignment between the images in the regions of the plurality of wavelengths based on the active marker in each of the images in the regions of the plurality of wavelengths, An imaging system.

2. When the regions of the plurality of wavelengths split by the optical member are Region 1, Region 2, ···, and Region n, the wavelength of the active marker is within the wavelength region of the intersection of Region 1, Region 2, ···, and Region n, The imaging system according to Claim 1.

3. The optical member splits the light into two adjacent wavelength regions, The wavelength of the active marker includes a wavelength sandwiched between the two wavelength regions, The imaging system according to Claim 1.

4. When the regions of the plurality of wavelengths split by the optical member are Region 1, Region 2, ···, and Region n, When the wavelength of the active marker is a wavelength within Region X which is one of Region 1, Region 2, ···, and Region n, The regions other than Region X among Region 1, Region 2, ···, and Region n are extended by additionally providing a region of the wavelength of the active marker, The imaging system according to Claim 1.

5. When the regions of the plurality of wavelengths split by the optical member are Region 1, Region 2, ···, and Region n, When the wavelength of the active marker is not in any of Region 1, Region 2, ···, and Region n, Each of the regions 1, 2, ···, and n is extended by additionally providing a region of the wavelength of the active marker. The imaging system according to claim 1.

6. Let the wavelength function of the sensitivity of the imaging element be F1(λ), Let the function describing the wavelength dependence of the spectral intensity in each of the regions of the plurality of wavelengths dispersed by the optical member be F2i(λ) (for the plurality of wavelength regions in order i = 1, 2, ···, n), When the function describing the wavelength dependence of the intensity of the active marker is F3(λ), The following formula (1) holds, The imaging system according to any one of claims 1 to 5. 【Equation 1】

7. A method for aligning images, comprising: Dispersing light from a subject into regions of a plurality of wavelengths, Imaging images of the plurality of wavelength regions with at least one imaging element, Irradiating the subject with an active marker, Performing alignment between the images of the plurality of wavelength regions based on the active marker in the images of the plurality of wavelength regions. A method for aligning images.

Images