Imaging device and imaging system

Abstract

The present invention provides an imaging device that can rapidly extract a desired imaging area from a wide field of view and capture it at high resolution. [Solution] The imaging device 1 comprises a first optical system 10 that forms an intermediate image 18 of a subject, a second optical system 20 that re-images at least a part of the intermediate image to form a final image 35 and can change the magnification of the final image, an image sensor 36 that captures the final image, and a first drive unit 24 that moves the second optical system and the image sensor in a direction intersecting the optical axis of the first optical system.

Description

Technical Field

[0001] The present invention relates to an imaging device.

Background Art

[0002] A camera is attached to a rotatable pan-tilt head device so that the camera can be directed at an object (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

[0004] According to a first aspect, an imaging device includes a first optical system that forms an intermediate image of a subject, an optical system that recombines at least a part of the intermediate image to form a final image, the optical system being a second optical system capable of changing the magnification of the final image, an imaging element that images the final image, and a driving unit that moves the second optical system and the imaging element in a direction intersecting the optical axis of the first optical system.

Brief Description of the Drawings

[0005] [Figure 1] A diagram showing an overview of the imaging device according to the first embodiment. [Figure 2] A diagram for explaining the intermediate image and the imaging area, which is a part of the intermediate image recombined on the imaging element by the second optical system. [Figure 3] A diagram showing an enlarged view of a part of the first optical system and the second optical system. [Figure 4] A diagram showing an example of the usage state of the imaging device according to the first embodiment. [Figure 5] A diagram showing an example of an image captured by the imaging device. [Figure 6] A diagram showing another example of an image captured by the imaging device. [Figure 7]A diagram showing an overview of the imaging device of the second embodiment. [Figure 8] A diagram showing an example of the usage state of the imaging device according to the second embodiment. [Figure 9] This figure shows an example of an image captured by the imaging device of the second embodiment and an example of an image generated by the virtual image generation unit. [Figure 10] This figure shows another example of an image captured by the imaging device of the second embodiment and an image generated by the virtual image generation unit. [Figure 11] A diagram showing a distance measuring unit in which part of the optical system is shared with the first and second optical systems. [Modes for carrying out the invention]

[0006] In this specification, "image data" means data representing a video or a still image. In this specification, "image" means image data displayed in a way that is visible to the naked eye, such as on a display.

[0007] (Imaging device of the first embodiment) Figure 1 shows an overview of the imaging device 1 of the first embodiment. The X, Y, and Z directions indicated by arrows in Figure 1 are defined as the + direction. The X, Y, and Z directions are mutually orthogonal directions. In the following, the position in the X direction will be referred to as the X position, the position in the Y direction as the Y position, and the position in the Z direction as the Z position. Note that the X, Y, and Z directions in each figure referred to later are the same directions as the X, Y, and Z directions shown in Figure 1.

[0008] The imaging device 1 of the first embodiment comprises a first optical system 10, a second optical system 20, an image sensor 36, and a control unit 50. The first optical system 10 forms an intermediate image 18 of a subject (not shown). The second optical system 20 re-images at least a portion of the intermediate image 18 formed by the first optical system 10 to form a final image 35 of the subject.

[0009] An image sensor 36 having an imaging surface is positioned at the location where the final image 35 of the subject is formed. The imaging surface of the image sensor 36 coincides with the final image 35, and the centers of its imaging surface in the X and Y directions generally coincide with the optical axis AX2 of the second optical system 20. This image sensor 36 is held by the second lens barrel 21.

[0010] Furthermore, the imaging plane of the image sensor 36 and the image plane of the final image 35 produced by the second optical system 20 do not necessarily have to coincide. For example, the imaging plane of the image sensor 36 and the image plane of the final image 35 produced by the second optical system 20 may be offset in a direction along the optical axis AX2. For example, it is sufficient that the image of the subject can be seen in the image generated by imaging by the image sensor 36, or that the image data of the subject can be recognized in the image data using existing image processing.

[0011] Furthermore, the centers of the imaging surface of the image sensor 36 in the X and Y directions do not necessarily have to coincide with the optical axis AX2 of the second optical system 20. For example, the centers of the imaging surface of the image sensor 36 in the X and Y directions may be offset from the optical axis AX2 of the second optical system 20. For example, it is sufficient that the image of the subject can be seen in the image generated by imaging with the image sensor 36, or that the image data of the subject can be recognized in the image data using existing image processing.

[0012] For example, the optical axis AX1 of the first optical system 10 and the optical axis AX2 of the second optical system 20 are both parallel to the Z direction. Also, for example, the intermediate image plane on which the intermediate image 18 is formed is parallel to the XY plane perpendicular to the Z direction, and the final image plane on which the final image 35 is formed is also parallel to the XY plane.

[0013] The lenses 12-15 constituting the first optical system 10 are fixed to the first lens barrel 11, and the first lens barrel 11 is fixed to the housing 38, which is the outer frame of the imaging device 1. One or more of the lenses 12-15 of the first optical system 10 do not have to be fixed to the first lens barrel 11, and may be provided so as to be movable relative to the first lens barrel 11. Also, the first lens barrel 11 does not have to be fixed to the housing 38. When the subject is at a finite distance from the imaging device 1 (first optical system 10), the first optical system 10 may have a reduction ratio. When the subject is at infinity relative to the imaging device 1, the size of the intermediate image 18 (intermediate image formation region 19) formed by the first optical system 10 may be smaller than the radial (orthogonal to the optical axis) size of the optical element closest to the subject among the optical elements constituting the first optical system 10. In these cases, there is an advantage in that the movement stroke of the second optical system 20 can be reduced. The lenses 25-28 that constitute the second optical system 20 are held in the second lens barrel 21, as will be described later. The second lens barrel 21 is held in the housing 38 via the first drive unit 24.

[0014] The first drive unit 24 includes, for example, a stator 23 fixed to the housing 38 and a mover 22 fixed to the second lens barrel 21 and movable in the XY plane relative to the stator 23. The first drive unit 24 holds the second lens barrel 21 and moves it in the XY plane relative to the housing 38. In other words, the first drive unit 24 moves the second optical system 20 and the image sensor 36, which are supported by the second lens barrel 21, in the XY plane relative to the first optical system 10, which is fixed to the housing 38. As the first drive unit 24 drives the second optical system 20, the center position of the region (i.e., the imaging region) that is re-imaged onto the image sensor 36 as the final image 35 in the intermediate image 18 formed on the intermediate image plane by the first optical system 10 changes.

[0015] The first drive unit 24 may be, for example, a linear motor having a stator 23 and a mover 22, or may be configured to have a stepping motor and a lead screw. Note that the first drive unit 24 may be configured to have a stepping motor and a guide bar. The second lens barrel 21 moves in the XY-plane direction by the first drive unit 24 while supporting the second optical system 20 and the imaging element 36 so that the optical positional relationship between the second optical system 20 and the imaging element 36 (for example, the relative positional relationship between the second optical system 20 and the imaging element 36) remains unchanged.

[0016] Note that the moving direction of the second lens barrel 21 by the first drive unit 24 is not necessarily limited to the XY-plane direction, and may be a direction intersecting the Z direction, that is, a direction intersecting the optical axis AX1 of the first optical system 10. In other words, the first drive unit 24 may move the second optical system 20 and the imaging element 36 in a direction intersecting the optical axis AX1 of the first optical system 10. The moving direction of the second lens barrel 21 may be a rotational direction, for example, at least one of the rotational directions about the X-axis (θx direction), the rotational direction about the Y-axis (θy direction), and the rotational direction about the Z-axis (θz direction).

[0017] Among the lenses 25 to 28 constituting the second optical system 20, the lenses 25 and 28 are fixed to the second lens barrel 21. On the other hand, the lens 26 is supported by a movable support portion 29 that is movably supported in the optical axis direction (for example, ±Z direction) with respect to the second lens barrel 21 by a drive mechanism 31 such as a linear motor. The lens 27 is supported by a movable support portion 30 that is movably supported in the optical axis direction (for example, ±Z direction) with respect to the second lens barrel 21 by a drive mechanism 32 such as a linear motor.

[0018] That is, the lenses 26 and 27 are movable in the optical axis direction with respect to the lenses 25 and 28 by the drive mechanisms 31 and 32. Note that the drive mechanisms 31 and 32 are not limited to the mechanisms including the above-described linear motor, and may be a moving mechanism having a stepping motor and a lead screw.

[0019] In addition, the lens 26 may be configured to be movable in both the +Z direction and the -Z direction by the drive mechanism 31, or may be configured to be movable only in either the +Z direction or the -Z direction. Further, the lens 27 may be configured to be movable in both the +Z direction and the -Z direction by the drive mechanism 32, or may be configured to be movable only in either the +Z direction or the -Z direction.

[0020] Hereinafter, the drive mechanisms 31 and 32 are collectively or individually also referred to as the "second drive unit" 33. By the second drive unit 33 driving the lens 26 and the lens 26, the lens 26 and the lens 27 move in the Z direction, and the imaging magnification of the second optical system 20 (the magnification of the final image 35 with respect to the intermediate image 18, that is, the magnification of the final image 35) changes. Hereinafter, the second lens barrel 21 that holds the second optical system 20 and the imaging device 36 is also referred to as a "holding unit". Note that the second lens barrel 21 may integrally hold the second optical system 20 and the imaging device 36. Further, the second lens barrel 21 may be configured to be integrally formed, or may be configured by combining a plurality of components.

[0021] In FIG. 1, the lenses 12 to 15 and the lenses 25 to 28 are each shown as a single lens, but each may be a lens group composed of a plurality of lenses, or may include a mirror, a diffractive optical element, or the like. Further, the number of lens groups constituting the first optical system 10 is not limited to the four shown in the figure, and may be any other number.

[0022] The number of lens groups constituting the second optical system 20 is not limited to the four shown in the figure, and may be any number of one or more. Further, among the lens groups constituting the second optical system 20, the number of lens groups (or lenses) moved in the Z direction by the second drive unit 33 is not limited to the two groups described above, and may be any other number.

[0023] The image formation position of the final image 35 may be aligned with the imaging plane of the image sensor 36 by moving some of the lenses 26, 27, etc., in the optical axis direction by the second drive unit 33. Alternatively, there may be a drive mechanism that moves lenses 25, 28, etc., other than the lenses 26, 27 moved in the optical axis direction by the second drive unit 33, and this drive mechanism may be used to align the image formation position of the final image 35 with the imaging plane of the image sensor 36. Alternatively, there may be a drive mechanism that moves the image sensor 36 in the optical axis direction, and this drive mechanism may be used to align the imaging plane of the image sensor 36 with the image formation position of the final image 35.

[0024] As described above, in the imaging device 1 of the first embodiment, the second optical system 20 re-images at least a portion of the intermediate image 18 of the subject formed by the first optical system 10 to form the final image 35 of the subject on the image sensor 36.

[0025] Figure 2 shows the relationship between the intermediate image 18 and the imaging region IA1, which is a part of the intermediate image 18 that is re-imaged by the second optical system 20 and formed as the final image 35 on the image sensor 36. The intermediate image 18 is viewed from the +Z direction. The intermediate image 18 of the subject is formed by the first optical system 10 within the imaging region of the first optical system 10, which is, for example, a roughly circular intermediate image formation region 19. As described above, the imaging region IA1 is the portion of the intermediate image 18 that is re-imaged onto the image sensor 36 as the final image 35, and therefore corresponds to the field of view of the imaging device 1. Accordingly, the imaging region IA1 may also be referred to as the field of view of the second optical system 20.

[0026] When the optical axis AX2 of the second optical system 20 coincides with the optical axis AX1 of the first optical system 10, and the imaging magnification of the second optical system 20 (magnification of the final image 35) is at its minimum, the second optical system 20 re-images the intermediate image 18 within the rectangular wide-field imaging area IA0 shown by the dashed line onto the image sensor 36 as the final image 35.

[0027] The imaging region IA1, which is re-imaged onto the image sensor 36 as the final image 35 from among the intermediate images 18, has its center position CT changed when the second optical system 20 and the second lens barrel 21 holding the image sensor 36 are moved by the first drive unit 24. The distance DX in the X direction and the distance DY in the Y direction from the center position CT of the imaging region IA1 with respect to the optical axis AX1 of the first optical system 10 coincide with the distance in the X direction and the distance in the Y direction from the optical axis AX1 of the first optical system 10 to the optical axis AX2 of the second optical system 10, respectively.

[0028] Furthermore, the width WX in the X direction and the width WY in the Y direction, which are the dimensions of the imaging area IA1, change due to the change in the imaging magnification of the second optical system 20 (magnification of the final image 35) that occurs when the second drive unit 33 is driven. The second drive unit 33 changes the overall imaging magnification of the imaging device 1 by changing the imaging magnification of the second optical system 20.

[0029] As described above, since the imaging area IA1 corresponds to the field of view of the imaging device 1, it can be said that the field of view of the imaging device 1 changes when the first drive unit 24 or the second drive unit 33 is driven. More specifically, the center of the field of view of the imaging device 1 changes when the first drive unit 24 is driven. Also, the width of the field of view of the imaging device 1 changes when the second drive unit 33 is driven.

[0030] The relative sizes of the wide-field imaging region IA0 and the intermediate image formation region 19 of the first optical system 10 are not necessarily limited to the relationship shown in Figure 2. For example, the wide-field imaging region IA0 may encompass the entire intermediate image formation region 19, in which case all of the intermediate images 18 of the first optical system 10 can be re-imaged as the final image 35 on the image sensor 36 by the second optical system 20. Alternatively, the entire wide-field imaging region IA0 may be encompassed by the intermediate image formation region 19. The wide-field imaging region IA0 may also be inscribed within the circle that defines the outer edge of the intermediate image formation region 19.

[0031] Alternatively, the first drive unit 24 may be configured to move the first lens barrel 11 containing the first optical system 10 in the XY plane, instead of moving the second lens barrel 21 containing the second optical system 20 and the image sensor 36 in a direction intersecting the optical axis AX1 of the first optical system 10 (for example, in the XY plane direction). Even with this configuration, the first drive unit 24 can change the positional relationship between the first optical system 10 and the second optical system 20 and the image sensor 36 in the direction intersecting the optical axis AX2 of the second optical system 20 by moving the first optical system 10.

[0032] Furthermore, the first drive unit 24 may be configured to move the second lens barrel 21, which includes the second optical system 20 and the image sensor 36, in a direction intersecting the optical axis AX1 (for example, in the XY plane), as well as move the first lens barrel 11, which includes the first optical system 10, in a direction intersecting the optical axis AX2 (for example, in the XY plane). Even in this configuration, the first drive unit 24 can move the first optical system 10, the second optical system 20 and the image sensor 36 relative to each other in a direction intersecting the optical axis AX1 of the first optical system 10 by moving the first optical system 10 and the second optical system 20 and the image sensor 36.

[0033] Furthermore, the second optical system 20 and the image sensor 36 do not necessarily have to be held integrally by the holding unit (second lens barrel 21). For example, the second optical system 20 may be held by the second lens barrel 21, and the image sensor 36 may be held by a separate holding mechanism from the second lens barrel 21. The first drive unit 24 may move the second lens barrel 21 and the other holding mechanism in a direction intersecting the optical axis AX1 of the first optical system 10 so as not to change the optical positional relationship between the second optical system 20 and the image sensor 36 (the relative positional relationship between the second optical system 20 and the image sensor 36). In this case, the second lens barrel 21 and the other holding mechanism will move synchronously.

[0034] Furthermore, the imaging magnification of the second optical system 20 (i.e., the magnification of the final image 35) may be changed by means other than the movement of the lenses 26 and 27 in the Z direction as described above. For example, a so-called liquid lens may be provided in the second optical system 20, and the imaging magnification of the second optical system 20 may be changed by changing the voltage applied to the liquid lens. In this case, the second drive unit 33 becomes a voltage control unit that controls the voltage applied to the liquid lens.

[0035] The imaging magnification of the second optical system 20 may also be changed by inserting or removing a predetermined optical system from the second optical system 20. In this case, the second drive unit 33 becomes an insertion / removal unit for inserting or removing the predetermined optical system. The optical system arranged in the optical path of the light from the first optical system 10 (i.e., at least a part of the second optical system 20) may be configured to be interchangeable between an optical system with a first magnification and an optical system with a second magnification, which have different imaging magnifications.

[0036] In this case, the magnification of the final image 35 changes depending on whether the optical system placed on the optical path of the light from the first optical system 10 is a first-magnification optical system or a second-magnification optical system. In this case, the second drive unit 33 acts as an exchange unit that swaps the optical system placed on the optical path of the light from the first optical system 10 between a first-magnification optical system and a second-magnification optical system. Alternatively, the imaging magnification of the optical system (second optical system 20) may be changed by reversing at least a portion of the optical system (second optical system 20) placed on the optical path of the light from the first optical system 10.

[0037] Figure 3 is an enlarged view showing parts of the first optical system 10 and the second optical system 20. The two light beams LB1 and LB2 shown in Figure 3 are light beams that enter the first optical system 10, form part of the intermediate image 18, and enter the second optical system 20, originating from mutually different subjects (not shown).

[0038] The light beam LB1 is directed from the first optical system 10 towards the first location Q1 in the intermediate image formation region 19 where the intermediate image 18 is formed. The light beam LB2 is directed from the first optical system 10 towards the second location Q2 in the intermediate image formation region 19 where the intermediate image 18 is formed. The distance D1 from the optical axis AX1 of the first optical system 10 to the first location Q1 is different from the distance D2 from the optical axis AX1 of the first optical system 10 to the second location Q2. The principal ray PR1 of luminous flux LB1 is the ray that passes through the center of luminous flux LB1, and the principal ray PR2 of luminous flux LB2 is the ray that passes through the center of luminous flux LB2.

[0039] As described above, in the imaging device 1 of the first embodiment, the second optical system 20 re-images at least a portion of the intermediate image 18 of the subject formed by the first optical system 10 to form the final image 35 of the subject on the image sensor 36. Generally, in such a configuration, if the optical axis AX2 of the second optical system 20 is misaligned in a direction that intersects the optical axis AX1 of the first optical system 10, the incident angle of the light beam incident on the second optical system 20 changes, which deteriorates the imaging performance of the second optical system 20.

[0040] In contrast, in the imaging device 1 of the first embodiment, the principal rays PR1, PR2, etc. of each light beam (light beams LB1, LB2, etc.) from the subject imaged by the first optical system 10 are incident on the intermediate image formation region 19 approximately parallel to the optical axis AX1. In other words, the first optical system 10 is so-called telecentric on the intermediate image 18 side. The angle of the direction of propagation of the principal rays (principal rays PR1, PR2, etc.) of the light beams (light beams LB1, LB2, etc.) incident on the intermediate image formation region 19 with respect to the optical axis AX1 may be, for example, within 1°. In this case, the difference between the angle of the principal ray PR1 of the light beam LB1 traveling from the first optical system 10 to the first location Q1 in the intermediate image formation region 19 with respect to the optical axis AX1, and the angle of the principal ray PR2 of the light beam LB2 traveling from the first optical system 10 to the second location Q2 in the intermediate image formation region 19 with respect to the optical axis AX1 can be said to be within 1°.

[0041] Furthermore, the angle between the optical axis AX1 and the direction of propagation of the principal rays (principal rays PR1, PR2, etc.) of the light beams (light beams LB1, LB2, etc.) incident on the intermediate image-forming region 19 may be within 0.5° or within 0.3°. In this case, the difference between the angle between the optical axis AX1 and the principal ray PR1 of the light beam LB1 traveling from the first optical system 10 to the first location Q1 in the intermediate image-forming region 19, and the angle between the optical axis AX1 and the principal ray PR2 of the light beam LB2 traveling from the first optical system 10 to the second location Q2 in the intermediate image-forming region 19, may be within 0.5° or within 0.3°. Also, the angle between the optical axis AX1 and the direction of propagation of the principal rays of all light beams incident on the intermediate image-forming region 19 may be within 0.5° or within 0.3°. Furthermore, the angle between the two principal rays incident at different positions in the intermediate image-forming region 19 and the optical axis AX1 in the direction of propagation may be 0.5° or 0.3° or less.

[0042] Therefore, the principal rays (principal rays PR1, PR2, etc.) of the light beams (light beams LB1, LB2, etc.) that enter the second optical system 20 via the intermediate image 18 also travel approximately parallel to the optical axis AX2 of the second optical system 20 and enter the second optical system 20. For this reason, even if the second optical system 20 is moved by the first drive unit 24 in a direction perpendicular to the optical axis AX1 of the first optical system 10 (in the XY plane), the angle of incidence of the light beams (light beams LB1, LB2, etc.) entering the second optical system 20 hardly changes. Consequently, even if the second optical system 20 is moved in the XY plane relative to the first optical system 10, the second optical system 20 maintains good imaging performance and forms a high-resolution final image 35 on the image sensor 36.

[0043] To make the first optical system 10 telecentric on the intermediate image 18 side, for example, an aperture diaphragm 16 may be provided at a predetermined position. Alternatively, the telecentricity on the intermediate image 18 side may be achieved by limiting the effective diameter of a predetermined lens, such as lens 14, to a predetermined size.

[0044] Furthermore, by making the second optical system 20 telecentric on the intermediate image 18 side, i.e., the first optical system 10 side, it is possible to prevent changes in the incident angle of the incident light beam (light beams LB1, LB2, etc.) to the second optical system 20 due to movement of the second optical system 20 in the XY plane. To achieve this, for example, an aperture diaphragm (not shown) can be provided inside the second optical system 20, and this aperture diaphragm can be set so that the principal rays PR1, PR2, etc. of the light beam (light beams LB1, LB2, etc.) passing through the second optical system 20 travel approximately parallel to the optical axis AX2 of the second optical system 20. Instead of an aperture diaphragm, the effective diameter of the lenses 25-28, etc. that constitute the second optical system 20 can be limited to a predetermined system to make the second optical system 20 telecentric as described above.

[0045] The angle between the optical axis AX2 and the direction of propagation of the principal rays (principal rays PR1, PR2, etc.) of the light beams (light beams LB1, LB2, etc.) incident from the intermediate image-forming region 19 to the second optical system 20 may, for example, be within 1°. In this case, the difference between the angle between the optical axis AX2 and the principal ray PR1 of the light beam LB1 traveling from the first location Q1 in the intermediate image-forming region 19 to the second optical system 20, and the angle between the optical axis AX2 and the principal ray PR2 of the light beam LB2 traveling from the second location Q2 in the intermediate image-forming region 19 to the second optical system 20, can be said to be within 1°.

[0046] Furthermore, the angle between the optical axis AX2 of the direction of propagation of the principal rays (principal rays PR1, PR2, etc.) of the light beams (light beams LB1, LB2, etc.) incident from the intermediate image-forming region 19 to the second optical system 20 may be within 0.5° or within 0.3°. In this case, the difference between the angle between the optical axis AX2 of the principal ray PR1 of the light beam LB1 traveling from the first location Q1 of the intermediate image-forming region 19 to the second optical system 20 and the angle between the optical axis AX2 of the principal ray PR2 of the light beam LB2 traveling from the second location Q2 of the intermediate image-forming region 19 to the second optical system 20 may be within 0.5° or within 0.3°.

[0047] Furthermore, the angle between the optical axis AX2 in the direction of propagation of the principal rays of all light beams incident from the intermediate image-forming region 19 to the second optical system 20 may be 0.5° or less, or 0.3° or less. Also, the angle between the optical axis AX1 in the direction of propagation of two principal rays from different positions among the light beams incident from the intermediate image-forming region to the second optical system 20 may be 0.5° or less, or 0.3° or less.

[0048] Furthermore, both the first optical system 10 and the second optical system 20 may be telecentric on the intermediate image 18 side. In this case, the difference between the angle with respect to the optical axis AX1 in the direction of propagation of the principal rays (principal rays PR1, PR2, etc.) of the light beams (light beams LB1, LB2, etc.) incident from the first optical system 10 to the intermediate image forming region 19 and the angle with respect to the optical axis AX2 in the direction of propagation of the principal rays (principal rays PR1, PR2, etc.) of the light beams (light beams LB1, LB2, etc.) incident from the intermediate image forming region 19 to the second optical system 20 may be, for example, within 1°.

[0049] Furthermore, the optical axis AX1 of the first optical system 10 and the optical axis AX2 of the second optical system 20 in the intermediate image formation region 19 may be approximately parallel. Also, not limited to the intermediate image formation region 19, the optical axis AX1 of the entire first optical system 10 and the optical axis AX2 of the entire second optical system 20 may be approximately parallel.

[0050] Furthermore, the difference between the angle of the principal rays (principal rays PR1, PR2, etc.) of the light beams (light beams LB1, LB2, etc.) incident from the first optical system 10 into the intermediate image-forming region 19 with respect to the optical axis AX1 in the direction of propagation, and the angle of the principal rays (principal rays PR1, PR2, etc.) of the light beams (light beams LB1, LB2, etc.) incident from the intermediate image-forming region 19 into the second optical system 20 with respect to the optical axis AX2 in the direction of propagation, may be within 0.5° or within 0.3°. These conditions may be satisfied for all light beams passing through the intermediate image-forming region 19, or they may be satisfied for just one of the light beams passing through the intermediate image-forming region 19.

[0051] Furthermore, it is not necessary for both the first optical system 10 and the second optical system 20 to be telecentric on the intermediate image 18 side. If either the first optical system 10 or the second optical system 20 is telecentric on the intermediate image 18 side, the other can simply be an optical system with a large numerical aperture (or a bright optical system) that allows light beams (light beams LB1, LB2, etc.) passing through the other optical system and the imaging area IA1 to pass without obstruction.

[0052] In Figure 3, the principal rays PRL and PRR, shown as dotted lines, both represent the principal rays of the light beam that forms an image at the outer edge of the intermediate image 18. Here, the outer edge of the intermediate image 18 corresponds to the outer circumference of the intermediate image forming region 19 of the intermediate image 18 shown in Figure 2. The first optical system 10 has a half-angle of view of angle θ and an angle of view of angle 2θ, centered in the +Z direction which is parallel to the optical axis AX1.

[0053] The field of view of the first optical system 10 may be, for example, 170° or more. Furthermore, since the field of view of the subject captured by the imaging device 1 is narrower than the field of view of the first optical system 10, the field of view of the first optical system 10 can also be called the maximum field of view.

[0054] The control unit 50 of the imaging device 1 will now be described, again referring to Figure 1. The control unit 50 controls various operations of the imaging device 1. The control unit 50 comprises an imaging control unit 51, an image generation unit 52, an analysis unit 53, a field of view control unit 54, a storage unit 55, and an interface unit 56. The imaging control unit 51, image generation unit 52, analysis unit 53, field of view control unit 54, storage unit 55, and interface unit 56 communicate with each other via wiring 58 such as a bus.

[0055] The imaging control unit 51 controls the image sensor 36 by sending a control signal S2 to it. Specifically, the imaging control unit 51 controls the image sensor 36 by starting and ending the output of low-resolution video (through image), starting and ending the output of high-resolution video, capturing still images, setting the light reception time (exposure time) of the final image 35, and setting the gain of the photoelectric conversion.

[0056] The image generation unit 52 receives the imaging signal S1 from the image sensor 36 and generates image data of the final image 35 captured by the image sensor 36. The image generation unit 52 may also have a function to correct distortion of the final image 35 caused by distortion in at least one of the first optical system 10 and the second optical system 20 when generating image data. The image generation unit may also have a function to correct deterioration of the final image 35 caused by aberrations other than distortion in at least one of the optical systems of the first optical system 10 and the second optical system 20. The analysis unit 53 analyzes the subject information contained in the image data of the final image 35 generated by the image generation unit 52. Details of the subject information analyzed by the analysis unit 53 will be described later.

[0057] The field of view control unit 54 sends a drive signal S3 to the first drive unit 24, which drives a drive member such as a linear motor included in the first drive unit 24 (moving the mover 22 relative to the stator 23), and moves the second optical system 20 and the image sensor 36 in a direction that intersects the optical axis AX1 of the first optical system (for example, in the XY plane). The field of view control unit 54 sends a drive signal S4 to the second drive unit 24, which drives the drive members such as linear motors included in the drive mechanisms 31 and 32 of the second drive unit 24, thereby moving the lenses 26 and 27 in the Z direction and changing the magnification of the second optical system 20, that is, the magnification of the final image 35.

[0058] The field of view control unit 54 drives at least one of the first drive unit 24 and the second drive unit 24 based on the subject information analyzed by the analysis unit 53, thereby changing at least one of the center position and size (range) of the imaging area IA1 (field of view of the imaging device 1). It can also be said that the field of view control unit 54 changes at least one of the center position of the field of view of the imaging device 1 and the size of the field of view of the imaging device 1 by driving at least one of the first drive unit 24 and the second drive unit 24.

[0059] Furthermore, the field of view control unit 54 may, regardless of the subject information analyzed by the analysis unit 53, send a drive signal S3 to the first drive unit 24 based on a signal regarding a change in the center position of the field of view input from an operation unit (not shown) via the interface unit 56, thereby moving the second optical system 20 and the image sensor 36.

[0060] The field of view control unit 54 may also send a drive signal S4 to the second drive unit 33 based on a signal regarding the change in the size of the field of view input from an operation unit (not shown) via the interface unit 56, thereby moving the lenses 26 and 27.

[0061] The field of view control unit 54 may also send a drive signal S3 to the first drive unit 24 and a drive signal S4 to the second drive unit 33 based on signals regarding changes in the center position of the field of view and signals regarding changes in the size of the field of view, which are input from an operation unit (not shown) via the interface unit 56.

[0062] Furthermore, the operator may input at least one of the signals related to changing the central position of the field of view and the signal related to changing the size of the field of view to the field of view control unit 54 via the interface unit 56 by operating an operation unit (not shown). The storage unit 55 includes storage materials such as memory elements and magnetic disks, and stores image data of the subject captured by the image sensor 36 and generated by the image generation unit 52 as needed.

[0063] The interface unit 56 outputs image data of the subject captured by the image sensor 36 and generated by the image generation unit 52, or image data stored in the storage unit 55, to an external device via a network line NW. The interface unit 56 may also receive commands from an external device to the imaging device 1. The interface unit 56 may be equipped with a wireless transmission and reception mechanism, and the output of image data and input of commands may be performed wirelessly.

[0064] The imaging control unit 51, image generation unit 52, analysis unit 53, field of view control unit 54, storage unit 55, and interface unit 56 may each be independent, or mechanically separated, hardware. Alternatively, some of the imaging control unit 51, image generation unit 52, analysis unit 53, field of view control unit 54, storage unit 55, and interface unit 56 may be integrated as one or more hardware units.

[0065] Furthermore, at least one of the imaging control unit 51, image generation unit 52, analysis unit 53, field of view control unit 54, storage unit 55, and interface unit 56 may be composed of hardware and software that controls it.

[0066] Figure 4 shows an example of the usage state of the imaging device 1 of the first embodiment. In the example shown in Figure 4, multiple imaging devices 1a to 1c of the first embodiment are installed, for example, on the wall of the living room LM of a home where one or more users live. The multiple imaging devices 1a to 1c of the first embodiment constitute the imaging system 2, which will be described later. The number of imaging devices 1 that make up the imaging system 2 is not limited to the three shown in Figure 4, but may be any number of two or more.

[0067] Furthermore, when using imaging device 1, it is not always necessary to use multiple imaging devices 1a to 1c in combination; it is acceptable to use only one imaging device 1 from among imaging devices 1a to 1c. In this example, the imaging device 1 (1a-1c) generally continuously captures images of subjects in the living room LM, for example, capturing and recording the daily life of a family at opportune moments.

[0068] In the initial or normal state, the imaging device 1, for example, sets the imaging magnification of the second optical system 20 to a low magnification, that is, sets the size of the imaging area IA1 (width WX, width WY) relative to the intermediate image formation area 19 shown in Figure 2 to be larger, in order to image a wide area of ​​the subject. As described above, the position and size of the imaging area IA1 relative to the intermediate image formation area 19 are set by the field of view control unit 54 driving the first drive unit 24 and the second drive unit 22.

[0069] The image generation unit 52 receives the imaging signal S1 from the image sensor 36 and generates image data (wide-area image data) of the final image 35. In the initial state or normal state, the imaging device 1 may set the imaging magnification of the second optical system 20 to the minimum magnification, in other words, maximize the field of view of the imaging device 1, and image the wide-field imaging area IA0 shown in Figure 2.

[0070] Figure 5(a) shows a wide-area image Im1 based on wide-area image data of the living room LM, as an example of an image captured by the imaging device 1. As described above, a wide range of subjects within the living room LM are captured. The analysis unit 53 analyzes the subject information contained in the wide-area image Im1, which is the final image 35 generated by the image generation unit 52.

[0071] For example, the analysis unit 53 analyzes the wide-area image data containing subject information generated by the image generation unit 52 to detect the presence or absence of a region of interest IA in the wide-area image Im1 that should be captured or recorded. The region of interest IA is, as an example, a region in the image data that includes a person, a part of a person such as a person's face, an animal, a part of an animal such as an animal's face, etc.

[0072] Furthermore, the area of ​​interest (IA) is not limited to at least a part of a person or animal, but can be any area that includes the object of interest (e.g., the subject to be imaged or recorded). For example, the area of ​​interest (IA) may include at least a part of items worn or carried by a person (clothing, shoes, bags, canes, etc.), dangerous goods such as guns and explosives, moving objects such as vehicles, ships, aircraft (drones, etc.), or buildings. For example, the area of ​​interest (IA) may include at least a part of the machining chamber of a machine tool (the space where a workpiece is machined with a tool), and at least a part of the tool changing device that exchanges a tool attached to the spindle of the machining chamber for another type of tool.

[0073] For example, the area of ​​interest IA may be defined as the interior of the machining chamber of a machine tool, including at least a portion of the machining tool attached to the spindle of the machining chamber, the workpiece placed on the stage of the machining chamber, the machining point of the workpiece by the machining tool, the chips generated by machining the workpiece by the machining tool, and the cutting fluid applied to the machining area by the machining tool. For example, the area of ​​interest IA may be defined as the interior of a tool exchange device that exchanges a machining tool attached to the spindle of the machining chamber for another type of machining tool, including at least a portion of the stored machining tools.

[0074] For example, the analysis unit 53 may analyze the wide-area image data containing subject information generated by the image generation unit 52 by performing object detection processing using machine learning to detect the region of interest IA from the wide-area image Im1. As another example, the analysis unit 53 may detect the region of interest IA by performing object detection processing using deep learning.

[0075] More specifically, the analysis unit 53 may detect the region of interest IA from the wide-area image Im1 using an object detection algorithm that utilizes a CNN (Convolutional Neural Network). Alternatively, the analysis unit 53 may detect the region of interest IA from the wide-area image Im1 using other object detection processes based on deep learning.

[0076] For example, the analysis unit 53 may use object detection algorithms such as R-CNN (Region with CNN features), Faster R-CNN (Faster Region with CNN features), or Mask R-CNN (Mask Region with CNN features) as region-proposal type algorithms. Furthermore, the analysis unit 53 is not limited to region-proposal type algorithms and may use other deep learning-based object detection algorithms such as YOLO (You Only Look Once) or SSD (Single Shot Multibox Detector).

[0077] Furthermore, the analysis unit 53 may detect the region of interest (IA) using algorithms other than deep learning, such as linear regression, decision trees, and SVM (Support Vector Machine). Also, the analysis unit 53 may detect the region of interest (IA) using existing template matching processes, rather than being limited to machine learning.

[0078] For example, since the wide-area image Im1 shown in Figure 5(a) contains images of multiple people, the analysis unit 53 detects the region containing the multiple people, indicated by the dashed line, as the region of interest IA. Note that the analysis unit 53 is not limited to detecting only one region of interest IA; it may detect multiple regions of interest IA from the image. When the analysis unit 53 detects a region of interest IA, it calculates the position and size of that region of interest IA in the wide-area image Im1. The analysis unit 53 then transmits the position and size of the region of interest IA in the wide-area image Im1 to the field of view control unit 54.

[0079] The field of view control unit 54 receives a signal from the analysis unit 53 and sends a drive signal S3 to the first drive unit 24, moving the second optical system 20 and the image sensor 36 in a direction intersecting the optical axis AX1 (for example, in the XY plane), so that the center position CT of the imaging area IA1 (see Figure 2) approximately coincides with the center position of the region of interest IA. The field of view control unit 54 sends a drive signal S4 to the second drive unit 24, changing the magnification of the second optical system 20, i.e., the magnification of the final image 35, so that the size of the imaging area IA1 approximately coincides with the size of the region of interest IA.

[0080] As a result, an image corresponding to the region of interest IA is formed on the image sensor 36, and the imaging device 1 captures an enlarged image Im2, shown in Figure 5(b), which corresponds to an enlarged image of the region of interest IA in the wide-area image Im1. In the imaging device 1, the change in the central position CT of the imaging area IA1 (see Figure 2) can be performed by moving only the second optical system 20 in the XY plane, without rotating the entire imaging device 1 or the entirety of the first optical system 10 and the second optical system 20. This allows the imaging device 1 to change the central position CT of the imaging area IA1, i.e., change the field of view of the imaging device 1, at high speed.

[0081] The analysis unit 53 also analyzes the enlarged image Im2, that is, the subject in the region of interest IA, and may, for example, detect whether the subject (i.e., a person included in the region of interest) is showing a specific facial expression such as a smile, or whether it is making a specific gesture or hand movement.

[0082] For example, the analysis unit 53 may detect specific facial expressions such as smiles, or specific gestures or hand movements of people (people included in the region of interest) by performing the object detection process described above or existing tracking processes (described later). The analysis unit 53 may also detect specific facial expressions such as smiles, or specific gestures or hand movements of people (people included in the region of interest) by other image processing methods, not limited to the object detection process described above or existing tracking processes.

[0083] Furthermore, not limited to the enlarged image Im2, when the analysis unit 53 analyzes wide-area image data generated by the image generation unit 52 in the initial or normal state, it may detect whether a person or other object included in the region of interest is showing a specific facial expression such as a smile, or performing a specific gesture or hand movement, and may perform one or more of the following actions A through G. If the above facial expression or gesture or hand movement is detected for the subject, the analysis unit 53 may perform one or more of the following actions A through G.

[0084] A: The imaging control unit 51 is instructed to capture a still image on the image sensor 36. B: The imaging control unit 51 is given a command to start or stop the output of low-resolution video data to the image sensor 36. C: A command is issued to the imaging control unit 51 to start or stop the output of high-resolution video data to the image sensor 36.

[0085] D: A command is issued to the memory unit 55 to start or stop storing the image data of the subject captured by the image sensor 36 and generated by the image generation unit 52. E: A command is issued to the memory unit 55 to start or stop adding a predetermined identification signal (flag) to the stored image data.

[0086] F: The interface unit 56 is given a command to start or stop the transmission of image data of the subject captured by the image sensor 36 and generated by the image generation unit 52 to an external device. G: A command is issued to the field of view control unit 54 to widen the imaging area IA1, that is, to decrease the imaging magnification of the second optical system 20, or to narrow the imaging area IA1, that is, to increase the imaging magnification of the second optical system 20.

[0087] Furthermore, if the analysis unit 53 detects the above-mentioned facial expressions, gestures, or hand movements of the subject (for example, a person included in the area of ​​interest) and initiates any of the actions A through G described above, the analysis unit 53 may terminate those actions after a predetermined time has elapsed.

[0088] Furthermore, the analysis unit 53 may initiate any of the operations A through G described above, regardless of the facial expressions, gestures, or hand movements of the subject (for example, a person included in the region of interest), when it detects the region of interest IA from the wide-area image Im1, or when the field of view control unit 54 subsequently determines that the size of the imaging area IA1 is approximately equal to the size of the region of interest IA.

[0089] The analysis unit 53 may also detect whether or not the region of interest IA in the enlarged image Im2 is moving (movement of the subject to be imaged or recorded (person, animal, etc.) included in the region of interest IA). If the region of interest IA is moving, the analysis unit 53 may issue a command to the field of view control unit 54 to move the imaging area IA1 (field of view of the imaging device 1) in accordance with the movement of the region of interest IA.

[0090] Upon receiving this command, the field of view control unit 54 sends a drive signal S3 to the first drive unit 24, causing the second optical system and the image sensor 36 to move. Therefore, even if the subject to be imaged or recorded, which is included in the region of interest IA, moves relative to the imaging device 1, the region of interest IA does not move out of the imaging area IA1 of the imaging device 1, and the imaging device 1 can continue to image the region of interest IA.

[0091] The analysis unit 53 may also issue a command to the field of view control unit 54 to move the imaging area IA1 (field of view of the imaging device 1), as well as a command to change the size of the imaging area IA1 (field of view of the imaging device 1). The field of view control unit 54 may then send a drive signal S3 to the first control unit 24 and a drive signal S4 to the second control unit 33, thereby moving the positions of the second optical system 20 and the image sensor 36, while also moving the positions of the lenses 26 and 27.

[0092] In this case, at least a portion of the period during which the positions of the second optical system 20 and the image sensor 36 move will overlap with at least a portion of the period during which the positions of the lenses 26 and 27 move. However, the entire period during which the positions of the second optical system 20 and the image sensor 36 move does not have to completely overlap with the entire period during which the positions of the lenses 26 and 27 move. In other words, at least a portion of the period during which the imaging area IA1 (field of view of the imaging device 1) is moving may overlap with at least a portion of the period during which the size of the imaging area IA1 (field of view of the imaging device 1) is changing.

[0093] For example, the analysis unit 53 may detect the movement of the region of interest IA between multiple enlarged images Im2 captured by the image sensor 36 at different times by executing an existing tracking process. As an example, the analysis unit 53 may use the image data of the region of interest IA detected from the wide-area image Im1 by the object detection process described above as a template, and use an existing template matching process to detect the region of interest IA from each enlarged image Im2 captured at different times, and calculate the amount and direction of movement (i.e., temporal displacement) of the region of interest IA on the enlarged image Im2.

[0094] The analysis unit 53 then calculates the amount and direction of movement of the imaging region IA1 based on the calculated amount and direction of movement of the region of interest IA, and issues a command to the field of view control unit 54 to move the imaging region IA1. Upon receiving this command, the field of view control unit 54 sends a drive signal S3 corresponding to the amount and direction of movement of the imaging region IA1 to the first drive unit 24, and moves the positions of the second optical system 20 and the image sensor 36.

[0095] The analysis unit 53 does not necessarily have to use the image data of the region of interest IA detected from the wide-area image Im1 as a template. For example, the analysis unit 53 may use the image data of the region of interest IA detected from the enlarged image Im2 captured at a first time by the object detection process described above as a template, detect the region of interest IA from the enlarged image Im2 captured at a second time after a predetermined time, and calculate the amount and direction of movement of the region of interest IA between the first and second time points.

[0096] Furthermore, the analysis unit 53 may calculate the amount and direction of movement of the region of interest IA not limited to template matching processing, but also using other existing tracking processing methods.

[0097] The analysis unit 53 may use existing deep learning to predict the amount and direction of movement of the region of interest IA at a predetermined time interval based on the amount and direction of movement of the region of interest IA at different time intervals calculated by the tracking process described above. In this case, the analysis unit 53 calculates (predicts) the amount and direction of movement of the imaging region IA1 at a predetermined time interval based on the predicted amount and direction of movement of the region of interest IA at a predetermined time interval, and issues a command to the field of view control unit 54 to move the imaging region IA1.

[0098] Furthermore, the analysis unit 53 is not limited to the magnified image Im2, but may also detect whether or not the region of interest IA of any image data captured by the image sensor 36 has moved, and may issue a command to the field of view control unit 54 to move the imaging area IA1 (field of view of the imaging device 1) in accordance with the movement of the region of interest IA. For example, the analysis unit 53 may also detect whether or not the region of interest IA of the wide-area image Im1 has moved.

[0099] For example, if the subject to be imaged or recorded, which is included in the region of interest IA, moves relative to the imaging device 1 in the initial state or normal state, the analysis unit 53 may detect the movement of the region of interest IA in the wide-area image Im1 detected by analyzing the wide-area image data generated by the image generation unit 52, and issue a command to the field of view control unit 54 to move the imaging area IA1 in accordance with the movement of the region of interest IA.

[0100] In this case as well, the analysis unit 53 may calculate the amount and direction of movement (displacement) of the region of interest IA by the tracking process described above. Alternatively, the analysis unit 53 may use existing deep learning to predict the amount and direction of movement of the region of interest IA per predetermined time period based on the amount and direction of movement of the region of interest IA at different time intervals calculated by the tracking process described above.

[0101] Furthermore, not only does the subject to be imaged or recorded, which is included in the region of interest IA, move, but if the imaging device 1 and the subject move relative to each other, the analysis unit 53 detects the movement of the region of interest IA on the image and issues a command to the field of view control unit 54 to move the imaging area IA1 in accordance with the movement of the region of interest IA. This ensures that the region of interest IA does not move out of the imaging area IA1 of the imaging device 1, and that imaging of the region of interest IA continues.

[0102] Furthermore, if the object detection process described above is performed but the region of interest IA cannot be recognized within the enlarged image Im1, the analysis unit 53 may issue a command to the field of view control unit 54 to re-detect the subject to be imaged or recorded, assuming that the subject to be imaged or recorded within the region of interest IA has moved out of the imaging area IA1, and may lower the imaging magnification of the second optical system 20. By lowering the imaging magnification of the second optical system 20, the field of view of the imaging device 1 is widened, thereby expanding the search range for the subject to be imaged or recorded.

[0103] Furthermore, if the distance between the subject to be imaged or recorded and the imaging device 1 changes, the analysis unit 53 may detect whether or not there has been a change in the size of the region of interest IA in the magnified image Im2 (the size of the image of the subject to be imaged or recorded (person, animal, etc.) included in the region of interest IA). If the size of the region of interest IA has changed, the analysis unit 53 may issue a command to the field of view control unit 54 to change the size of the imaging region IA1 in accordance with the change in the size of the region of interest IA. Upon receiving this command, the field of view control unit 54 sends a drive signal S3 to the second drive unit 33 to change the imaging magnification of the second optical system.

[0104] For example, if the analysis unit 53 detects that the size of the region of interest IA has changed, it may issue a command to the field of view control unit 54 to change the imaging area IA1 in order to counteract the change in the size of the region of interest IA. For example, if the analysis unit 53 detects that the region of interest IA has become larger, it may issue a command to the field of view control unit 54 to enlarge the imaging area IA1. Upon receiving this command, the field of view control unit 54 reduces the imaging magnification of the second optical system.

[0105] On the other hand, if the analysis unit 53 detects that the region of interest IA has become smaller, it may issue a command to the field of view control unit 54 to reduce the size of the imaging area IA1. Upon receiving this command, the field of view control unit 54 increases the imaging magnification of the second optical system. Therefore, even if the distance between the imaging device 1 and the subject to be imaged or recorded, which is included in the region of interest IA, changes, the imaging device 1 can continue to image the region of interest IA at a roughly constant size relative to the imaging area IA1 (the field of view of the imaging device 1).

[0106] The analysis unit 53 may also issue a command to the field of view control unit 54 to change the imaging magnification (size of the imaging area IA1) and a command to change the position of the imaging area IA1. The field of view control unit 54 may then send a drive signal S4 to the second control unit 33 and a drive signal S3 to the first control unit 24, moving the positions of the lenses 26 and 27 while moving the positions of the second optical system 20 and the image sensor 36.

[0107] For example, the analysis unit 53 may detect changes in the size of the region of interest IA on the image between multiple enlarged images Im2 captured by the image sensor 36 at different times by performing the object detection process described above. For example, the analysis unit 53 may detect the region of interest IA from each enlarged image Im2 captured at a first time and at a second time a predetermined time has elapsed from the first time by the object detection process described above, and calculate the amount of change in the size of the region of interest IA at the second time relative to the size of the region of interest IA at the first time.

[0108] The analysis unit 53 may then calculate the change in size of the imaging region IA1 based on the calculated change in size of the region of interest IA, and issue a command to the field of view control unit 54 to change the size of the imaging region IA1. Upon receiving this command, the field of view control unit 54 sends a drive signal S3 corresponding to the change in size of the imaging region IA1 to the second drive unit 33, causing the imaging magnification of the second optical system 20 to change.

[0109] Furthermore, the analysis unit 53 may, in addition to the object detection process described above, perform other existing image processing to detect changes in the size of the region of interest IA on the image between multiple enlarged images Im2 captured by the image sensor 36 at different times.

[0110] The analysis unit 53 may use existing deep learning to predict the change in the size of the region of interest IA per predetermined time period based on the change in the size of the region of interest IA between different time periods calculated by the object detection device described above. In this case, the analysis unit 53 calculates (predicts) the change in the size of the imaging area IA1 per predetermined time period based on the predicted change in the size of the region of interest IA per predetermined time period, and issues a command to the field of view control unit 54 to change the size of the imaging area IA1.

[0111] Furthermore, the analysis unit 53 is not limited to the enlarged image Im2, but may also detect whether or not there is a change in the size of the region of interest IA of any image captured by the image sensor 36, and may issue a command to the field of view control unit 54 to move the size of the imaging area IA1 (field of view of the imaging device 1) in accordance with the change in the size of the region of interest IA. For example, the analysis unit 53 may also detect whether or not there is a change in the size of the region of interest IA of the wide-area image Im1.

[0112] For example, if the distance between the subject to be imaged or recorded, which is included in the region of interest IA, and the imaging device 1 changes in the initial or normal state, the image generation unit 52 may detect a change in the size of the region of interest IA in the wide-area image Im1 by analyzing the wide-area image data it has generated, and issue a command to the field of view control unit 54 to change the size of the imaging region IA1 in accordance with the change in the size of the region of interest IA.

[0113] In the example shown in Figure 4, the imaging device 1 (1a-1c) is assumed to be installed in the living room LM, but it is not limited to this. For example, one or more imaging devices 1 may be installed in a predetermined location inside a building to image and record any object (subject to be imaged or recorded). For example, they may be installed in a predetermined location inside a nursery school, school, medical facility, nursing home, conference room, shop, train station, etc., to image and record any object, such as people. Furthermore, one or more imaging devices 1 may be installed in a predetermined location inside a mobile vehicle to image and record any object (subject to be imaged or recorded).

[0114] For example, the device may be installed in a designated location inside a vehicle, ship, aircraft, etc., to image and record any object. Alternatively, one or more imaging devices 1 may be installed in a designated location outdoors to image and record any object (subject to be imaged or recorded), such as people or animals. For example, one or more imaging devices 1 may be installed in a designated location on a building such as a vending machine, streetlamp, telephone pole, bridge, shop entrance, or station entrance to image and record any object.

[0115] Furthermore, one or more imaging devices 1 may be installed at a predetermined location on a moving object such as a vehicle, ship, or aircraft (drone, etc.) to capture and record images of any object (subject to be imaged or recorded), such as people or animals, from the moving object.

[0116] Figure 6 shows an example of an image based on image data generated by imaging by the imaging device 1 in another example of the usage state of the imaging device 1 of the first embodiment. In this example, the imaging device 1 is installed outdoors as an example and used as a surveillance camera to monitor the area around the imaging device 1. However, even when used as a surveillance camera, one or more imaging devices 1 may be installed indoors. One or more imaging devices 1 may be installed as a surveillance camera in predetermined locations where surveillance is necessary, such as shops, train stations, airports, medical facilities, nursing homes, prisons, military facilities, borders, roads, and parks, to image and record any object (subject to be imaged or recorded). One or more imaging devices 1 may also be installed on a moving object such as a vehicle, ship, or aircraft (drone, etc.), to image and record any object that needs to be monitored from the moving object.

[0117] Figure 6(a) shows a wide-area image Im3 based on wide-area image data of the surrounding scenery of the imaging device 1 installed outdoors, as an example of an image captured by the imaging device 1. The analysis unit 53 analyzes the subject information contained in the wide-area image Im3 and detects the presence or absence of a region of interest IA that should be captured or recorded. The analysis unit 53 may also detect the region of interest IA by performing the object detection process described above, in the same manner as the usage state of the imaging device 1 shown in Figures 4 and 5 above.

[0118] The wide-area image Im3 shown in Figure 6(a) contains an image of a person. For example, the analysis unit 53 detects the region containing the image of the person, indicated by the dashed line, as the region of interest IA. The analysis unit 53 calculates the position and size of the region of interest IA in the wide-area image Im3 and transmits this information to the field of view control unit 54.

[0119] The field of view control unit 54 receives a signal from the analysis unit 53 and sends a drive signal S3 to the first drive unit 24, moving the second optical system 20 and the image sensor 36 in the XY plane, so that the center position CT (see Figure 2) of the imaging area IA1 approximately coincides with the center position of the region of interest IA. As a result, an image of the subject corresponding to the region of interest A2 is formed on the image sensor 36, and the imaging device 1 captures an enlarged image Im4 shown in Figure 6(b), which corresponds to an enlarged image of the portion of the region of interest A2 in the wide-area image Im3.

[0120] The analysis unit 53 may also detect whether or not the region of interest IA in the enlarged image Im4 has moved. If the subject to be imaged or recorded (person, animal, etc.) included in the region of interest IA moves relative to the imaging device 1 (for example, if the subject to be imaged or recorded included in the region of interest IA moves), the analysis unit 53 may issue a command to the field of view control unit 54 to move the imaging area IA1 in accordance with the movement of the region of interest IA.

[0121] Upon receiving this command, the field of view control unit 54 sends a drive signal S3 to the first drive unit 24, causing the second optical system 20 and the image sensor 36 to move. Therefore, even if the subject to be imaged or recorded, which is included in the region of interest IA, moves relative to the imaging device 1, the region of interest IA will not move out of the imaging area IA1 of the imaging device 1, and the imaging device 1 will continue to image the region of interest IA.

[0122] The analysis unit 53 may also detect the movement of the region of interest IA between multiple enlarged images Im2 captured by the image sensor 36 at different times by performing the tracking process described above in the same manner as the usage state of the imaging device 1 shown in Figures 4 and 5 above. The analysis unit 53 may also detect the movement of the region of interest IA by other existing image processing methods, not limited to the existing tracking process. The analysis unit 53 may also issue a command to the field of view control unit 54 to change the size of the imaging region IA1, along with a command to move the imaging region IA1.

[0123] Furthermore, the analysis unit 53 may, in the same manner as the usage state of the imaging device 1 shown in Figures 4 and 5 above, use existing deep learning to predict the amount and direction of movement of the region of interest IA at a predetermined time interval based on the amount and direction of movement of the region of interest IA at different time intervals calculated by the tracking process described above. In this case, the analysis unit 53 calculates (predicts) the amount and direction of movement of the imaging region IA1 at a predetermined time interval based on the predicted amount and direction of movement of the region of interest IA at a predetermined time interval, and issues a command to the field of view control unit 54 to move the imaging region IA1.

[0124] Furthermore, the analysis unit 53 may analyze the size of the region of interest IA in the magnified image Im4 in accordance with the relative movement of the subject to be imaged or recorded and the imaging device 1 (for example, the movement of the subject). If the proportion of the region of interest IA in the magnified image Im4 becomes higher than a predetermined proportion, the analysis unit 53 may issue a command to the field of view control unit 54 to reduce the imaging magnification of the second optical system 20.

[0125] On the other hand, if the proportion of the region of interest IA in the magnified image Im4 falls below a predetermined proportion, a command may be issued to the field of view control unit 54 to increase the imaging magnification of the second optical system 20. A case where the proportion of the region of interest IA in the magnified image Im4 falls above a predetermined proportion is, for example, when the distance between the subject to be imaged or recorded and the imaging device 1 decreases, causing at least a portion of the region of interest IA to fall outside the magnified image Im4.

[0126] Furthermore, a case where the proportion of the region of interest IA in the enlarged image Im4 falls below a predetermined proportion is, for example, when the distance between the subject to be imaged or recorded, which is included in the region of interest IA, and the imaging device 1 increases, making it impossible to recognize the region of interest IA on the enlarged image Im4.

[0127] The analysis unit 53 may calculate the size of the region of interest IA in the image using the object detection process described above. The analysis unit 53 may also issue a command to the field of view control unit 54 to change the imaging magnification, along with a command to change the position of the imaging region IA1.

[0128] The analysis unit 53 may calculate the size of the region of interest IA in the magnified image Im4 and issue a command to the field of view control unit 54 to change the imaging magnification of the second optical system 20 so that the ratio of the region of interest IA to the magnified image Im4 becomes a constant value. Alternatively, the analysis unit 53 may calculate the size of the region of interest IA in the image using the object detection process described above.

[0129] Furthermore, even if the object detection process described above is performed, if the region of interest IA cannot be recognized in particular within the enlarged image Im4, the analysis unit 53 may issue a command to the field of view control unit 54 to re-detect the subject to be imaged or recorded, assuming that the subject to be imaged or recorded, which is included in the region of interest IA, has moved out of the imaging area IA1 that captures the enlarged image Im4, and may lower the imaging magnification of the second optical system 20. By lowering the imaging magnification of the second optical system 20, the field of view of the imaging device 1 is widened, thereby expanding the search range for the subject to be imaged or recorded.

[0130] Furthermore, if the region of interest IA cannot be recognized within the magnified image Im4, the analysis unit 53 may, assuming that the subject to be captured or recorded within the region of interest IA has become smaller relative to the imaging area IA1 from which the magnified image Im4 is captured, issue a command to the field of view control unit 54 to re-detect the subject to be captured or recorded, thereby increasing the imaging magnification of the second optical system 20.

[0131] By increasing the imaging magnification of the second optical system 20, even when the subject to be imaged or recorded is located far away from the imaging device 1, the region of interest IA, which includes the subject to be imaged or recorded, can be detected again on the image data. The analysis unit 53 may also issue a command to the field of view control unit 54 to change the position of the imaging region IA1, along with a command to change the imaging magnification.

[0132] Furthermore, if the analysis unit 53 has performed the object detection process described above but cannot recognize the region of interest IA within the enlarged image Im4, it may issue a command to the field of view control unit 54 to change the position of the imaging region IA1 in order to re-detect the subject to be imaged or recorded that is included in the region of interest IA (it may also drive the first drive unit 24). In this case, the analysis unit 53 may issue a command to the field of view control unit 54 so that the position of the imaging region IA1 changes while changing the imaging magnification of the second optical system 20.

[0133] In the example shown in Figure 6, the imaging device 1 may perform the operations A through G described above in the same manner as in the example shown in Figure 5. The analysis unit 53 also analyzes whether the person in the region of interest A2 is carrying a dangerous object such as a gun or bomb, and if a predetermined dangerous object is found to be in possession, it may initiate one of the operations C, D, E, F, or G described above.

[0134] (Effects of the imaging device of the first embodiment) (1) The imaging device 1 of the first embodiment includes a first optical system 10 that forms an intermediate image 18 of a subject, a second optical system 20 that re-images at least a part of the intermediate image 18 to form a final image 35 and can change the magnification of the final image 35, an image sensor 36 that captures the final image 35, and a first drive unit 24 that moves the second optical system 20 and the image sensor 36 in a direction intersecting the optical axis AX1 of the first optical system 10. With this configuration, the imaging device 1 can quickly extract a desired imaging area IA1 from a wide-field imaging area (such as the wide-field imaging area IA0) and capture it with high resolution.

[0135] (2) The first optical system 10 may be telecentric on the intermediate image 18 side. In this case, even if the second optical system 20 moves in a direction that intersects the optical axis AX1 of the first optical system 10, the incident angle of the light beams (light beams LB1, LB2, etc.) incident on the second optical system 20 hardly changes. Therefore, the second optical system 20 can maintain good imaging performance and form a high-resolution final image 35 on the image sensor 36.

[0136] (3) The maximum field of view of the first optical system 10 may be 170° or more. In this case, a wider field of view imaging area (wide field of view imaging area IA0, etc.) can be imaged. (4) The system may further include an analysis unit 53 capable of analyzing the image data of the subject generated by imaging with the image sensor 36.

[0137] (5) The system may further include a field of view control unit 54 that performs at least one of the following based on the results of the analysis by the analysis unit 53: driving the first drive unit 24 and changing the magnification of the final image 35. With this configuration, a desired imaging area IA1 can be rapidly extracted and imaged at high resolution based on information of the subject being imaged by a wide field of view imaging area (wide field of view imaging area IA0, etc.). Alternatively, an image of the subject can be formed within the imaging area IA1 at an appropriate size.

[0138] (6) The system may further include an imaging control unit 51 that controls the start and end of recording of image data generated by imaging with the image sensor 36 based on the results of the analysis by the analysis unit 53. With this configuration, the start and end of recording of image data can be controlled based on information about the subject, such as the facial expression, gestures, or the shape of an object held by the subject being photographed.

[0139] (Imaging device of the second embodiment) Figure 7 shows an overview of the imaging device 1A of the second embodiment. The configuration of the imaging device 1A of the second embodiment is generally the same as that of the imaging device 1 of the first embodiment described above, so the same reference numerals are used for the same components, and their descriptions are omitted as appropriate.

[0140] The imaging device 1A of the second embodiment includes a first optical system 10, a second optical system 20, an image sensor 36, and a control unit 50, as well as a distance measuring unit 40 enclosed by a dashed line. The control unit 50 also includes a virtual image generation unit 57.

[0141] The distance measuring unit 40 measures the distance from the imaging device 1A to at least a portion of the subject. The distance measuring unit 40 includes, as an example, a light-emitting unit 41, a light-transmitting optical system 42, a light-receiving unit 44, a light-receiving optical system 45, and a measuring unit 47. In response to a command from the measuring unit 47, the light-emitting unit 41, such as a laser diode array, pulses and emits measurement light 43 toward the subject via the light-transmitting optical system 42, and the light-receiving unit 44 receives detection light 46, which is the measurement light 43 reflected by at least a portion of the subject, via the light-receiving optical system 45. The light-receiving unit 44 can also be called a detection element because it detects the detection light 46.

[0142] The measurement unit 47 calculates the distance to at least a part of the subject based on the time from the emission of measurement light 43 by the light-emitting unit 41 to the reception of detection light 46 by the light-receiving unit 44, and the speed of light. The measurement unit 47 may be located outside the imaging device 1A. For example, the measurement unit 47 may be located in a control device located outside the imaging device 1A and configured to communicate signals with each part of the imaging device 1A (for example, the light-emitting unit 41, the light-receiving unit 44, and at least a part of the virtual image generation unit 57). The measurement unit 47 can also be called a distance calculation unit because it calculates the distance to at least a part of the subject.

[0143] As an example, the light-receiving unit 44 may be an image sensor having two-dimensional resolution, in which single-photon avalanche diodes are arranged in a two-dimensional grid. In this case, in addition to the distance to the subject, the azimuth angle of the subject relative to the imaging device 1A can also be measured. The measurement unit 47 may be included in the light-receiving unit 44. In this case, the light-receiving unit 44 itself may calculate the distance to at least a part of the subject. For example, if an image sensor is used for the light-receiving unit 44, the distance to at least a part of the subject may be calculated using a processing circuit of that image sensor.

[0144] The measurement unit 47 receives a command via control signal S5 from the virtual image generation unit 57 of the control unit 50 and calculates the distance to the subject. It then transmits information regarding the distance to the subject as a distance signal S6 to the virtual image generation unit 57.

[0145] The field of view of the light transmitting optical system 42 and the light receiving optical system 45 may be the same as the field of view of the first optical system 10 described above, or it may be different from the field of view of the first optical system 10. For example, the field of view (maximum field of view) of the light transmitting optical system 42 and the light receiving optical system 45 may be 170° or more. The wavelengths of the measurement light 43 emitted by the light-emitting unit 41 and the detection light 46 received by the light-receiving unit 44 may be wavelengths included in the wavelength range of the light received by the image sensor 36, or they may be wavelengths different from the wavelength range of the light received by the image sensor 36. For example, the light received by the image sensor 36 may be visible light, and the wavelength of the measurement light 43 emitted by the light-emitting unit 41 may be a wavelength in the infrared region.

[0146] The light transmitting optical system 42 and the light receiving optical system 45 are not separate optical systems, but may be an optical system in which at least a portion of the optical path of the measurement light 43 emitted from the light emitting unit 41 and the optical path of the detection light 46 received by the light receiving unit 44 overlap. For example, an optical system may be combined with an optical path branching element such as a half mirror, with the light emitting unit 41 placed in one optical path branched by the optical path branching element and the light receiving unit 44 placed in the other branched optical path.

[0147] In Figure 1, the optical path from the first optical system 10 through the second optical system 20 to the image sensor 36 is shown as separate from the optical path of the measurement light 43 from the light-emitting unit 41 to the light-transmitting optical system 42, or from the optical path of the detection light 46 from the light-receiving optical system 45 to the light-receiving unit 44. However, at least a portion of these optical paths may overlap. Alternatively, the first optical system 10 and the second optical system 20 may be used as a light-receiving optical system 45, and the image sensor 36 may be used as a light-receiving unit 44.

[0148] The light-emitting unit 41 is not limited to emitting pulsed measurement light 43, but may also emit measurement light 43 whose intensity is modulated over time. In this case, the distance measuring unit 40 may measure the distance to the subject based on the phase of the temporal change of the signal corresponding to the amount of detected light 46 received by the light-receiving unit 44.

[0149] Instead of using an image sensor with the two-dimensional resolution described above as the light-receiving unit 44, the distance measuring unit 40 may measure the distance to the subject by scanning and illuminating the subject with one or more pulsed laser beams or lasers with fluctuating emission intensity. Alternatively, the distance to the subject may be measured using an imaging optical system equipped with an image sensor having a so-called image plane phase-difference focus detection function. The distance measurement unit 40 may be a separate assembly from the imaging device 1A. The distance measurement unit 40 may also include other existing configurations capable of measuring the distance to the subject.

[0150] The virtual image generation unit 57 generates image data (virtual image data) of a subject captured from a position different from the position where the imaging device 1A is located, based on the image data of the subject captured by the image sensor 36 and generated by the image generation unit 52, and the distance to the subject measured by the distance measurement unit 40.

[0151] The virtual image generation unit 57 communicates signals with the imaging control unit 51, image generation unit 52, analysis unit 53, field of view control unit 54, storage unit 55, and interface unit within the control unit 50 via wiring 58 such as a bus. The virtual image generation unit 57 may be an independent, or mechanically separated, piece of hardware.

[0152] Alternatively, it may be integrated as hardware for any of the following: the imaging control unit 51, the image generation unit 52, the analysis unit 53, the field of view control unit 54, the storage unit 55, and the interface unit 56. The virtual image generation unit 57 may consist of hardware and software to control it. The virtual image generation unit 57 may be located outside the imaging device 1A. For example, the virtual image generation unit 57 may be located in a control device located outside the imaging device 1A and configured to communicate signals with each part of the imaging device 1A.

[0153] The memory unit 55 may store image data of the subject captured by the image sensor 36 and generated by the image generation unit 52, or, in place of this image data, image data generated by the virtual image generation unit 57. The interface unit 56 may output to an external device via a network line NW or wireless connection image data generated by the virtual image generation unit 57, along with or in lieu of the image data stored in the storage unit 55, image data captured by the image sensor 36 and generated by the image generation unit 52.

[0154] Figure 8 shows an example of the usage state of the imaging device 1Aa of the second embodiment. In Figure 8, the imaging device 1Aa is positioned in a predetermined position relative to the subject 60, which is, for example, an automobile. The imaging device 1Aa captures image data of the subject 60 using the image sensor 36. That is, based on the imaging signal S1 captured by the image sensor 36, the image generation unit 52 generates image data of the subject 60.

[0155] The distance measuring unit 40 of the imaging device 1Aa measures the distance and azimuth angle from the imaging device 1Aa to a plurality of points P1, ..., Pj, ..., Pn on the subject 60, in roughly synchronized motion with the imaging of the subject 60 described above. Here, n is the total number of points on the subject 60 to be measured, and j is an arbitrary natural number used as an index.

[0156] Based on the measured distances and azimuth angles between multiple points P1 to Pn, the virtual image generation unit 57 of the control unit 50 of the imaging device 1Aa calculates the three-dimensional positional relationship of each part of the subject 60 with respect to the imaging device 1Aa using a known method. As an example, the virtual image generation unit 57 calculates the XYZ coordinate values ​​of each part (points P1 to Pn) of the subject 60, with the position where the imaging device 1Aa is located as the origin.

[0157] The virtual image generation unit 57 generates virtual image data, which is virtual image data obtained when the subject 60 is imaged from virtual imaging devices 1v1 and 1v2 located at different positions from the actual imaging device 1Aa, based on the image data of the subject 60 and the XYZ coordinate values ​​of each point P1 to Pn. The position of the virtual imaging device 1v1 or virtual imaging device 1v2 may be input to the imaging device 1a by the user as a relative position to the imaging device 1Aa, for example.

[0158] Furthermore, the imaging device 1Ab shown in Figure 8, together with the imaging device 1Aa, constitutes the imaging system 2A of the second embodiment, and its details will be described later.

[0159] Figure 9(a) shows image Im5, which is an example of an image based on image data of a subject 60 generated by imaging by the imaging device 1Aa of the second embodiment, and Figure 9(b) shows virtual image Im6, which is an example of a virtual image based on virtual image data generated by the virtual image generation unit 57. In image Im5, for example, the subject 60 is a car and the imaging device 1Aa are close together, so the perspective of the subject 60 is exaggerated, that is, the front of the car (subject 60) is excessively large compared to the rear.

[0160] The virtual image Im6 shown in Figure 9(b) is a virtual image based on virtual image data generated by the virtual image generation unit 57, and corresponds to an image obtained when the subject is captured from a virtual imaging device 1v1 located further away from the subject 60 than the imaging device 1Aa. Compared to image Im5, virtual image Im6 is a preferable image in which the perspective of the subject is corrected.

[0161] Figure 10(a) shows image Im7, which is an example of an image of a subject 60 captured by the imaging device 1Aa of the second embodiment, and Figure 10(b) shows virtual image Im8, which is an example of a virtual image based on virtual image data generated by the virtual image generation unit 57. Image Im7 was captured by imaging device 1a (see Figure 8), which is positioned at a location with little difference in Z position relative to the subject 60, which is an automobile, for example. Therefore, in image Im7, the structure of the upper surface (-Z side surface) of the subject 60 is compressed and represented.

[0162] The virtual image Im8 shown in Figure 10(b) is a virtual image based on virtual image data generated by the virtual image generation unit 57, and corresponds to an image obtained when the subject is captured from a virtual imaging device 1v2 that is closer to the subject 60 than the imaging device 1Aa and is on the -Z side than the imaging device 1a. Compared to image Im7, virtual image Im8 is an image of the subject 60 viewed from above (-Z side), and the top surface of the subject 60 is represented in more detail.

[0163] In summary, the virtual image generation unit 57 generates image data (virtual image data) of the subject 60 as if it were captured from a position different from the position of the imaging device 1Aa, based on the image data of the subject captured by the image sensor 36 and the distance to at least a part of the subject 60 measured by the distance measurement unit 40.

[0164] (Effects of the imaging device in the second embodiment) (7) The imaging device 1A of the second embodiment includes, in addition to the imaging device 1 of the first embodiment described above, a distance measuring unit 40 that measures the distance to at least a part of the subject. It also includes a virtual image generation unit that generates image data (virtual image data) of the subject as if it were photographed from a position different from the position of the imaging device 1, based on the image data of the subject generated by the imaging of the image sensor 36 and the distance to at least a part of the subject measured by the distance measuring unit 40. With this configuration, the imaging device 1A of the second embodiment can generate virtual image data (virtual images Im6, Im8) of a subject captured from a position different from the actual position where the imaging device 1A is located.

[0165] The imaging device 1 of the first embodiment, or the imaging device 1A of the second embodiment, does not need to have one or more of the imaging control unit 51, image generation unit 52, analysis unit 53, field of view control unit 54, storage unit 55, or interface unit 56 that are included in the control unit 50 in the above description.

[0166] For example, one or more of the imaging control unit 51, image generation unit 52, analysis unit 53, field of view control unit 54, storage unit 55, or interface unit 56 may be located outside the imaging devices 1 and 1A. For example, one or more of the imaging control unit 51, image generation unit 52, analysis unit 53, field of view control unit 54, storage unit 55, or interface unit 56 may be located in a control device located outside the imaging devices 1 and 1A, and may be configured to communicate signals with each part of the imaging devices 1 and 1A.

[0167] In the imaging device 1 of the first embodiment, or the imaging device 1A of the second embodiment, the first optical system 10 and the second optical system 20 do not necessarily have to be telecentric optical systems on the intermediate image 18 side. Also, the second optical system 20 does not necessarily have to have a second drive unit 33. Alternatively, the second optical system 20 and the image sensor 36 do not necessarily have to be held by a holding unit 21.

[0168] In the imaging device 1A of the second embodiment, a part of the optical system of the distance measuring unit 40 may be shared with the first optical system 10 and the second optical system 20. Figure 11 shows a distance measuring unit 40 in which a portion of the optical system is shared with the first optical system 10 and the second optical system 20. In Figure 11, components that achieve the same function as in the above-described embodiment are denoted by the same reference numerals.

[0169] In Figure 11, the measurement light 43 from the light-emitting unit 41 constituting the distance measurement unit 40 is emitted along the optical axis AX3 which intersects the optical axis AX2 of the second optical system 20, and is reflected by a half mirror 61 that is positioned at an angle to the optical axes AX2 and AX3. The measurement light 43 is then reflected by a mirror 62 located on the opposite side of the second optical system 20 with respect to the optical axis AX2, passes through the half mirror 61, and enters the lens 28 of the second optical system 20.

[0170] The measurement light 43 incident on the second optical system 20 is projected onto the subject via lenses 27, 26, and 25 (not shown) of the second optical system 20 and the first optical system 10 (see Figure 7 for all). The detection light 46, which is the measurement light 43 reflected by at least a portion of the subject, reaches the half mirror 61 via the first optical system 10 and the second optical system 20 (see Figure 7 for all). The detection light 46 reflected by the half mirror 61 is then reflected by the half mirror 63 and reaches the light receiving unit 44 that constitutes the distance measuring unit 40.

[0171] On the other hand, the light LB from the subject for forming an image on the image sensor 36 is reflected by the half mirror 61, then transmitted through the half mirror 63, and reaches the image sensor 36 via the first optical system 10 and the second optical system 20. In this way, at least a portion of the light transmitting optical system 42 and the light receiving optical system 45 that constitute the distance measuring unit 40 may be shared with the first optical system 10 and the second optical system 20.

[0172] (Imaging system of the first embodiment) The imaging system 2 of the first embodiment includes a plurality of imaging devices 1 (1a to 1c) of the first embodiment, as shown in Figure 4. The plurality of imaging devices 1 (1a to 1c) constituting the imaging system 2 communicate with each other via a network line NW (see Figure 1) or wirelessly. The number of imaging devices 1 included in the imaging system 2 is not limited to the three shown, but may be any number.

[0173] In the imaging system 2, when one of the multiple imaging devices 1a to 1c detects a region of interest IA, it may transmit information regarding the location of that region of interest IA to the other imaging devices 1a to 1c in the imaging system 2. The information regarding the location of the region of interest IA shared by the multiple imaging devices 1Aa to 1Ac may be, for example, the azimuth angle of a subject included in the region of interest IA with respect to one imaging device 1Aa, or the coordinates of the region of interest IA in the XYZ coordinate system virtually defined by the imaging system 2.

[0174] This allows other imaging devices 1b and 1c within the imaging system 2 to easily image subjects included in the region of interest IA. Furthermore, the same subject included in the region of interest IA can be imaged and recorded from different viewpoints using each imaging device 1a to 1c.

[0175] (Imaging system of the second embodiment) The imaging system 2A of the second embodiment includes a plurality of imaging devices 1A (1Aa to 1Ac) of the second embodiment, as shown in Figure 8. The plurality of imaging devices 1Aa to 1Ac constituting the imaging system 2A communicate with each other via a network line NW (see Figure 7) or wirelessly. The number of imaging devices 1A included in the imaging system 2A is not limited to the three shown, but may be any number.

[0176] In the imaging system 2A of the second embodiment, as in the imaging system 2 of the second embodiment described above, when one of the multiple imaging devices 1Aa to 1Ac detects a region of interest IA, it may transmit information regarding the location of that region of interest IA to the other imaging devices 1Aa to 1Ac in the imaging system 2A.

[0177] In the imaging system 2A of the second embodiment, not only the azimuth angle of a subject included in the region of interest IA, but also the distance to the subject can be shared with respect to one imaging device 1Aa. This makes it easier for the other imaging devices 1Ab and 1Ac in the imaging system 2A to image subjects included in the region of interest IA.

[0178] The virtual image data described above may be generated using the imaging system 2A of the second embodiment. In this case, two or more of the multiple imaging devices 1Aa to 1ac will image the subject 60 and measure the distance and azimuth angle to at least a part of the subject 60. This information will then be shared among the multiple imaging devices 1Aa to 1ac. Based on this information, the virtual image generation unit 57 included in the imaging devices 1Aa to 1Ac may generate virtual image data that corresponds to the image obtained when the subject 60 is imaged from a virtual imaging device 1v1 or Iv2.

[0179] Furthermore, the imaging device 1 of the first embodiment, the imaging device 1A, imaging system 2, or imaging system 2A of the second embodiment described above may be installed in designated locations such as nurseries, schools, medical facilities, nursing care facilities, conference rooms, shops, train stations, parks, vending machines, streetlights, telephone poles, prisons, military facilities, borders, roads, and parks. When the imaging device 1, 1A, or imaging system 2, 2A is installed in a nursery, the imaging device 1, 1A, or imaging system 2, 2A may, for example, use a part of a child's body as the region of interest IA and change the position and size of the imaging area IA1 (field of view of the imaging device 1) (magnification of the final image 35) to follow changes in the position and size of the region of interest IA due to the child's activities.

[0180] Furthermore, when the imaging device 1, 1A, or imaging system 2, 2A is installed in a nursing care facility, the imaging device 1, 1A, or imaging system 2, 2A may change the position and size of the imaging region IA1 to follow changes in the position and size of the region of interest IA, such as changes in the posture of the person requiring care, for example, by designating the face or a part of the body of the person requiring care as the region of interest IA.

[0181] Furthermore, when the imaging device 1, 1A, or imaging system 2, 2A is installed in a conference room, for example, the faces of conference attendees may be used as the region of interest (IA), and the position and size of the imaging area IA1 may be changed to follow changes in the position and size of the region of interest (IA) due to changes in the posture of conference attendees.

[0182] Furthermore, when the imaging device 1, 1A, or imaging system 2, 2A is installed near a station ticket gate, near a shop's passageway or entrance, near a vending machine, streetlamp, or utility pole, the imaging device 1, 1A, or imaging system 2, 2A may change the position and size of the imaging area IA1 to follow the changes in the position and size of the region of interest IA as the person moves, with the passing person being the region of interest IA. Furthermore, the imaging device 1, 1A, or imaging system 2, 2A may detect the characteristics of the passing person based on the image data of the region of interest IA using the object detection process described above or other existing image processing methods.

[0183] For example, the imaging device 1, 1A, or imaging system 2, 2A may detect at least one of the following characteristics of a passing person based on image data of the region of interest IA: gender, age, body type (height, weight, etc.), race, hairstyle, clothing, speed of movement, direction of movement, etc. In this case, the detected characteristics of the passing person can be used for market research, etc. Note that the imaging device 1, 1A, or imaging system 2, 2A may detect characteristics of moving objects other than passing people, such as passing vehicles.

[0184] For example, the imaging device 1, 1A, or imaging system 2, 2A may detect at least one of the following: the type of vehicle, color, speed, and direction of movement of a passing vehicle. The imaging device 1, 1A, or imaging system 2, 2A may be set to a wide-field imaging area (wide-field imaging area IA0, etc.) until at least a portion of a passing person or vehicle is in its field of view. When at least a portion of a person or vehicle enters the field of view of the imaging device 1, 1A, or imaging system 2, 2A, the object detection process described above or other existing image processing may be used to detect the region of interest IA (i.e., an image of at least a portion of the passing person or vehicle), and the position and size of the imaging area IA1 may be adjusted to match the detected region of interest IA.

[0185] Furthermore, the imaging device 1 of the first embodiment, the imaging device 1A, imaging system 2, or imaging system 2A of the second embodiment described above do not need to be fixedly installed indoors or outdoors, and may be installed on a mobile device such as a vehicle, ship, or aircraft (e.g., a drone).

[0186] Furthermore, when the imaging device 1, 1A, or imaging system 2, 2A is installed on an unmanned aerial vehicle such as a drone, the object being searched for (e.g., a person, animal, vehicle, ship, aircraft, etc.) may be used as the region of interest IA. For example, the imaging device 1, 1A, or imaging system 2, 2A may search within a wide-field imaging area (wide-field imaging area IA0, etc.) until the object being searched comes into view as the unmanned aerial vehicle flies. When at least a portion of the object being searched enters the field of view of the imaging device 1, 1A, or imaging system 2, 2A, the region of interest IA (i.e., an image of at least a portion of the object being searched) may be detected by the object detection process described above or other existing image processing, and the position and size of the imaging area IA1 may be adjusted to match the detected region of interest IA. Furthermore, the imaging device 1, 1A, or imaging system 2, 2A may change the position and size of the imaging area IA1 to follow the changes in the position and size of the region of interest IA due to the relative movement of the object being searched and the unmanned aerial vehicle.

[0187] Furthermore, when the imaging device 1, 1A, or imaging system 2, 2A is installed on an attack drone (for example, an attack drone) as an example of an unmanned aerial vehicle, the object to be attacked (for example, a person, vehicle, ship, aircraft, building, etc.) may be designated as the region of interest (IA). For example, the imaging device 1, 1A, or imaging system 2, 2A may search for the target object in a wide-field imaging area (wide-field imaging area IA0, etc.) through the flight of the unmanned aerial vehicle until the object to be attacked comes into view.

[0188] Furthermore, if at least a portion of the target object enters the field of view of imaging device 1, 1A, or imaging system 2, 2A, the object detection process described above or other existing image processing may detect the region of interest IA (i.e., an image of at least a portion of the target object), and the position and size of imaging area IA1 may be adjusted to match the detected region of interest IA. Based on the position and size of the region of interest IA in the image, the attack unmanned aerial vehicle may approach the target object until it enters the attack range of the attack unmanned aerial vehicle, and then attack the target object. The imaging device 1, 1A, or imaging system 2, 2A may also change the position and size of imaging area IA1 to follow changes in the position and size of the region of interest IA due to the relative movement of the target object and the unmanned aerial vehicle.

[0189] For example, it is conceivable to mount two cameras (a wide-angle camera and a telephoto camera with a gimbal) on an unmanned aerial vehicle such as a drone, but this is undesirable because, due to limitations on payload weight and size, objects mounted on unmanned aerial vehicles must be small and lightweight. On the other hand, imaging devices 1, 1A, or imaging systems 2, 2A, which can capture both wide-angle and telephoto images with a single unit, are smaller and lighter than two cameras, making them suitable for mounting on unmanned aerial vehicles.

[0190] Furthermore, the objects to be imaged by imaging devices 1, 1A, or imaging systems 2, 2A are not limited to the people and animals mentioned above, but may also be industrial machinery such as machine tools. In this case, imaging devices 1, 1A, or imaging systems 2, 2A analyze whether the industrial machinery is functioning normally based on the captured image data. If an abnormality is found in the shape or operation of the industrial machinery, imaging devices 1, 1A, or imaging systems 2, 2A take action such as transmitting image data of the abnormal part to an external device.

[0191] Furthermore, the imaging device 1, 1A, or imaging system 2, 2A may be installed inside the machining chamber of the machine tool (the space where the workpiece is machined with the machining tool). In this case, for example, the region of interest IA may be a region that includes the machining tool to which the spindle is attached, the workpiece placed on the stage, and at least a portion of the machining points of the workpiece by the machining tool.

[0192] The imaging device 1, 1A, or imaging system 2, 2A may change the position and size (magnification of the final image 35) of the imaging region IA1 (field of view of imaging device 1) to follow changes in the position and size of the region of interest IA due to the relative movement of the main axis and the workpiece (stage). The imaging device 1, 1A, or imaging system 2, 2A may also detect the movement and size changes of the region of interest IA and change the position and size of the imaging region IA1 (field of view of imaging device 1) by appropriately combining and executing the processes described in each of the embodiments described above.

[0193] Furthermore, the imaging device 1, 1A, or imaging system 2, 2A may detect at least one of the following based on image data of the region of interest IA generated by existing image processing: the distance between the tool and the workpiece, the length of the tool (amount of wear), the shape of the tool, the breakage (fracture) of the tool, the amount of tool protrusion, the shape of the workpiece, the positional displacement of the workpiece (relative to the reference position of the stage), the shape of the chips, the amount of chips, and the amount of cutting fluid applied to the machined area.

[0194] For example, the imaging device 1, 1A, or imaging system 2, 2A may be installed in at least one of the following locations: the wall of the machining room, the ceiling of the machining room, the spindle head, and the stage. For example, the imaging device 1 or 1A may be installed on the spindle.

[0195] The imaging device 1 or 1A may be configured to be detachably attached to the spindle via a shank. In this case, the imaging device 1 or 1A attached to the spindle may be replaced with any workpiece by a tool changing device. Alternatively, the imaging device 1 or 1A may be stored in the tool changing device, and the workpiece attached to the spindle may be replaced with the imaging device 1 or 1A.

[0196] When the imaging device 1 is mounted on the main spindle, the imaging device 1 mounted on the main spindle may be configured such that the second optical system 20, second housing 21, first drive unit 24, second drive unit 33, and field of view control unit 54 are omitted from the configuration shown in Figure 1, and the imaging device 36 is positioned on the image plane of the subject imaged by the first optical system 10 (i.e., the intermediate image formation region 19) (i.e., an imaging device with a wide field of view in which the field of view cannot be moved or resized). In this configuration, the analysis unit 53 or interface unit 56 may be omitted. The image generation unit 52 may be located outside the imaging device 1. For example, the image generation unit 52 may be located in a control device located outside the imaging device 1 and configured to communicate signals with each part of the imaging device 1 (e.g., the image sensor 36).

[0197] Furthermore, when the imaging device 1A is mounted on the main shaft, the imaging device 1A mounted on the main shaft may have the configuration shown in Figure 7 or Figure 11. In this case, the imaging device 1A mounted on the main shaft may have the second optical system 20, second housing 21, first drive unit 24, second drive unit 33, and field of view control unit 54 omitted from the configuration shown in Figure 7 or Figure 11, and the imaging device 36 may be positioned on the image plane of the subject imaged by the first optical system 10 (i.e., the intermediate image formation region 19) (i.e., an imaging device with a wide field of view that can measure distance but whose field of view cannot be moved or its size changed).

[0198] In this configuration, the analysis unit 53, interface unit 56, and virtual image generation unit 57 are not required. The measurement unit 47 may be located outside the imaging device 1A. For example, the measurement unit 47 may be located in a control device located outside the imaging device 1A and configured to communicate signals with each part of the imaging device 1A (for example, the light-emitting unit 41, the light-receiving unit 44, and at least a part of the virtual image generation unit 57).

[0199] The image generation unit 52 may be located outside the imaging device 1A. For example, the image generation unit 52 may be located in a control device positioned outside the imaging device 1A and configured to communicate signals with each part of the imaging device 1A (for example, the image sensor 36).

[0200] Furthermore, the imaging device 1, 1A, or imaging system 2, 2A may be installed in two or more locations among the walls of the processing room, the ceiling of the processing room, the spindle head, and the stage. For example, they may be installed in different locations on the walls of the processing room, or they may be installed in one or more locations on the walls of the processing room and one or more locations on the ceiling of the processing room.

[0201] When multiple imaging devices 1, 1A, or imaging systems 2, 2A are installed in this manner, they may be arranged so that the region of interest within the processing room can be imaged from different directions (from different viewpoints). Furthermore, when multiple imaging devices 1, 1A, or imaging systems 2, 2A are installed, they may be arranged so that their respective maximum field of view areas partially overlap.

[0202] Furthermore, the imaging device 1, 1A, or imaging system 2, 2A may be installed in a part of the machine tool other than the machining chamber. For example, it may be installed inside a tool changing device that exchanges a machining tool attached to the spindle in the machining chamber for a different type of machining tool. The tool changing device may, for example, be an automatic tool changer (ATC).

[0203] Furthermore, the imaging device 1, 1A, or imaging system 2, 2A may be installed inside an optical processing apparatus that processes a workpiece with processing light. In this case, the imaging device 1, 1A, or imaging system 2, 2A may detect the processing area (the area irradiated with processing light) on the workpiece based on image data of the region of interest IA generated by existing image processing. If the optical processing apparatus is a three-dimensional additive manufacturing apparatus, the imaging device 1, 1A, or imaging system 2, 2A may also detect the supply status of the molding material based on image data of the region of interest IA generated by existing image processing.

[0204] (Effects of the imaging systems of the first and second embodiments) (9) The imaging system 2 of the first and second embodiments described above comprises a plurality of imaging devices 1(1a~1c) of the first embodiment or imaging devices 1A(1Aa~1Ac) of the second embodiment, wherein at least one of the plurality of imaging devices 1a~1c, 1Aa~1Ac captures an image of a subject using information from an imaging device different from at least one other imaging device. This configuration allows, for example, one imaging device 1a, 1Aa to share information about the location of the region of interest IA identified within the subject with the other imaging devices 1b, 1c, 1Ab, 1Ac, enabling imaging devices 1b, 1c, 1Ab, 1Ac to easily image the subject included in the region of interest IA. Alternatively, imaging devices 1a-1c and 1Aa-1Ac can each image and record the same subject included in the region of interest IA from different viewpoints. In the example above, there was only one domain of interest (IA), but there may be multiple domains of interest (IA).

[0205] Although various embodiments and modifications have been described above, the present invention is not limited to these. Furthermore, each embodiment and modification may be applied individually or in combination. Other embodiments conceivable within the scope of the technical idea of ​​the present invention are also included within the scope of the present invention.

[0206] (Note) Those skilled in the art will understand that the above-described embodiments or their variations are specific examples of the following embodiments.

[0207] (Section 1) An imaging device comprising: a first optical system for forming an intermediate image of a subject; a second optical system for re-imaging at least a portion of the intermediate image to form a final image; an image sensor for capturing the final image; and a drive unit for moving at least one of the first optical system, the second optical system, and the image sensor in a direction intersecting the optical axis of the first optical system.

[0208] (Section 2) The imaging apparatus described in paragraph 1, wherein the first optical system is telecentric on the intermediate image side.

[0209] (Section 3) The imaging apparatus described in paragraph 2, wherein the difference between the angle of the principal ray of light traveling from the first optical system to a first location in the intermediate image-forming region with respect to the optical axis of the first optical system and the angle of the principal ray of light traveling from the first optical system to a second location in the intermediate image-forming region, which is at a different distance from the optical axis of the first optical system than the first location, is 1° or less.

[0210] (Section 4) An imaging device according to paragraph 1 or 2, wherein the second optical system is telecentric on the first optical system side.

[0211] (Section 5) The imaging apparatus described in paragraph 4, wherein the difference between the angle of the principal ray of light traveling from a first location in the intermediate image-forming region to the second optical system with respect to the optical axis of the second optical system and the angle of the principal ray of light traveling from a second location in the intermediate image-forming region, where the distance of the second optical system from the optical axis is different from that of the first location, with respect to the optical axis of the second optical system is 1° or less.

[0212] (Section 6) An imaging device according to any one of paragraphs 1 to 5, wherein the difference between the angle of the principal ray of light from the first optical system with respect to the optical axis of the first optical system and the angle of the principal ray of light incident on the second optical system with respect to the optical axis of the second optical system is within 1°.

[0213] (Section 7) An imaging device according to any one of paragraphs 1 to 6, wherein the maximum angle of view of the first optical system is 170° or more.

[0214] (Section 8) An imaging device according to any one of paragraphs 1 to 7, wherein the drive unit moves the second optical system and the image sensor in a direction intersecting the optical axis of the first optical system.

[0215] (Section 9) The imaging device described in paragraph 8 further comprises a holding unit for holding the second optical system and the image sensor, wherein the drive unit moves the holding unit in a direction intersecting the optical axis of the first optical system.

[0216] (Section 10) An imaging device according to paragraph 8 or 9, wherein the drive unit moves the second optical system and the image sensor in a direction parallel to a plane perpendicular to the optical axis of the first optical system.

[0217] (Section 11) An imaging device according to any one of paragraphs 1 to 10, wherein the second optical system is an imaging device capable of changing the magnification of the final image.

[0218] (Section 12) The imaging device described in paragraph 11, wherein the second optical system includes a plurality of optical members, and when the drive unit is a first drive unit, the imaging device further includes a second drive unit that moves at least one of the plurality of optical members along the optical axis of the second optical system, and the second optical system changes the magnification of the final image by driving the second drive unit.

[0219] (Section 13) An imaging apparatus according to paragraph 11 or 12, further comprising a field of view control unit that performs at least one of driving the drive unit and changing the magnification of the final image based on image data generated by imaging of the image sensor.

[0220] (Section 14) The imaging apparatus described in paragraph 13 further comprises an imaging control unit that controls the start and end of recording of image data generated by imaging of the image sensor, based on the image data generated by imaging of the image sensor.

[0221] (Section 15) An imaging device according to paragraph 13 or 14, wherein the field of view control unit drives the drive unit such that when at least one of the subject and the imaging device moves, the region of interest of the subject does not move out of the field of view of the imaging device.

[0222] (Section 16) An imaging device according to any one of paragraphs 13 to 15, wherein the field of view control unit reduces the magnification of the final image when at least one of the subject and the imaging device moves and the region of interest of the subject moves out of the field of view of the imaging device.

[0223] (Section 17) An imaging device according to any one of paragraphs 13 to 16, wherein the field of view control unit changes the magnification of the final image according to the size of the region of interest of the subject relative to the field of view of the imaging device.

[0224] (Section 18) The imaging device described in paragraph 17, wherein the field of view control unit changes the magnification of the final image so that the size of the region of interest of the subject relative to the field of view of the imaging device remains constant.

[0225] (Section 19) An imaging device according to any one of paragraphs 13 to 19, further comprising an analysis unit for analyzing image data generated by imaging with the image sensor, wherein if the analysis unit cannot recognize an image of at least a portion of the subject, the field of view control unit performs at least one of driving the drive unit and changing the magnification of the final image.

[0226] (Section 20) In the imaging apparatus described in paragraph 19, the field of view control unit is an imaging apparatus that reduces the magnification of the final image.

[0227] (Section 21) In the imaging device described in paragraph 19, the field of view control unit is an imaging device that increases the magnification of the final image.

[0228] (Section 22) An imaging device as described in paragraph 13, wherein the field of view control unit performs the driving of the drive unit and the change of the magnification of the final image based on image data generated by imaging of the image sensor.

[0229] (Section 23) An imaging device according to any one of paragraphs 1 to 10, further comprising a field of view control unit that drives the drive unit based on image data generated by imaging with the image sensor.

[0230] (Section 24) The imaging device described in paragraph 23, wherein the field of view control unit controls the start and end of recording of image data generated by imaging of the image sensor, based on the image data generated by imaging of the image sensor.

[0231] (Section 25) An imaging device according to paragraph 23 or 24, wherein the field of view control unit drives the drive unit such that when at least one of the subject and the imaging device moves, the region of interest of the subject moves out of the field of view of the imaging device.

[0232] (Section 26) An imaging device according to any one of paragraphs 1 to 25, further comprising an analysis unit capable of analyzing image data of the subject generated by imaging with the image sensor.

[0233] (Section 27) An imaging device comprising an image sensor, a first optical system for forming an intermediate image of a subject, a second optical system for re-imaging at least a portion of the intermediate image to form a final image on the image sensor, and a variable central position of the portion of the intermediate image that is re-imaging on the image sensor by the second optical system.

[0234] (Section 28) The imaging apparatus described in paragraph 27, wherein the second optical system is capable of changing the magnification of the final image, and the size of the portion of the intermediate image that is re-imaged onto the image sensor by the second optical system is variable.

[0235] (Section 29) An imaging device comprising: a light source that emits measuring light for measuring the distance to the subject; an optical system having a maximum field of view of 170° or more and irradiating the subject with the measuring light from the light source; a detection element that detects detection light generated from the subject as a result of the measuring light irradiating the subject, via the optical system; and an image sensor that captures an image of the subject formed by the optical system.

[0236] (Section 30) An imaging device according to paragraph 29, wherein at least a portion of the optical paths of the detected light from the optical system detected by the detection element and the optical paths of the light from the optical system received by the image sensor are superimposed.

[0237] (Section 31) An imaging device according to paragraph 29 or 30, wherein the wavelength of the detection light detected by the detection element is different from the wavelength of the light received by the image sensor.

[0238] (Section 32) An imaging system comprising: an imaging device as described in any one of paragraphs 29 to 31; a distance calculation unit that calculates the distance to at least a portion of the subject based on the detection result of the detection element of the imaging device; and a virtual image generation unit that generates image data of the subject as if it were imaged from a position different from the position of the imaging device, based on image data of the subject generated by imaging with the image element of the imaging device and the distance calculation result to at least a portion of the subject by the distance calculation unit.

[0239] (Section 33) An imaging system comprising: a light source that emits measurement light; an optical system having a maximum field of view of 170° or more that irradiates the measurement light from the light source onto the subject; a detection element that detects detection light generated from the subject as a result of the measurement light irradiating the subject onto the subject, via the optical system; a distance calculation unit that calculates the distance to at least a part of the subject based on the detection light detected by the detection element; and an image sensor that captures an image of the subject formed by the optical system.

[0240] (Section 34) An imaging system according to paragraph 33, wherein at least a portion of the optical paths of the detected light from the optical system detected by the detection element and the optical paths of the light from the optical system received by the image sensor are superimposed.

[0241] (Section 35) An imaging device according to paragraph 33 or 34, wherein the wavelength of the detection light detected by the detection element is different from the wavelength of the light received by the image sensor.

[0242] (Section 36) An imaging system according to any one of paragraphs 33 to 35, further comprising a virtual image generation unit that generates image data of the subject when it is imaged from a position different from the position of the image sensor, based on the image data of the subject generated by imaging with the image sensor and the distance calculation result of the distance calculation unit up to at least a part of the subject.

[0243] (Section 37) An imaging device comprising: a light source that emits measurement light; an optical system having a maximum field of view of 170° or more that irradiates the measurement light from the light source onto the subject; a detection element that detects detection light generated from the subject as a result of the measurement light irradiating the subject onto the subject via the optical system; and an image sensor that captures an image of the subject formed by the optical system, wherein the device calculates the distance to at least a part of the subject based on the detection light.

[0244] (Section 38) An imaging device as described in paragraph 37, wherein at least a portion of the optical paths of the detected light from the optical system detected by the detection element and the optical paths of the light from the optical system received by the image sensor are superimposed.

[0245] (Section 39) An imaging device according to paragraph 37 or 38, wherein the wavelength of the detection light detected by the detection element is different from the wavelength of the light received by the image sensor.

[0246] (Section 40) An imaging system comprising: an imaging device described in any one of paragraphs 37 to 39; and a virtual image generation unit that generates image data of the subject as if it were imaged from a position different from the position of the imaging device, based on image data of the subject generated by imaging with the image sensor of the imaging device and the distance calculation result of the imaging device to at least a part of the subject.

[0247] (Section 41) An imaging system comprising: an image sensor for capturing an image of a subject; a distance measuring unit for measuring the distance to at least a portion of the subject; and a virtual image generation unit that generates image data of the subject as if it were captured from a position different from the position of the imaging device, based on the image data of the subject generated by the imaging of the image sensor and the distance calculation result to at least a portion of the subject by the distance measuring unit. [Explanation of Symbols]

[0248] 1,1a~1c: Imaging device, 2: Imaging system, 10: First optical system, 12~15: Lens, 18: Intermediate image, 19: Intermediate image formation area, 20: Second optical system, 24: First drive unit, 25~28: Lens, 35: Final image, 36: Image sensor, 38: Housing, 40: Distance measurement unit, 41: Light emission unit, 42: Light receiving unit, 43: Light transmitting lens, 44: Light receiving lens, 47: Measurement unit, 50: Control unit, 51: Imaging control unit, 52: Image generation unit, 53: Analysis unit, 54: Field of view control unit, 55: Memory unit, 56: Interface unit, 57: Virtual image generation unit, IA0: Wide field of view imaging area, IA1: Imaging area

Claims

1. A first optical system that forms an image of the subject, An optical system that re-images at least a portion of the aforementioned image to form a final image, comprising a second optical system capable of changing the magnification of the final image, An image sensor for capturing the final image, A control device that analyzes image data generated by imaging with the image sensor, detects a region of interest within the image formed by the first optical system, and generates control information to change the position of the region formed as the final image based on the detection result of the region of interest, An imaging device equipped with the following features.

2. The first optical system is telecentric on the image side formed by the first optical system, The imaging apparatus according to claim 1.

3. The difference between the angle of the principal ray of light directed from the first optical system to a first location in the image formation region formed by the first optical system with respect to the optical axis of the first optical system and the angle of the principal ray of light directed from the first optical system to a second location in the formation region that is at a different distance from the optical axis of the first optical system with respect to the optical axis of the first optical system is within 1°. The imaging apparatus according to claim 2.

4. The second optical system is telecentric on the first optical system side. The imaging apparatus according to claim 1 or claim 2.

5. The difference between the angle of the principal ray of light traveling from a first location in the intermediate image-forming region to the second optical system with respect to the optical axis of the second optical system and the angle of the principal ray of light traveling from a second location in the intermediate image-forming region, which is at a different distance from the optical axis of the second optical system than the first location, with respect to the optical axis of the second optical system is within 1°. The imaging apparatus according to claim 4.

6. The difference between the angle of the principal ray of light from the first optical system with respect to the optical axis of the first optical system and the angle of the principal ray of light incident on the second optical system with respect to the optical axis of the second optical system is within 1°. The imaging apparatus according to any one of claims 1 to 5.

7. The control device analyzes the image data generated by the imaging of the image sensor in order to make the position of the region formed as the final image follow the moving subject, detects the region of interest within the image formed by the first optical system, and repeatedly generates control information to change the position of the region formed as the final image based on the detection result of the region of interest, The imaging apparatus according to any one of claims 1 to 6.

8. The control device calculates the distance to at least a portion of the subject, The imaging apparatus according to any one of claims 1 to 7.

9. A light source that emits measurement light onto the subject, The system includes a sensor that receives the measurement light emitted from the light source and reflected by at least a portion of the subject as detection light, The control device calculates the distance to at least a portion of the subject based on the time from the emission of the measurement light by the light source until the detection light is received by the sensor. The imaging apparatus according to claim 8.

10. The control device generates the image data obtained when the subject is photographed from a position different from the position of the imaging device, based on the distance calculation result up to at least a part of the subject. The imaging apparatus according to claim 8 or claim 9.

11. In the image, the central position of the region to be re-imaged onto the image sensor as the final image is changed. The imaging apparatus according to any one of claims 1 to 10.

12. The control device detects a plurality of regions of interest within the image formed by the first optical system. The imaging apparatus according to any one of claims 1 to 11.

13. The control device detects the region of interest using machine learning, The imaging apparatus according to any one of claims 1 to 12.

14. The region of interest is a region that includes at least one of a person, a part of a person, an animal, and a part of an animal. The imaging apparatus according to any one of claims 1 to 13.

15. Further comprising a motor for moving the second optical system and the image sensor in a direction intersecting the optical axis of the first optical system, The control device generates control information for performing at least one of the following based on the analysis results of the image data: driving the motor and changing the magnification of the final image. The imaging apparatus according to any one of claims 1 to 14.

16. Based on the results of the analysis of the image data by the control device, control information is generated for controlling the start and end of recording the image data generated by imaging with the image sensor. The imaging apparatus according to claim 15.

17. The second optical system and the image sensor are further provided with a motor that moves them in a direction intersecting the optical axis of the first optical system, The control device controls the motor based on the detection result of the region of interest and generates control information to change the position of the region formed as the final image. The imaging apparatus according to any one of claims 1 to 14.

18. A motor is provided to move the second optical system and the image sensor in a direction intersecting the optical axis of the first optical system, The imaging apparatus according to any one of claims 1 to 14.

19. The motor is a first motor, The second optical system includes a plurality of optical elements, and further includes a second motor that moves at least one of the plurality of optical elements along the optical axis of the second optical system. The second optical system changes the magnification of the final image by driving the second motor. The imaging apparatus according to claim 18.

20. Further comprising the second optical system and a holder for holding the image sensor, The motor moves the holder in a direction intersecting the optical axis of the first optical system. The imaging apparatus according to claim 18 or claim 19.

21. The motor moves the second optical system and the image sensor in a direction parallel to a plane perpendicular to the optical axis of the first optical system. The imaging apparatus according to any one of claims 18 to 20.

22. The second optical system moves with respect to the optical axis of the first optical system. The imaging apparatus according to any one of claims 1 to 21.

23. The image sensor moves with respect to the optical axis of the first optical system. The imaging device according to any one of claims 1 to 22.

24. The image formed by the first optical system is an intermediate image. The imaging device according to any one of claims 1 to 23.

25. The control device generates control information for detecting the region of interest in the image data of the subject generated by the image sensor at a second time point by performing template matching with the region of interest detected in the image data of the subject generated by the image sensor at a first time point prior to the second time point. The imaging apparatus according to any one of claims 1 to 24.

26. The control device calculates the amount and direction of movement of the region of interest between the image data of the subject generated by the image sensor at different times, and generates control information for changing the position of the region formed as the final image based on the calculated amount and direction of movement. The imaging apparatus according to any one of claims 1 to 24.

27. ​​The control device predicts the amount and direction of movement of the region of interest at a future time based on the amount and direction of movement of the region of interest calculated between the image data of the subject generated by the image sensor at different times, and generates control information to change the position of the region formed as the final image based on the predicted amount and direction of movement. The imaging device according to claim 26.

28. The control device predicts the amount and direction of movement of the region of interest at a future time based on the amount and direction of movement of the region of interest calculated between the image data of the subject generated by the image sensor at different times using deep learning, and generates control information to change the position of the region formed as the final image based on the predicted amount and direction of movement. The imaging device according to claim 26.

29. The control device detects a change in the size of the region of interest between the image data of the subject generated by the image sensor at different times, and generates control information for changing the magnification of the final image so that the size of the region of interest is fixed with respect to the region displayed as the final image, based on the detection result of the change in size. The imaging apparatus according to any one of claims 1 to 24.

30. A first optical system that forms an intermediate image of a subject and is telecentric on the side of the intermediate image, A second optical system that re-images at least a portion of the intermediate image to form a final image, An image sensor for capturing the final image, Equipped with, In the aforementioned intermediate image, the central position of the region to be re-imaged onto the image sensor as the final image is changed. The difference between the angle of the principal ray of light traveling from the first optical system to a first location in the intermediate image-forming region with respect to the optical axis of the first optical system and the angle of the principal ray of light traveling from the first optical system to a second location in the intermediate image-forming region that is at a different distance from the optical axis of the first optical system with respect to the optical axis of the first optical system is within 1°. Imaging device.

31. The second optical system is telecentric on the first optical system side. The imaging device according to claim 30.

32. The difference between the angle of the principal ray of light traveling from a first location in the intermediate image-forming region to the second optical system with respect to the optical axis of the second optical system and the angle of the principal ray of light traveling from a second location in the intermediate image-forming region, which is at a different distance from the optical axis of the second optical system than the first location, with respect to the optical axis of the second optical system is within 1°. The imaging device according to claim 31.

33. The difference between the angle of the principal ray of light from the first optical system with respect to the optical axis of the first optical system and the angle of the principal ray of light incident on the second optical system with respect to the optical axis of the second optical system is within 1°. The imaging apparatus according to any one of claims 30 to 32.

34. An imaging system comprising a plurality of imaging devices according to any one of claims 1 to 33, At least one of the plurality of imaging devices captures the subject using information from an imaging device different from the at least one imaging device. Imaging system.

Images