Imaging system and display method

Abstract

This imaging system comprises: a first optical system; a first imaging element for generating first image data indicating a first image by imaging an image formed by the first optical system; a second optical system; a second imaging element for generating second image data indicating a second image by imaging an image formed by the second optical system; a changing device for changing the direction or the position of the second imaging element or at least a part of the second optical system; and a control device. The first image includes a first object and a second object. The control device outputs, to a display device, display signals for displaying, in different display regions on the display device, the second image of the first object generated by controlling the changing device such that the image of the first object is formed on an imaging surface of the second imaging element via the second optical system on the basis of the first image data, and a second image of the second object generated by controlling the changing device such that the image of the second object is formed on the imaging surface via the second optical system on the basis of the first image data.

Description

Imaging system and display method

[0001] The present invention relates to an imaging system and a display method.

[0002] Patent document 1 describes an imaging system that has a wide-angle camera, a close-up camera that captures part of the imaging range of the wide-angle camera, and a drive mirror that moves the imaging range of the close-up camera.

[0003] International Publication No. 2018 / 138349

[0004] An imaging system according to an embodiment of the present invention comprises a first optical system, a first imaging element that generates first image data representing a first image by capturing an image formed by the first optical system, a second optical system, a second imaging element that generates second image data representing a second image by capturing an image formed by the second optical system, a modification device that changes at least one of the direction and position of at least a part of the second optical system and at least one of the second imaging element, and a control device, wherein the first image includes a first object and a second object, and the control device outputs display signals to the display device to cause each of the following images to be displayed in different display areas of the display device: a second image of the first object generated by controlling the modification device so that an image of the first object is formed on the imaging surface of the second imaging element via the second optical system based on the first image data, and causing the second imaging element to capture the image of the first object; and a second image of the second object generated by controlling the modification device so that an image of the second object is formed on the imaging surface via the second optical system based on the first image data, and causing the second imaging element to capture the image of the second object.

[0005] An imaging system according to another embodiment of the present invention comprises a first optical system, a first imaging element that generates first image data by capturing an image formed by the first optical system, a second optical system, a second imaging element that generates second image data by capturing an image formed by the second optical system, and a control device, wherein the first image includes an object, a first reflecting member, and a second reflecting member, and the control device outputs display signals to the display device to cause each of the following images to be displayed in different display areas of the display device: a second image generated by controlling the modification device so that an image of the object reflected on the first reflecting member is formed on the imaging plane via the second optical system based on the first image data, thereby causing the second imaging element to capture the image of the object reflected on the first reflecting member; and a second image generated by controlling the modification device so that an image of the object reflected on the second reflecting member is formed on the imaging plane of the second imaging element based on the first image data, thereby causing the second imaging element to capture the image of the object reflected on the second reflecting member.

[0006] An imaging system according to another embodiment of the present invention comprises a first optical system, a first imaging element that generates first image data by capturing an image formed by the first optical system, a second optical system, a second imaging element that generates second image data by capturing an image formed by the second optical system, a modification device that changes at least one of the direction and position of at least a part of the second optical system and at least one of the second imaging element, and a control device, wherein the first image includes a first object and a second object, and the control device outputs a display signal to the display device that alternately displays, in the display area of ​​the display device, a second image of the first object generated by controlling the modification device so that an image of the first object is formed on the imaging plane via the second optical system based on the first image data and causing the second imaging element to capture the image of the first object, and a second image of the second object generated by controlling the modification device so that an image of the second object is formed on the imaging plane of the second imaging element based on the first image data.

[0007] A display method according to an embodiment of the present invention is a display method executed by an imaging system having a first optical system, a second optical system, a first imaging element, a second imaging element, and a modification device, and includes generating first image data including a first object and a second object by having the first imaging element capture an image formed by the first optical system, generating second image data by having the second imaging element capture an image formed by the second optical system, controlling the modification device so that the image of the first object is formed on the imaging plane via the second optical system based on the first image data, and imaging the image of the first object with the second imaging element based on the first image data, and outputting a display signal to the display device to display each of the following in different display areas of the display device.

[0008] FIG. 1 is a schematic diagram of an imaging system according to the first embodiment. FIG. 2 is a schematic configuration diagram of the imaging system according to the first embodiment. FIG. 3 is a functional block diagram of a memory and a processor according to the first embodiment. FIG. 4 is a diagram showing an example of an image of an object according to the first embodiment. FIG. 5 is a flow diagram showing the flow of a display method according to the first embodiment. FIG. 6 is a timing chart for explaining the flow of the display method according to the first embodiment. FIG. 7 is a timing chart for explaining the flow of the display method according to the first embodiment. FIGS. 8A and 8B are diagrams showing an example of an image of an object according to a modification of the first embodiment. FIG. 9 is a flow diagram showing the flow of a display method according to a modification of the first embodiment. FIG. 10 is a functional block diagram of a memory and a processor according to a modification of the first embodiment. FIGS. 11A and 11B are diagrams showing an example of an image of an object according to a modification of the first embodiment. FIG. 12 is a diagram showing an example of an image of an object according to the modification of the first embodiment. FIG. 13 is a flow diagram showing the flow of a display method according to a modification of the first embodiment. FIG. 14 is a flow diagram showing the flow of a display method according to the modification of the first embodiment. Fig. 15 is a schematic configuration diagram of an imaging system according to a second embodiment. Fig. 16 is a functional block diagram of a memory and a processor according to the second embodiment. Fig. 17 is a schematic diagram of an imaging system according to a third embodiment. Fig. 18 is a functional block diagram of a memory and a processor according to the third embodiment. Fig. 19 is a flow chart showing the flow of a display method according to the third embodiment. Fig. 20 is a schematic configuration diagram of an imaging system according to a fourth embodiment.

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0010] [First Embodiment] Fig. 1 is a schematic diagram of an imaging system 100 according to a first embodiment of the present invention. The imaging system 100 captures images of a first object TG1 and a second object TG2 among a plurality of objects TG, and generates wide-angle image data representing a wide-angle image including the first object TG1 and the second object TG2. The imaging system 100 also captures images of the first object TG1 and generates enlarged image data representing an enlarged image including the first object TG1, and captures images of the second object TG2 and generates enlarged image data representing an enlarged image including the second object TG2. The wide-angle image and wide-angle image data are examples of a first image and first image data, respectively. The enlarged image and enlarged image data are examples of a second image and second image data, respectively.

[0011] The multiple objects TG may include multiple types of objects that differ in appearance characteristics such as size, shape, color, etc. The multiple objects TG may move relative to the imaging system 100. In the example shown in Fig. 1, the multiple objects TG move relative to the imaging system 100 by being transported by a conveyor CV.

[0012] 2 is a schematic diagram of the imaging system 100. The imaging system 100 includes an imaging optical system 110, a light splitting member 120, a magnifying optical system 130, a first imaging device 140, a second imaging device 150, an adjustment device 160, and a control device 170. The imaging optical system 110, the light splitting member 120, the magnifying optical system 130, the first imaging element 140, the second imaging element 150, and the adjustment device 160 are housed and held in a housing 101. The imaging system 100 is also connected to a display device 174. The imaging optical system is an example of a first optical system, and the magnifying optical system is an example of a second optical system.

[0013] In this embodiment, the optical axis of the first magnifying optical system 131 of the imaging optical system 110 and the magnifying optical system 130 is referred to as the first optical axis AX1. The optical axis of the second magnifying optical system 132 of the magnifying optical system 130 is referred to as the second optical axis AX2. The second optical axis AX2 may intersect with the first optical axis AX1. The second optical axis AX2 may be perpendicular to the first optical axis AX1. The direction along the first optical axis AX1 is referred to as the Y direction. The direction along the second optical axis AX2 is referred to as the Z direction. The direction perpendicular to the Y direction and the Z direction is referred to as the X direction. In other words, the X direction, the Y direction, and the Z direction are perpendicular to each other. For example, as shown in FIG. 2 , the Z direction may be vertical, and the X direction and the Y direction may be horizontal.

[0014] The imaging optical system 110 includes a front lens 111, a rear lens 112, and an aperture stop 113. The imaging optical system 110 may be telecentric on the side of the light dividing member 120 (the side of the intermediate image plane Imd described below). The front lens 111 is disposed farther from the light dividing member 120 than the aperture stop 113. The rear lens 112 is disposed closer to the light dividing member 120 than the aperture stop 113. In FIG. 2 , the front lens 111 and the rear lens 112 are each schematically shown as a single lens. The front lens 111 and the rear lens 112 may each be composed of a single lens or multiple lenses. The front lens 111 and the rear lens 112 may each include one or more lenses as well as optical elements other than lenses. The front lens 111 may include a focusing lens that is movable along the optical axis during focusing.

[0015] The imaging optical system 110 may be interchangeable. The interchangeable imaging optical system 110 may be, for example, at least one of a fixed focal length lens and a fisheye lens. In other words, a fixed focal length lens or a fisheye lens may be used as the imaging optical system 110. Furthermore, the interchangeable imaging optical system 110 may be made up of multiple types of fixed focal lengths with different focal lengths.

[0016] The imaging optical system 110 does not have to be interchangeable. In this case, the imaging optical system 110 may be at least one of a single-focus lens and a fisheye lens. The maximum angle of view of the imaging optical system 110 (i.e., the maximum angle of view on the object side of the image captured by the first image capture device 140) may be 170° or more. The maximum angle of view of the imaging optical system 110 may be less than 170°. For example, the maximum angle of view of the imaging optical system 110 may be 160°.

[0017] The light splitting member 120 is disposed between the imaging optical system 110 and the magnifying optical system 130. The light splitting member 120 splits the light LT that has passed through the imaging optical system 110, and directs one light beam to the first image capture device 140 and the other light beam to the magnifying optical system 130. For example, the light splitting member 120 may be a half mirror that performs amplitude splitting on the light LT that has passed through the imaging optical system 110. Specifically, the light splitting member 120 may reflect a portion of the light LT that has passed through the imaging optical system 110 in the +Z direction toward the first image capture device 140 and transmit the remaining light. The half mirror may be a prism type or a flat type. The ratio of the transmittance to the reflectance of the half mirror may be 1:1 or another ratio. For example, this ratio may be 2:1.

[0018] The light splitting member 120 is a dichroic mirror and may split the wavelength of the light LT that has passed through the imaging optical system 110. For example, the light splitting member 120 may reflect light in the near-infrared wavelength range, of the light LT that has passed through the imaging optical system 110, in the +Z direction toward the first image capture device 140, and transmit light in other wavelength ranges (visible light). The light splitting member 120 may transmit light in the near-infrared wavelength range, of the light LT that has passed through the imaging optical system 110, and reflect light in other wavelength ranges (visible light) in the +Z direction toward the first image capture device 140.

[0019] The light splitting member 120 may be a polarizing beam splitter, and may polarize and split the light LT that has passed through the imaging optical system 110. The light splitting member 120 may be disposed midway through the imaging optical system 110. In this case, the light splitting member 120 splits the light LT that has passed through a portion of the imaging optical system 110. For example, the rear lens 112 may be disposed between the light splitting member 112 and the first image plane Im1 and between the light splitting member 120 and the intermediate image plane Imd.

[0020] Of the light LT that has passed through at least a portion of the imaging optical system 110, the first image Im1 is formed by light that is reflected by the light splitting member 120. In other words, the first image Im1 is formed by one of the light beams split by the light splitting member 120. Furthermore, of the light LT that has passed through the imaging optical system 110, the intermediate image Imd is formed by light that has passed through the light splitting member 120. In other words, the intermediate image Imd is formed by the other of the light beams split by the light splitting member 120. The intermediate image Imd is also referred to as a third image. Here, the position where the intermediate image Imd is formed may be conjugate to the position where the first image Im1 is formed. In other words, the intermediate image Imd may be an image conjugate to the first image Im1. The first image Im1 formed by one of the light beams split by the light splitting member 120 can also be referred to as an intermediate image. Note that the light LT that has passed through at least a part of the imaging optical system 110 is split by the light splitting member 120, and the first image Im1 and the intermediate image Imd are formed by the respective split light beams, so it can also be said that the light splitting member 120 contributes to image formation. Therefore, the light splitting member 120 may be a part of the imaging optical system 110.

[0021] The magnifying optical system 130 forms a second image Im2 by enlarging and re-imaging a portion of the intermediate image Imd formed by the light LT that has passed through the imaging optical system 110. In other words, the second image Im2 is formed via the magnifying optical system 130 using the other light split by the light splitting member 120. As described above, the position at which the intermediate image Imd is formed may be conjugate with the position at which the first image Im1 is formed. By having the magnifying optical system 130 enlarge and re-imaging a portion of the intermediate image Imd, an effect equivalent to re-imaging a portion of the first image Im1 (a portion at the same relative position as the intermediate image Imd) can be obtained. In other words, the magnifying optical system 130 enlarges and re-imaging a portion of the intermediate image Imd that corresponds to a portion of the first image Im1.

[0022] The magnifying optical system 130 may be telecentric on the intermediate image Imd (beam splitting member 120) side. This suppresses divergence of the chief ray near the intermediate image Imd, allowing the aperture of the magnifying optical system 130 to be reduced. This allows the imaging system 100 to be miniaturized and manufacturing costs to be reduced. Furthermore, it is possible to suppress variations in the angle of incidence of light incident on the beam splitting member 120 (beam splitting surface). In the example shown in FIG. 2 , both the imaging optical system 110 and the magnifying optical system 130 are telecentric on the intermediate image Imd (beam splitting member 120) side, but this is not a limitation. For example, only one of the imaging optical system 110 and the magnifying optical system 130 may be telecentric on the intermediate image Imd (beam splitting member 120) side.

[0023] The magnifying optical system 130 includes a first magnifying optical system 131 and a second magnifying optical system 132. Light that has passed through the light splitting member 120 is incident on the first magnifying optical system 131. In other words, the other light split by the light splitting member 120 is incident on the first magnifying optical system 131. Light from the first magnifying optical system 131 is incident on the second magnifying optical system 132. In FIG. 2 , the first magnifying optical system 131 and the second magnifying optical system 132 are each schematically shown as a single lens. The first magnifying optical system 131 and the second magnifying optical system 132 may be composed of one or more lenses. Each of the first magnifying optical system 131 and the second magnifying optical system 132 may include optical elements other than lenses in addition to one or more lenses.

[0024] The optical elements constituting the first magnifying optical system 131 and the second magnifying optical system 132 may be housed in a lens barrel. In this case, the inner circumferential surface of the lens barrel may be anti-reflection treated. This prevents flare caused by light reflection on the inner surface of the lens barrel. The anti-reflection treatment may be achieved, for example, by arranging flocked paper on the inner circumferential surface. The first magnifying optical system 131 and the second magnifying optical system 132 may also include one or more annular masks. This prevents light reflected on the inner surface of the lens barrel from being blocked by the mask, thereby preventing flare caused by light reflection on the inner surface of the lens barrel. In this case, the inner diameter of the lens barrel may be larger than the outer diameter of the lens. This prevents light reflected on the inner surface of the lens barrel from leaking outside the mask due to tolerances in the outer diameter of the mask, thereby preventing such light from entering the lens and preventing flare.

[0025] The optical axis (second optical axis AX2) of the second magnifying optical system 132 may be perpendicular to the optical axis (first optical axis AX1) of the first magnifying optical system 131. The optical axis of the second magnifying optical system 132 may intersect with the optical axis of the first magnifying optical system 131 at an angle other than 90 degrees. For example, the optical axis of the second magnifying optical system 132 may intersect with the optical axis of the first magnifying optical system 131 within an angle range of 90 degrees ± 5 degrees. The reflecting surface 162 of the first reflecting member 161 included in the adjustment device 160 is disposed at or near the intersection position where the optical axis of the first magnifying optical system 131 intersects with the optical axis of the second magnifying optical system 132. The intersection position where the optical axis of the first magnifying optical system 131 intersects with the optical axis of the second magnifying optical system 132 is also referred to as the intersection position where the optical axis of the first magnifying optical system 131 intersects with the optical axis of the second magnifying optical system 132. The vicinity of the intersection position may be, for example, a position different from the intersection position where the reflecting surface 162 of the first reflecting member 161 can reflect light from the first magnifying optical system 131 toward the second magnifying optical system 132. The second magnifying optical system 132 forms the second image Im2 with the light from the first magnifying optical system 131 reflected by the reflecting surface 162 of the first reflecting member 161. In other words, it can be said that the second image Im2 is formed by the first magnifying optical system 131 and the second magnifying optical system 132. It can also be said that the magnifying optical system 130 includes the first reflecting member 161 in addition to the first magnifying optical system 131 and the second magnifying optical system 132.

[0026] The second magnifying optical system 132 may be interchangeable. The interchangeable second magnifying optical system 132 may be, for example, an optical system with a fixed magnification or a variable magnification optical system (zoom lens). In other words, the second magnifying optical system 132 may be an optical system with a fixed magnification or a variable magnification optical system (zoom lens). Furthermore, the interchangeable second magnifying optical system 132 may be made up of multiple types of optical systems with different magnifications.

[0027] The second magnification optical system 132 does not have to be interchangeable. In this case, too, the second magnification optical system 132 may be a fixed magnification optical system or a variable magnification optical system. Because the second magnification optical system 132 is a fixed magnification optical system or a variable magnification optical system, the magnification optical system 130 can also be said to be a fixed magnification optical system or a variable magnification optical system. The magnification optical system 130 may be configured to re-image the second image Im2 by reducing a portion of the intermediate image Imd, or may be configured to re-image the second image Im2 at the same size as the portion of the intermediate image Imd. When the magnification optical system 130 re-images the second image Im2 without enlarging the portion of the intermediate image Imd, the second imaging device 150 may enlarge the second image Im2 to generate enlarged image data (digital zoom).

[0028] The first imaging device 140 includes a first imaging element 141 that captures a first image Im1. The first imaging element 141 may be a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The first imaging element 141 may also be an infrared image sensor capable of capturing infrared images. In this case, the first imaging element 141 may be a near-infrared image sensor capable of capturing near-infrared images, or a mid-infrared image sensor capable of capturing mid-infrared images. The first imaging element 141 may also be an event-based vision sensor. For example, if the light reflected by the light dividing member 120 is visible light, the first imaging element 141 may be a CMOS image sensor or an event-based vision sensor. If the light reflected by the light dividing member 120 includes light in the near-infrared wavelength range, the first imaging element 141 may be a near-infrared image sensor. If the light reflected by the light dividing member 120 includes light in the mid-infrared wavelength range, the first imaging element 141 may be a mid-infrared image sensor.

[0029] The first image sensor 141 is disposed at or near the position of the image plane where the first image Im1 is formed. The vicinity of the position of the image plane where the first image Im1 is formed is, for example, near the range where the first image sensor 141 can capture the first image Im1 so that the target object TG can be identified in the image of the first image Im1.

[0030] A plurality of pixels are arranged two-dimensionally in the imaging region of the first imaging element 141. The plurality of pixels are arranged at a predetermined arrangement pitch in the X direction and the Y direction. For example, more than 1,000 pixels may be arranged in each of the X direction and the Y direction. The pixels of the first imaging element 141 perform photoelectric conversion of light that is reflected by the light splitting member 120 and enters the first imaging element 141. The first imaging device 140 outputs image data of the first image Im1 generated based on the photoelectric conversion at each pixel of the first imaging element 141 to the control device 170.

[0031] The second imaging device 150 includes a second imaging element 151 that captures the second image Im2. The second imaging element 151 may be a CMOS image sensor. The second imaging element 151 may also be an infrared image sensor. In this case, the second imaging element 151 may be a near-infrared image sensor or a mid-infrared image sensor. The second imaging element 151 may also be an event-based vision sensor. For example, if the light passing through the light dividing member 120 is visible light, the second imaging element 151 may be a CMOS image sensor or an event-based vision sensor. If the light passing through the light dividing member 120 includes light in the near-infrared wavelength range, the second imaging element 151 may be a near-infrared image sensor.

[0032] When the light reflected by and the light transmitted through the light dividing member 120 are both visible light, the first imaging element 141 may be an event-based vision sensor, and the second imaging element 151 may be a CMOS image sensor. When the light reflected by the light dividing member 120 is visible light, and the light transmitted through the light dividing member 120 includes light in the near-infrared wavelength range, the first imaging element 141 may be an event-based vision sensor, and the second imaging element 151 may be a near-infrared image sensor. When the light reflected by and the light transmitted through the light dividing member 120 both include light in the near-infrared wavelength range, the first imaging element 141 and the second imaging element 151 may be near-infrared image sensors. When the light reflected by and the light transmitted through the light dividing member 120 both include light in the mid-infrared wavelength range, the first imaging element 141 and the second imaging element 151 may be mid-infrared image sensors.

[0033] The second image sensor 151 is disposed at or near the position of the image plane where the second image Im2 is formed. The vicinity of the position of the image plane where the second image Im2 is formed is, for example, near the range where the second image sensor 151 can capture the second image Im2 so that the target object TG can be identified in the image of the second image Im2.

[0034] A plurality of pixels are arranged two-dimensionally in the imaging region of the second imaging element 151. The plurality of pixels are arranged at a predetermined arrangement pitch in the X direction and the Y direction. For example, more than 1,000 pixels may be arranged in each of the X direction and the Y direction. The pixels of the second imaging element 151 perform photoelectric conversion of light that passes through the light splitting member 120, passes through the magnifying optical system 130, and enters the second imaging element 151. The second imaging device 150 outputs image data of the second image Im2 generated based on the photoelectric conversion at each pixel of the second imaging element 151 to the control device 170.

[0035] The pixel arrangement pitch of the second imaging element 151 may be smaller than the pixel arrangement pitch of the first imaging element 141. In other words, the number of pixels per unit area of ​​the effective area (area where pixels are arranged) of the second imaging element 151 may be larger (higher density) than the number of pixels per unit area of ​​the effective area (area where pixels are arranged) of the first imaging element 141. In other words, the second imaging element 151 may be an imaging element with a higher resolution than the first imaging element 141. The pixel arrangement pitch of the second imaging element 151 may be equal to or smaller than the pixel arrangement pitch of the first imaging element 141. For example, the area of ​​the effective area of ​​the second imaging element 151 may be smaller than the area of ​​the effective area of ​​the first imaging element 141. The area of ​​the effective area of ​​the second imaging element 151 may be equal to or larger than the area of ​​the effective area of ​​the first imaging element 141.

[0036] The change device 160 includes a reflecting member 161 having a reflecting surface 162 and a driving device 163. As described above, the reflecting surface 162 of the reflecting member 161 may be disposed at or near the intersection of the optical axis (first optical axis AX1) of the first magnifying optical system 131 and the optical axis (second optical axis AX2) of the second magnifying optical system 132. The reflecting surface 162 of the reflecting member 161 may also be disposed at the pupil position or a pupil conjugate position of the optical system consisting of the imaging optical system 110 and the first magnifying optical system 131. This allows the area of ​​the reflecting surface 162 of the reflecting member 161 to be reduced. The reflecting surface 162 of the reflecting member 161 reflects at least a portion of the light that has passed through the light splitting member 120 and the first magnifying optical system 131 toward the second magnifying optical system 132. In other words, the reflecting surface 162 of the reflecting member 161 reflects at least a part of the other light split by the light splitting member 120 toward the second imaging device 150 .

[0037] The driving device 163 includes, for example, a voice coil motor (VCM) and rotates the reflecting member 161 in accordance with a control signal supplied from the control device 170. The driving device 163 may rotate the reflecting member 161 around two rotation axes, one extending in the X direction and the other extending in the Y direction. In other words, the adjustment device 160 may be a two-axis gimbal mirror that can rotate the reflecting member 161 around two rotation axes. This allows the driving device 163 to rotate the reflecting member 161 to an angle that reflects light traveling along the optical axis (first optical axis AX1) of the first magnifying optical system 131 toward the optical axis (second optical axis AX2) of the second magnifying optical system 132, and an angle that reflects light traveling off the optical axis of the first magnifying optical system 131 toward the optical axis of the second magnifying optical system 132. The adjustment device 160 is not limited to a two-axis gimbal mirror and may be another device. For example, the modification device 160 may have two reflecting members 161. In this case, the driving device 163 may rotate each of the two reflecting members 161 around a different rotation axis. The driving device 163 may also rotate each of the two reflecting members 161 around rotation axes that are perpendicular to each other. In other words, the modification device 160 may be a two-axis galvanometer mirror. The modification device 160 is not limited to a device that rotates the reflecting member 161 around two rotation axes, but may also be a device that rotates the reflecting member 161 around a single rotation axis (for example, a single-axis gimbal mirror or a single-axis galvanometer mirror) or a device that rotates the reflecting member 161 around three rotation axes (for example, a three-axis gimbal mirror or a three-axis galvanometer mirror).

[0038] The control device 170 is, for example, a PC (Personal Computer), etc. The control device 170 includes a memory 171, an interface 172, and a processor 173.

[0039] The memory 171 is an example of a storage unit and stores data and programs. The memory 171 is, for example, a semiconductor memory, a magnetic disk, etc. The memory 171 stores an operating system program, a driver program, an application program, and various data used in the operation of the imaging system 100, which are used in processing by the processor 173.

[0040] The interface 172 is a device that enables the control device 170 to communicate with the display device 174. The interface 172 is, for example, a video input / output interface. The interface 172 may be a serial interface, a communication interface, or the like. The interface 172 outputs data supplied from the processor 173 to the display device 174.

[0041] The processor 173 is an example of a processing unit, and performs overall control of the operation of the control device 170. The processor 173 is, for example, a central processing unit (CPU), a digital signal processor (DSP), a large scale integration (LSI), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). The processor may also be referred to as a group of circuits. The processor 173 performs various processes based on programs stored in the memory 171.

[0042] The control device 170 may not include the memory 171. The control device 170 may be housed in the housing 101. A part of the configuration of the control device 170 may be housed in the housing 101.

[0043] The display device 174 is a device that displays images, and is, for example, a liquid crystal display, an organic EL (Electro-Luminescence) display, etc. The display device 174 displays images based on a display signal output from the control device 170. The display device 174 may be integrated with the control device 170.

[0044] 3 is a functional block diagram of the memory 171 and the processor 173. The memory 171 stores a first acquisition program 181, a drive control program 182, a second acquisition program 183, and a generation program 184. The processor 173 has a first acquisition unit 191, a drive control unit 192, a second acquisition unit 193, and a generation unit 194. Each of these units is a functional module realized by the processor 173 executing the first acquisition program 181, the drive control program 182, the second acquisition program 183, and the generation program 184 stored in the memory 171. Each of these units may be implemented in the control device 170 as a dedicated processing circuit.

[0045] The first acquisition unit 191 controls the first imaging device 140 to generate image data at each predetermined imaging cycle. The image data generated by the first imaging device 140 is wide-angle image data that shows a wide-angle image including the first object and the second object. The first acquisition unit 191 acquires the wide-angle image data generated by the first imaging device 140 and stores it in the memory 171 in association with the time when the wide-angle image data was generated.

[0046] The first acquisition unit 191 detects a first object and a second object from the wide-angle image data and generates position information for the first object and the second object in the wide-angle image. For example, the first acquisition unit 191 detects the objects from the wide-angle image data by template matching. The first acquisition unit 191 may detect the first object and the second object by inputting the wide-angle image data to a classifier that has been trained in advance to detect each object based on a predetermined machine learning method, such as a convolutional neural network (CNN). The position information is, for example, the position coordinates of a bounding box surrounding the detected object. The position coordinates of the bounding box may be represented by two-dimensional coordinates in the wide-angle image or by pixel numbers of pixels that constitute the wide-angle image.

[0047] The first acquisition unit 191 generates first identification information for identifying the first object and second identification information for identifying the second object. The first identification information and second identification information may include position information of the first object and the second object, respectively. The first identification information and second identification information may include information regarding the characteristics of the first object and the second object, respectively. The information regarding the characteristics of the object may be information regarding at least one of the shape, size, color, and type of the object. The information regarding the characteristics of the object may be generated based on feature quantities of the object. The feature quantities may be, for example, SIFT (Scale Invariant Feature Transform) features. The first identification information and second identification information may include partial image data including the first object and the second object, respectively. The partial image data may be generated by, for example, extracting regions corresponding to bounding boxes surrounding each object from the wide-angle image. The first identification information and second identification information may include information regarding the time when wide-angle image data indicating wide-angle images including the first object and the second object, respectively, was generated. The first identification information and the second identification information may include information regarding the amount of change in the position of the first object and the second object, respectively. The information regarding the amount of change in the position of the object is, for example, a difference between the position information of the object in the wide-angle image acquired immediately before and the position information of the object in the wide-angle image acquired immediately before. The information regarding the amount of change in the position of the object may be information based on the position information of the object in three or more wide-angle images.

[0048] The drive control unit 192 controls the change device 160 based on the wide-angle image data. The drive control unit 192 drives the drive device 163 to rotate the reflecting member 161 by supplying a control signal to the drive device 163. For example, the drive control unit 192 controls the rotation angle of the reflecting member 161 based on position information of the first object generated from the wide-angle image data so that an image of the first object is formed on the imaging surface of the second image sensor 151 via the magnifying optical system 130. The drive control unit 192 also controls the rotation angle of the reflecting member 161 based on position information of the second object generated from the wide-angle image data so that an image of the second object is formed on the imaging surface of the second image sensor 151 via the magnifying optical system 130. A table representing the relationship between the position information of the object and the rotation angle of the reflecting member 161 is stored in advance in the memory 171, and the drive control unit 192 can determine the rotation angle of the reflecting member 161 by referring to the table.

[0049] The drive control unit 192 may control the modification device 160 so that an image of the first object and an image of the second object are alternately formed on the imaging surface of the second imaging element 151. For example, the drive control unit 192 rotates the reflecting member 161 so that, after an image of the first object is captured by the second imaging element 151, an image of the second object is formed on the imaging surface of the second imaging element 151. Furthermore, the drive control unit 192 rotates the reflecting member 161 so that, after an image of the second object is captured by the second imaging element 151, an image of the first object is formed on the imaging surface of the second imaging element 151. This makes it possible to obtain enlarged images of multiple objects with a single imaging element.

[0050] The second acquisition unit 193 controls the second imaging device 150 to generate image data at predetermined imaging intervals. The image data generated by the second imaging device 150 is enlarged image data showing an enlarged image including the first object or an enlarged image including the second object. The second acquisition unit 193 acquires the enlarged image data generated by the second imaging device 150.

[0051] The second acquisition unit 193 may perform rotation correction on the enlarged image indicated by the acquired enlarged image data. That is, since the image of the object rotates on the imaging surface of the second imaging element 151 according to the rotation angle of the reflecting member 161, the image of the object included in the enlarged image also rotates. The second acquisition unit 193 performs rotation correction on the enlarged image so that the rotated image of the object is oriented in a predetermined direction. The relationship between the rotation angle for correcting the image of the object and the rotation angle of the reflecting member 161 may be stored in advance in the memory 171. The relationship between the rotation angle of the reflecting member 161 and position information of the object in the wide-angle image may also be stored in advance in the memory 171.

[0052] The second acquisition unit 193 determines whether the acquired enlarged image data was generated after the time required for driving has elapsed since the time when driving of the drive device 163 started. The time required for driving is the time required for the image of the first object to transition from a state in which an image of the second object is formed on the imaging surface of the second imaging element 151 to a state in which an image of the second object is formed on the imaging surface of the second imaging element, or the time required for the image of the second object to transition from a state in which an image of the second object is formed on the imaging surface of the second imaging element 151 to a state in which an image of the first object is formed on the imaging surface of the second imaging element. The time required for driving the drive device 163 may be a fixed value or may be a value calculated based on the rotation angle of the reflecting member 161 before driving the drive device 163 and the rotation angle of the target reflecting member 161. The fixed value is, for example, 20 ms.

[0053] If the enlarged image data was generated before the time required to drive the drive device 163 has elapsed, the second acquisition unit 193 deletes the acquired enlarged image data. If the enlarged image data was generated after the time required to drive the drive device 163 has elapsed, the second acquisition unit 193 associates the acquired enlarged image data with wide-angle image data generated at the same time and stores the associated data in the memory 171. An enlarged image represented by enlarged image data acquired before the time required to drive the drive device 163 has elapsed is likely to not include the desired object. Therefore, by the second acquisition unit 193 storing only enlarged image data generated after the time required to drive the drive device 163 has elapsed, only enlarged image data representing an enlarged image including the desired object is stored.

[0054] The second acquisition unit 193 associates identification information with the enlarged image data stored in the memory 171. If the enlarged image includes a first object, the second acquisition unit 193 associates the enlarged image data with first identification information. If the enlarged image includes a second object, the second acquisition unit 193 associates the enlarged image data with second identification information. Which object the enlarged image includes is determined based on whether the drive control unit 192 controlled the change device 160 based on the position information of the first object or the position information of the second object. For example, the second acquisition unit 193 may associate the first identification information with the enlarged image data acquired after the drive control unit 192 controlled the change device 160 based on the position information of the first object, and associate the second identification information with the enlarged image data acquired after the drive control unit 192 controlled the change device 160 based on the position information of the second object.

[0055] The generation unit 194 generates a display signal for displaying an image of the object on the display device 174 at each predetermined imaging cycle. For example, the generation unit 194 acquires the wide-angle image data stored in the memory 171 with the most recent generation time. The generation unit 194 acquires the enlarged image data stored in the memory 171 with the most recent generation time and associated with the first identification information. The generation unit 194 acquires the enlarged image data stored in the memory 171 with the most recent generation time and associated with the second identification information. Based on the acquired data, the generation unit 194 generates display signals for displaying the wide-angle image, the enlarged image including the first object, and the enlarged image including the second object in different display areas of the display device 174. The generation unit 194 outputs the display signal to the display device 174 via the interface 172.

[0056] FIG. 4 is a diagram showing an object image G1 displayed on the display device 174. The object image G1 includes a first display area A1, a second display area A2, and a third display area A3 as display areas. The first display area A1 displays an enlarged image IMG1 of the first object TG1. The second display area A2 displays an enlarged image IMG2 of the second object TG. That is, the enlarged image IMG1 of the first object TG1 and the enlarged image IMG2 of the second object TG2 are displayed in different areas of the display device 174. This makes it possible to grasp the characteristics of multiple objects simultaneously. Furthermore, the third display area A3 displays a wide-angle image IMG3 including the first object TG1 and the second object TG2. This makes it possible to further grasp the positional relationship between the multiple objects. The third display area A3 need not be provided. The third display area A3 does not have to display the wide-angle image IMG3 including the first object TG1 and the second object TG2.

[0057] The image G1 of the object is updated and displayed at a predetermined update period. As described above, the first object TG1 and the second object TG2 are alternately imaged by the second image sensor 151, and therefore the enlarged image IMG1 of the first object TG1 and the enlarged image IMG2 of the second object TG2 are alternately displayed. For example, if the enlarged image IMG2 and the wide-angle image IMG3 of the first object TG1 are updated at the first update time of the object image G1, the enlarged image IMG2 and the wide-angle image IMG3 of the second object TG2 are updated at the next second update time. In this case, the update period for the enlarged images IMG2 and IMG3 is twice the update period for the wide-angle image IMG1. The update period for the object image G1 may be a period corresponding to the refresh rate of the display device 174, or may be a different period. Furthermore, the update period may be shorter than a period corresponding to 30 fps. This allows the enlarged images IMG1 and IMG2 to be visually recognized as a smooth moving image when viewed by the human eye. The wide-angle image IMG3 may be updated at any time without synchronizing the update cycle with that of the enlarged images IMG1 and / or IMG2.

[0058] The enlarged images IMG1 and IMG2 and the wide-angle image IMG3 may display the objects in an identifiable manner. For example, the enlarged images IMG1 and IMG2 and the wide-angle image IMG3 may include first identification information identifying the first object TG1 and second identification information identifying the second object. In the example shown in FIG. 4 , the enlarged image IMG1 of the first object TG1 includes information indicating the type of the first object (object 1) as first identification information ID1, and the enlarged image IMG2 of the second object TG2 includes information indicating the type of the second object (object 2) as second identification information ID2. Furthermore, the wide-angle image IMG3 includes the first identification information ID1 and the second identification information ID2 near the images of the first object TG1 and the second object TG2 so as to identify the images of the first object TG1 and the second object TG2, respectively. The enlarged images IMG1 and IMG2 and the wide-angle image IMG3 may display the images of the objects in a manner appropriate to the objects. In the example shown in FIG. 4 , the first identification information and the second identification information are such that the first object TG1 included in the wide-angle image IMG3 is surrounded by a dashed rectangle, and the second object TG2 is surrounded by a dashed rectangle. The enlarged image IMG1 of the first object TG1 is surrounded by a dashed rectangle, just like the first object TG1 included in the wide-angle image IMG3. The enlarged image IMG2 of the second object TG2 is surrounded by a dashed rectangle, just like the second object TG2 included in the wide-angle image IMG3. This makes it easy to understand the relationship between the objects included in the wide-angle image and the enlarged image. The first identification information and the second identification information may be indicated by changing the color of the image or the image of the object, or by displaying an identification symbol.

[0059] Note that only one of the enlarged image and the wide-angle image may display the objects in a distinguishable manner. Also, the enlarged image and the wide-angle image do not necessarily display the objects in a distinguishable manner. In other words, multiple objects may be displayed in the same manner.

[0060] FIG. 5 is a flow diagram showing the flow of a display method executed by the imaging system 100. The display method includes a process of generating a display signal that causes the display device 174 to display the image of the object shown in FIG. 4. The display method is executed at each predetermined imaging cycle. In the following description of FIG. 5 , of the first object and the second object, the object whose enlarged image is updated and displayed by the display signal may be referred to as an updated object. Because the enlarged image of the first object and the enlarged image of the second object are updated alternately, the updated object is a different object from the object whose enlarged image was updated immediately before.

[0061] First, the first acquisition unit 191 acquires wide-angle image data representing a wide-angle image including a first object and a second object from the first imaging device 140 (step S101). The first acquisition unit 191 generates position information of the first object and the second object in the wide-angle image. The first acquisition unit 191 generates first identification information for identifying the first object and second identification information for identifying the second object.

[0062] Next, the drive control unit 192 determines the drive state of the drive device 163 (step S102).

[0063] If the driving device 163 has not yet been driven so that an image of the object to be updated is formed on the imaging surface of the second imaging element 151 (step S102-A), the driving control unit 192 starts driving the driving device 163 (step S103). The driving control unit 192 starts driving the driving device 163 so that an image of the object to be updated is formed on the imaging surface of the second imaging element 151 based on the position information of the object to be updated.

[0064] If the time required for driving has not elapsed since the drive of the drive device 163 started (step S102-B), the drive control unit 192 continues driving the drive device 163 so that an image of the object to be updated is formed on the imaging surface of the second imaging element 151 (step S104). The time required for driving the drive device 163 may be calculated based on position information of the object in the wide-angle image or the driving speed of the drive device 163. The time required for driving the drive device 163 may be a predetermined time. The time required for driving the drive device 163 may be a time set by the user. The time required for driving the drive device 163 may be acquired based on drive information of the drive device. Alternatively, the enlarged image data may be acquired (step S106) without determining the driving state of the drive device (steps S102 to S105).

[0065] If the time required for driving has elapsed since the drive device 163 started to drive, but the enlarged image data of the enlarged image has not been stored in the memory 171 (step S102-C), the drive control unit 192 maintains the drive device 163 in a stationary state until the enlarged image data is stored (step S105).

[0066] After step S103, S104, or S105, the second acquisition unit 193 acquires enlarged image data from the second imaging device 150 (step S106). The second acquisition unit 193 may perform rotation correction on the enlarged image.

[0067] Next, the second acquisition unit 193 determines whether the acquired enlarged image data was generated after the time required for driving has elapsed since the time when driving of the driving device 163 started (step S107).

[0068] If the enlarged image data was generated before the time required for driving has elapsed (step S107-No), the second acquisition unit 193 deletes the acquired enlarged image data (step S108). Note that the second acquisition unit 193 may store the acquired enlarged image data in the memory 171 in association with identification information indicating that the image data does not contain an object, without deleting the acquired enlarged image data.

[0069] If the enlarged image data was generated after the time required for driving has elapsed (Yes in step S107), the second acquisition unit 193 stores the enlarged image data in the memory 171 (step S109). The second acquisition unit 193 also associates the first identification information or the second identification information with the enlarged image data stored in the memory 171.

[0070] Next, generation unit 194 generates display signals (step S110). Based on the wide-angle image data and enlarged image data stored in memory 171, generation unit 194 generates display signals for displaying the wide-angle image, the enlarged image including the first object, and the enlarged image including the second object in different display areas of display device 174. Generation unit 194 outputs the display signals to display device 174 via interface 172. This completes the display method.

[0071] 6 and 7 are timing charts for further explaining the flow of the display method.

[0072] 6 shows an example in which the time TP1 required to drive the drive device 163 is longer than the image capture period TP2. In FIG. 6, time T1 is the time when the drive device 163 has not yet started to be driven and when the enlarged image data linked to the second identification information has just been stored in the memory 171. That is, at time T1, the first object is the object to be updated.

[0073] The display method is executed at time T1. At time T1, the first acquisition unit 191 acquires wide-angle image data from the first imaging device 140 (step S101). At time T2, the driving device 163 has not yet been driven, so the driving control unit 192 starts driving the driving device 163 so that an image of the first object, which is the object to be updated, is formed on the imaging surface of the second imaging element 151 (step S103). At time T3, the second acquisition unit 195 acquires enlarged image data from the second imaging device 150 (step S106). At this time, the time TP1 required for driving the driving device 163 has not yet elapsed and the driving device 163 is still driving, so the second acquisition unit 193 deletes the enlarged image data (step S109). At time T4, the generation unit 196 generates and outputs a display signal for displaying the wide-angle image and the enlarged image (step S110). The wide-angle image displayed at this time is represented by the wide-angle image data acquired at time T1, and the enlarged image is the same as the one displayed by the display signal generated immediately before time T1.

[0074] At time T5, when the imaging period TP2 has elapsed since time T1, the display method is executed again. At time T5, the first acquisition unit 191 acquires wide-angle image data from the first imaging device 140 (step S101). Because the driving device 163 is driving at time T6, the driving control unit 192 continues driving the driving device 163 (step S104). At time T7, the second acquisition unit 195 acquires an enlarged image from the second imaging device 150. At this time, the time TP1 required to drive the driving device 163 has not elapsed and the driving device 163 is still driving, so the second acquisition unit 193 deletes the enlarged image data (step S109). At time T8, the generation unit 196 generates and outputs a display signal for displaying the wide-angle image and the enlarged image (step S110). The wide-angle image displayed at this time is represented by the wide-angle image data acquired at time T5, and the enlarged image is the same as the one displayed by the display signal generated at time T4.

[0075] At time T9, which is the imaging period TP2 after time T5, the display method is executed again. At time T9, the first acquisition unit 191 acquires wide-angle image data from the first imaging device 140 (step S101). At time T10, the driving of the drive unit 163 has finished, but the enlarged image data of the enlarged image including the update target has not yet been stored. Therefore, the drive control unit 192 keeps the drive unit 163 stationary (step S105). At time T11, the second acquisition unit 195 acquires the enlarged image and stores it in the memory 171 in association with the first identification information (step S108). At time T12, the generation unit 196 generates and outputs a display signal for displaying the wide-angle image and the enlarged image (step S110). The wide-angle image displayed at this time is represented by the wide-angle image data acquired at time T5, the enlarged image of the first target object is acquired at time T11, and the enlarged image of the second target object is the same as the image displayed by the display signal generated at time T8.

[0076] At time T13, when the imaging period TP2 has elapsed since time T9, the display method is executed again. At time T13, the first acquisition unit 191 acquires wide-angle image data from the first imaging device 140 (step S101). At time T14, the drive unit 163 has not yet driven the second object, which is the next object to be updated, so that the image of the second object is formed on the imaging surface of the second imaging element 151. Therefore, the drive control unit 192 starts driving the drive unit 163 so that the image of the second object is formed on the imaging surface of the second imaging element 151 (step S103). Thereafter, the same process is repeated.

[0077] As shown in FIG. 6, when the time TP1 required to drive the drive device 163 is longer than the image capture period TP2, the update period TP3 of the enlarged image displayed on the display device 174 becomes longer than the update period TP4 of the wide-angle image.

[0078] 7 shows an example in which the time TP5 required to drive the change device 160 is shorter than the image capture period TP6. In FIG. 7, time T21 is the time when the drive device 163 has not yet started to be driven and when the enlarged image data linked to the second identification information has just been stored in the memory 171. That is, at time T21, the first object is the object to be updated.

[0079] At time T21, the display method is executed. At time T21, the first acquisition unit 191 acquires wide-angle image data from the first imaging device 140 (step S101). At time T22, the driving device 163 has not yet been driven, so the driving control unit 192 starts driving the driving device 163 so that an image of the first object, which is the object to be updated, is formed on the imaging surface of the second imaging element 151 (step S103). At time T23, the second acquisition unit 195 acquires enlarged image data from the second imaging device 150 (step S106). At this time, the time TP5 required for driving the driving device 163 has not yet elapsed and the driving device 163 is still driving, so the second acquisition unit 193 deletes the enlarged image data (step S109). At time T24, the generation unit 196 generates and outputs a display signal for displaying the wide-angle image and the enlarged image (step S110). The wide-angle image displayed at this time is represented by the wide-angle image data acquired at time T21, and the enlarged image is the same as the one displayed by the display signal generated immediately before time T1.

[0080] At time T25, when the imaging period TP6 has elapsed since time T21, the display method is executed again. At time T21, the first acquisition unit 191 acquires wide-angle image data from the first imaging device 140 (step S101). At time T26, the driving of the drive device 163 has finished, but the enlarged image data of the enlarged image including the updated object has not yet been stored, so the drive control unit 192 keeps the drive device 163 stationary (step S105). At time T27, the second acquisition unit 195 acquires the enlarged image and stores it in the memory 171 in association with the first identification information (step S108). At time T28, the generation unit 196 generates and outputs a display signal for displaying the wide-angle image and the enlarged image (step S110). The wide-angle image displayed at this time is represented by the wide-angle image data acquired at time T25, the enlarged image of the first object is acquired at time T27, and the enlarged image of the second object is the same as the image displayed by the display signal generated at time T24.

[0081] At time T29, when the imaging period TP6 has elapsed since time T25, the display method is executed again. At time T29, the first acquisition unit 191 acquires wide-angle image data from the first imaging device 140 (step S101). At time T30, the drive unit 163 has not yet driven the second object, which is the next object to be updated, so that the image of the second object is formed on the imaging surface of the second imaging element 151. Therefore, the drive control unit 192 starts driving the drive unit 163 so that the image of the second object is formed on the imaging surface of the second imaging element 151 (step S103). Thereafter, the same process is repeated.

[0082] As shown in FIG. 7, when the time TP1 required to drive the drive device 163 is longer than the image capture period TP2, the update period TP7 of the enlarged image displayed on the display device 174 becomes equal to the update period TP8 of the wide-angle image.

[0083] In this way, the generation unit 194 of the imaging system 100 according to the first embodiment generates display signals that cause the enlarged image of the first object and the enlarged image of the second object to be displayed in different regions of the display device 174. This makes it possible to grasp the characteristics of multiple objects simultaneously.

[0084] Furthermore, the second acquisition unit 193 of the imaging system 100 associates identification information for identifying the object with the enlarged image data acquired after the time required to drive the drive device 163 has elapsed. The generation unit 194 generates a display signal for displaying the enlarged image associated with the identification information on the display device 174. This makes it possible to display only enlarged images that include the object.

[0085] In this embodiment, the wide-angle image may include a first object, a second object, and a third object. In this case, the object image G1 further includes a third display area in which an enlarged image including the third object is displayed, and the generation unit 194 generates a display signal that causes the enlarged image including the first object, the enlarged image including the second object, and the enlarged image including the third object to be displayed in different areas of the display device 174. In addition, in the display method, the first object, the second object, and the third object are sequentially set as updated objects, and the enlarged image including the first object, the enlarged image including the second object, and the enlarged image including the third object are sequentially updated and displayed on the display device 174. Similarly, enlarged images of four or more objects may be displayed in different areas of the display device 174.

[0086] Although the enlarged images are alternately displayed in this embodiment, the present invention is not limited to this example. The enlarged images may be displayed in any order. For example, after the enlarged image of the first object is continuously updated a predetermined number of times, the enlarged image of the second object may be continuously updated a predetermined number of times. In this case, the number of times the enlarged image of the first object is continuously updated may differ from the number of times the enlarged image of the second object is continuously updated. The number of times the enlarged image of the object is continuously updated may also be set based on wide-angle image data. For example, the number of times the enlarged image of the object is continuously updated may be set based on the position, velocity, and appearance characteristics of the object in the wide-angle image. The velocity of the object may be calculated by detecting the object from multiple wide-angle images. Similarly, the enlarged images may be updated at any update frequency. The update frequency of the enlarged image of the first object may differ from the update frequency of the enlarged image of the second object. The update frequency of the enlarged images may also be set based on wide-angle image data. For example, the update frequency of the enlarged images may be set based on the position, velocity, and appearance characteristics of the object in the wide-angle image.

[0087] In this embodiment, the first acquisition unit 191 may generate the updated object by estimating the position information based on multiple sets of wide-angle image data. For example, the first acquisition unit 191 detects a first object and a second object from the wide-angle image data acquired in the immediately preceding step S101 and at least one set of wide-angle image data acquired before that, and associates the detected first objects with each other and the detected second objects with each other using a known tracking technique. The first acquisition unit 191 estimates the current position information of the updated object based on changes in the position of the first object and the position of the second object between the multiple wide-angle images. For example, the first acquisition unit 191 may estimate the current position information of the updated object by applying a Kalman filter. This makes it possible to display the object near the center of the enlarged image.

[0088] [Variation 1 of First Embodiment] Figures 8A and 8B are diagrams illustrating examples of object images displayed on the display device 174 in Variation 1 of the first embodiment. The object image G2 shown in Figure 8A includes a first display area A1 in which an enlarged image IMG1 of a first object is displayed and a third display area A3 in which a wide-angle image IMG3 is displayed, but does not include a second display area A2 in which an enlarged image IMG2 of a second object is displayed. The object image G3 shown in Figure 8B includes a second display area A2 in which an enlarged image IMG2 of a second object is displayed and a third display area A3 in which a wide-angle image IMG3 is displayed, but does not include a display area A1 in which an enlarged image IMG1 of the first object is displayed. Other elements included in the object images G2 and G3 are similar to the elements included in the object image G1 shown in Figure 4, and therefore are denoted by the same reference numerals and description thereof will be omitted.

[0089] In this modification, the display device 174 alternately displays an image G2 of the object including an enlarged image IMG1 of the first object shown in FIG. 8A and an image G3 of the object including an enlarged image IMG2 of the second object shown in FIG. 8B at a predetermined switching period. The switching period may be a fixed value or may be set by the user. An example of the fixed value is 5 seconds. The switching period does not have to be uniform. In other words, the time for which the image G2 of the object is displayed may be different from the time for which the image G3 of the object is displayed.

[0090] FIG. 9 is a flowchart showing the flow of a display method executed by the imaging system according to this modified example.

[0091] First, the first acquisition unit 191 acquires wide-angle image data from the first imaging device 140, similar to step S101 (step S201). The first acquisition unit 191 generates position information of the first object and the second object. The first acquisition unit 191 generates first identification information for identifying the first object and second identification information for identifying the second object.

[0092] Next, the drive control unit 192 determines whether the time for switching between the object images G2 and G3 has arrived (step S202). The switching time is the time when the switching period has elapsed since the time when the object image displayed on the display device 174 was switched.

[0093] If the switching time has arrived (step S202—Yes), the drive control unit 192 determines the driving state of the drive device 163 in the same manner as in step S102 (step S203). If the switching time has not arrived (step S202—No), the drive control unit 192 maintains the drive device 163 in a stationary state (step S206).

[0094] The processing in steps S203 to S211 is the same as the processing in steps S102 to S110. Note that the enlarged image data may be acquired (step S207) without determining the driving state of the driving device (steps S202 to S206).

[0095] As described above, in the first modification of the first embodiment, the generation unit 194 of the imaging system 100 generates a display signal that alternately switches between displaying an enlarged image of the first object and an enlarged image of the second object on the display device 174. Therefore, compared to the case where the enlarged image IMG1 of the first object and the enlarged image IMG2 of the second object are displayed simultaneously, it is possible to grasp the characteristics of the objects in more detail.

[0096] The wide-angle image may include a first object, a second object, and a third object. In this case, the generation unit 194 generates a display signal that causes the display device 174 to sequentially display an object image including an enlarged image of the first object, an object image including an enlarged image of the second object, and an object image including an enlarged image of the third object. In the display method, the first object, the second object, and the third object are sequentially set as updated objects. Similarly, object images including enlarged images of four or more objects may be sequentially displayed on the display device 174.

[0097] 10 is a functional block diagram of the memory 171 and the processor 173 according to Modification 2 of the first embodiment. In this modification, the memory 171 further stores a receiving program 195, and the processor 173 further includes a receiving unit 195 as a functional block. The receiving unit 195 is a functional module that is realized when the processor 173 executes the receiving program 195 stored in the memory 171. The receiving unit 195 may be implemented in the control device 170 as a dedicated processing circuit.

[0098] The reception unit 195 receives a selection of the first object or the second object through a user operation. The selection through the user operation is received, for example, via an input device such as a keyboard or a mouse connected to the control device 170. In this case, the user operation may be an operation of selecting one of the enlarged image IMG1 of the first object or the enlarged image IMG2 of the second object displayed on the display device 174. The reception unit 195 stores information indicating the selected object in the memory 171.

[0099] 11(A) and 11(B) are diagrams showing examples of object images displayed on the display device 174 in this modified example. In the object image G4 shown in FIG. 11(A), the enlarged image IMG1 of the first object is displayed at a larger scale than the enlarged image IMG2 of the second object. In the object image G5 shown in FIG. 11(B), the enlarged image IMG2 of the second object is displayed at a larger scale than the enlarged image IMG1 of the first object. Since the other elements included in the object images G4 and G5 are similar to the elements of the object image G1 shown in FIG. 4, the same reference numerals are used and description thereof will be omitted.

[0100] In this modification, when generating a display signal in step S110 of the display method, the generation unit 194 acquires information indicating an object selected by a user operation from the memory 171. When a first object is selected, the generation unit 194 generates a display signal that displays an object image G4 obtained by enlarging the enlarged image IMG1 of the first object. When a second object is selected, the generation unit 194 generates a display signal that displays an object image G5 obtained by enlarging the enlarged image IMG2 of the second object.

[0101] As described above, in the second modification of the first embodiment, the generation unit 194 of the imaging system 100 generates a display signal for enlarging the enlarged image of the object selected by a user operation from the enlarged image of the first object and the enlarged image of the second object, and displaying the enlarged image on the display device 174. Therefore, it becomes possible to grasp the features of a desired object in detail while grasping the features of a plurality of objects simultaneously.

[0102] In this modification, the generation unit 194 may enlarge and display only the enlarged image of the object selected by a user operation, out of the enlarged image of the first object and the enlarged image of the second object, and may not display the enlarged images of the objects not selected by the user on the display device 174. This makes it possible to grasp the characteristics of the desired object in more detail.

[0103] 12 is a diagram showing an example of an image G6 of an object displayed on the display device 174 in a third modification of the first embodiment. The image G6 of the object includes enlarged images IMG1 of a plurality of first objects arranged in chronological order and enlarged images IMG2 of a plurality of second objects arranged in chronological order. Since the other elements included in the image G6 of the object are similar to the elements of the image G1 of the object shown in FIG. 4, the same reference numerals are used and a description thereof will be omitted.

[0104] The enlarged image IMG1 of the multiple first objects is an image selected by a predetermined method from among the enlarged images stored in memory 171 and linked to first identification information. The predetermined method may be a method of selecting enlarged image data in order of the most recent generation time. The predetermined method may also be a method of selecting enlarged image data whose generation time satisfies a predetermined condition (e.g., 5 seconds ago, 10 seconds ago, and 15 seconds ago). The predetermined method may also be a method of selecting enlarged image data in which the change in feature amount of the first object is greater than that of other images. Similarly, the enlarged image IMG2 of the multiple second objects is selected by a predetermined method from among the enlarged images stored in memory 171 and linked to second identification information.

[0105] In this modification, when generating a display signal in step S110 of the display method, the generation unit 194 selects and acquires, using a predetermined method, a plurality of enlarged images from among the enlarged images stored in the memory 171 and associated with the first identification information. The generation unit 194 also selects and acquires, using a predetermined method, a plurality of enlarged images from among the enlarged images stored in the memory 171 and associated with the second identification information. The generation unit 194 generates a display signal that arranges the acquired plurality of enlarged images in chronological order and displays them on the display device 174.

[0106] In this way, in the third modification of the first embodiment, the generation unit 194 of the imaging system 100 generates a display signal for displaying enlarged images of the plurality of first objects and enlarged images of the plurality of second objects arranged in chronological order, thereby making it possible to grasp the characteristics of the plurality of objects at different times.

[0107] In this modification, a plurality of enlarged images of only one of the first object and the second object may be displayed in chronological order. For example, a plurality of enlarged images of an object selected by a user operation from the first object and the second object may be displayed in chronological order.

[0108] [Fourth Modification of First Embodiment] FIG. 13 is a flowchart showing the flow of a display method executed by the imaging system 100 according to a fourth modification of the first embodiment.

[0109] The processing in steps S301 to S303 is the same as the processing in steps S101 to S103.

[0110] In step S302, if the time required for driving has not elapsed since the drive of the drive device 163 started (step S302-B), the drive control unit 192 updates the target rotation angle of the reflecting member 161 and continues driving the drive device 163 (step S304). That is, the drive control unit 192 changes the target rotation angle of the reflecting member 161 from the rotation angle set based on the position information of the object generated immediately before the time when the drive of the drive device 163 started to the rotation angle set based on the position information of the object generated most recently. The drive control unit 192 continues driving the drive device 163 so that the rotation angle of the reflecting member 161 becomes the changed rotation angle.

[0111] The processing in steps S305 to S310 is the same as the processing in steps S101 to S110.

[0112] As described above, in the fourth modification of the first embodiment, the drive control unit 192 of the imaging system 100 updates the target rotation angle of the reflecting member 161 while the drive device 163 is driving. This makes it possible to display the first object or the second object near the center of the enlarged image, even if the object moves while the drive device 162 is driving so that an image of the object is formed on the imaging surface of the second imaging element 151.

[0113] Fifth Modification of First Embodiment FIG. 14 is a flowchart showing the flow of a display method executed by the imaging system 100 according to a fifth modification of the first embodiment.

[0114] The processing in steps S401 to S406 is the same as the processing in steps S101 to S106. The enlarged image data may be acquired (step S406) without determining the driving state of the driving device (steps S402 to S405).

[0115] After step S406, the second acquisition unit 193 determines whether the enlarged image indicated by the enlarged image data includes an update target (step S407). For example, the second acquisition unit 193 determines whether the enlarged image data includes an update target by detecting the update target from the enlarged image data by template matching.

[0116] If the enlarged image does not include the update target (step S407—No), the second acquisition unit 193 deletes the acquired enlarged image data (step S408). If the enlarged image includes the update target (step S407—Yes), the second acquisition unit 193 associates the enlarged image data with the first identification information or the second identification information and stores it in the memory 171 (step S409). The second acquisition unit 193 may associate the acquired enlarged image data with identification information indicating that the target is not included and store it in the memory 171 without deleting it. Next, the generation unit 194 generates a display signal (step S410). The processing of steps S408-S410 is the same as the processing of steps S108-S110.

[0117] As described above, in the fifth modification of the first embodiment, the second acquisition unit 193 of the imaging system 100 stores the enlarged image data when the enlarged image includes the object to be updated. That is, when the object to be updated is the first object, the control device 170 does not update the display of the first display area if it determines that the first object is not displayed in the enlarged image. Furthermore, when the object to be updated is the second object, the control device 170 does not update the display of the second display area if it determines that the second object is not displayed in the enlarged image. Therefore, even if the object moves while the drive device 163 is driving or if an error occurs in the rotation angle of the reflecting member 161, the first object or the second object is displayed in the enlarged image.

[0118] In this modification, when the enlarged image includes an object different from the object to be updated, the second acquisition unit 193 may associate the enlarged image data with identification information corresponding to the object and store the enlarged image data in the memory 171. For example, when the object to be updated is the first object, the second acquisition unit 193 may further determine whether the enlarged image includes a second object in step S407. When the enlarged image includes the second object, the second acquisition unit 193 stores the enlarged image data in the memory 171 by associating the second identification information with the enlarged image data instead of deleting the enlarged image data.

[0119] In this modification, the generation unit 194 may display a blank image instead of the enlarged image when the enlarged image does not include the update object. That is, when the update object is a first object, the control device 170 displays a blank image in the first display area if it determines that the first object is not displayed in the enlarged image. Furthermore, when the update object is a second object, the control device 170 displays a blank image in the second display area if it determines that the second object is not displayed in the enlarged image. A blank image is an image that does not include an object and is stored in advance in the memory 171. In this case, when the second acquisition unit 193 determines that the update object is not displayed in the enlarged image, in step S408, the second acquisition unit 193 stores information indicating that the update object is not displayed in the enlarged image in the memory 171. In step S410, when information indicating that the update object is not displayed in the enlarged image is stored in the memory 171, the generation unit 194 generates a display signal that causes a blank image to be displayed in the display area corresponding to the update object on the display device 174. This makes it possible to know that the target object was not detected.

[0120] Second Embodiment FIG. 15 is a schematic diagram of an imaging system 200 according to a second embodiment. The imaging system 200 includes an imaging optical system 110, a light splitting member 120, a magnifying optical system 230, a first imaging device 140, a second imaging device 150, a change device 260, a control device 270, and a housing 201. The housing 201 houses and holds the imaging optical system 110, the light splitting member 120, the magnifying optical system 230, the first imaging device 140, the second imaging device 150, and the change device 260. Components of the imaging system 200 that are similar to those of the imaging system 200 are denoted by the same reference numerals, and detailed descriptions thereof will be omitted. The light splitting member 120 may be disposed midway through the imaging optical system 110. In this case, the light splitting member 120 may split light LT that has passed through a portion of the imaging optical system 110. The light splitting member 120 may be part of the imaging optical system 110. For example, a rear lens 112 may be disposed between the light splitting member 112 and the first image plane Im1 and between the light splitting member 120 and the intermediate image plane Imd.

[0121] In this embodiment, the optical axis of the imaging optical system 110 and the optical axis of the magnifying optical system 230, which is coaxial with the imaging optical system 110, may be collectively referred to as the optical axis AX11. The direction along the optical axis AX11 of the imaging optical system 110 and the magnifying optical system 230 is referred to as the Y direction. The direction facing the first imaging device 140 is referred to as the Z direction. The direction perpendicular to the Y direction and the Z direction is referred to as the X direction. The X direction, Y direction, and Z direction are perpendicular to each other. For example, as shown in FIG. 15 , the Z direction may be a vertical direction, and the X direction and the Y direction may be horizontal directions.

[0122] The magnifying optical system 230 forms a second image Im2 by enlarging and re-imaging a portion of the intermediate image Imd formed by the light LT that has passed through the imaging optical system 110. The magnifying optical system 230 may re-image the intermediate image Imd formed by the light LT that has passed through the imaging optical system 110 at the same size as the second image Im2. The magnifying optical system 230 may reduce the intermediate image Imd formed by the light LT that has passed through the imaging optical system 110 and re-image it as the second image Im2. The magnifying optical system 230 may form the second image Im2 by enlarging and re-imaging a portion of the intermediate image Imd. The position where the intermediate image Imd is formed may be conjugate with the position where the first image Im1 is formed. By having the magnifying optical system 230 re-imaging at least a portion of the intermediate image Imd as the second image Im2, an effect equivalent to that of re-imaging at least a portion of the first image Im1 (a portion at the same relative position as the intermediate image Imd) can be obtained. In other words, the magnifying optical system 230 re-images a part of the intermediate image Imd that corresponds to a part of the first image Im1. Note that the position where the intermediate image Imd is formed may be conjugate with the position where the first image Im1 is formed.

[0123] The magnifying optical system 230 may be telecentric on the intermediate image Imd (beam splitting member 120) side. This suppresses divergence of the chief ray near the intermediate image Imd, allowing the aperture of the magnifying optical system 230 to be reduced. This allows the imaging system 200 to be miniaturized and manufacturing costs to be reduced. Furthermore, it is possible to suppress variations in the angle of incidence of light incident on the beam splitting member 120 (beam splitting surface). Note that, while both the imaging optical system 110 and the magnifying optical system 230 are telecentric on the intermediate image Imd (beam splitting member 120) side in the example shown in FIG. 15 , this is not a limitation. For example, only one of the imaging optical system 110 and the magnifying optical system 230 may be telecentric on the intermediate image Imd (beam splitting member 120) side.

[0124] The magnifying optical system 230 may be composed of one or more lenses. The magnifying optical system 230 may include optical elements other than lenses in addition to one or more lenses.

[0125] The change device 260 includes a holding device 261 and a driving device 262. The holding device 261 holds the second imaging device 250. The driving device 262 includes, for example, a linear motor or a stepping motor, and moves the holding device 261 in accordance with a control signal supplied from the control device 270. The driving device 262 is capable of translating the holding device 261, which holds the second imaging element 251, in two directions (e.g., the X direction and the Z direction) that intersect with the optical axis AX11 of the magnifying optical system 230. The driving device 262 moves the second imaging element 251 in the X direction and the Z direction, thereby changing the imaging range of the intermediate image Imd captured by the second imaging element 251. This also changes the imaging field of view on the object side (target side) imaged by the second imaging element 251.

[0126] The control device 270 has a memory 271, an interface 172, and a processor 273. The memory 271 stores data and programs, similar to the memory 171. The processor 273 controls the overall operation of the control device 270, similar to the processor 173.

[0127] 16 is a functional block diagram of the memory 271 and the processor 273. The memory 271 differs from the memory 171 in that it stores a drive control program 282 instead of the drive control program 182. The processor 273 differs from the processor 173 in that it has a drive control unit 292 as a functional block instead of the drive control unit 192. The drive control unit 292 is a functional module that is realized when the processor 273 executes the drive control program 282. The drive control unit 292 may be implemented in the control device 270 as a dedicated processing circuit.

[0128] The drive control unit 292 controls the change device 260 based on the wide-angle image data. The drive control unit 292 drives the drive device 262 to move the holding device 261 by supplying a control signal to the drive device 262. For example, the drive control unit 292 translates the holding device 261 that holds the second image sensor 151 based on position information of the first object generated from the wide-angle image data so that an image of the first object is formed on the imaging surface of the second image sensor 151 via the magnifying optical system 230 based on position information of the second object generated from the wide-angle image data. The drive control unit 192 also translates the holding device 261 that holds the second image sensor 151 based on position information of the second object generated from the wide-angle image data so that an image of the second object is formed on the imaging surface of the second image sensor 151 via the magnifying optical system 230. The relationship between the position information of the object and the position of the holding device 261 may be stored in advance in the memory 171.

[0129] The drive control unit 292 may control the change device 260 so that an image of the first object and an image of the second object are alternately formed on the imaging surface of the second imaging element 151. For example, the drive control unit 192 controls the position of the holding device 261 so that, after an image of the first object is imaged by the second imaging element 151, an image of the second object is formed on the imaging surface of the second imaging element 151. Furthermore, the drive control unit 292 controls the position of the holding device 261 so that, after an image of the second object is imaged by the second imaging element 151, an image of the first object is formed on the imaging surface of the second imaging element 151.

[0130] The display method performed by the imaging system 200 is similar to the display method performed by the imaging system 100 , except that the position of the holding device 261 is controlled instead of the rotation angle of the reflecting member 161 .

[0131] Third Embodiment Fig. 17 is a schematic diagram of an imaging system 300 according to a third embodiment of the present invention. As shown in Fig. 17, in this embodiment, the first reflecting member R1 and the second reflecting member R2 are disposed at different positions in both imaging systems 300. As an example, the first reflecting member R1 and the second reflecting member R2 may be disposed so as to reflect light from different portions of the object TG toward the imaging system 300. Note that the first reflecting member R1 and the second reflecting member R2 may be disposed so as to reflect light from overlapping portions of the object TG toward the imaging system 300.

[0132] The first reflecting member R1 may be a total reflection mirror or a half mirror. The light-reflecting surface of the first reflecting member R1 may have a flat, convex, or concave shape. Similarly, the second reflecting member R2 may be a total reflection mirror or a half mirror. The light-reflecting surface of the second reflecting member R2 may have a flat, convex, or concave shape. The imaging system 300 captures an image of the object TG and generates wide-angle image data representing a wide-angle image including the object TG. More specifically, the imaging system 300 may capture an image of a portion of the object TG by receiving light from the portion of the object TG reflected by the first reflecting member R1, capture an image of another portion of the object TG at least partially different from the portion of the object TG by receiving light from the other portion of the object TG at least partially different from the portion of the object TG, and generate wide-angle image data representing a wide-angle image including the portion of the object TG and the other portion of the object TG at least partially different from the portion of the object TG. The enlarged image may include at least one of the first reflecting member R1 and the second reflecting member R2.

[0133] Furthermore, the imaging system 300 may generate enlarged image data showing an enlarged image of a portion of the object TG by receiving light from a portion of the object TG reflected by the first reflecting member R1, and may capture an image of another portion of the object TG that is at least partially different from the portion of the object TG by receiving light from another portion of the object TG that is at least partially different from the portion of the object TG reflected by the second reflecting member R2, thereby generating enlarged image data showing an enlarged image of the other portion of the object TG that is at least partially different from the portion of the object TG. It can also be said that the imaging system 300 images the first reflecting member R1 to generate enlarged image data showing an enlarged image including the object TG reflected in the first reflecting member R1, and images the second reflecting member R2 to generate enlarged image data showing an enlarged image including the object TG reflected in the second reflecting member R2. In the example shown in FIG. 17 , the first reflecting member R1 is positioned so that the first surface S1 of the object TG is included in the enlarged image when the object TG is at the first position P1. The second reflecting member R2 is positioned so that the second surface S2 of the object TG is included in the enlarged image when the object TG is at the second position P2. Note that the first position P1 and the second position P2 may be the same position.

[0134] The schematic configuration of the imaging system 300 is similar to that of the imaging system 100 , except that the imaging system 300 has a memory 371 and a processor 373 instead of the memory 171 and the processor 173 .

[0135] 18 is a functional block diagram of the memory 371 and processor 373 of the imaging system 300. The memory 371 stores a first acquisition program 381, a drive control program 382, ​​a second acquisition program 383, and a generation program 384. The processor 373 has a first acquisition unit 391, a drive control unit 392, a second acquisition unit 393, and a generation unit 394 as functional blocks. Each of these units is a functional module realized by the processor 373 executing the first acquisition program 381, the drive control program 382, ​​the second acquisition program 383, and the generation program 384. Each of these units may be implemented in the control device as a dedicated processing circuit. The configurations of the memory 371 and the processor 373 are similar to those of the memory 171 and the processor 173 according to the first embodiment, and therefore description thereof will be omitted.

[0136] The first acquisition unit 391 controls the first imaging device 140 to generate image data at each predetermined imaging cycle. The image data generated by the first imaging device 140 is wide-angle image data that represents a wide-angle image including an object (e.g., a portion of the object TG and another portion of the object TG that is at least partially different from the portion of the object TG). The first acquisition unit 391 acquires the wide-angle image data generated by the first imaging device 140 and stores the wide-angle image data in the memory 171 in association with the time the wide-angle image data was generated.

[0137] The first acquisition unit 391 may detect an object (e.g., a portion of the object TG and another portion of the object TG at least partially different from the portion of the object TG) from the wide-angle image data and generate position information of the object (e.g., a portion of the object TG and another portion of the object TG at least partially different from the portion of the object TG) in the wide-angle image. For example, the first acquisition unit 391 may detect an object (e.g., a portion of the object TG and another portion of the object TG at least partially different from the portion of the object TG) from the wide-angle image data by template matching. The first acquisition unit 391 may detect an object (e.g., a portion of the object TG and another portion of the object TG at least partially different from the portion of the object TG) by inputting the wide-angle image data to a classifier that has been trained in advance to detect each object based on a predetermined machine learning method. The first acquisition unit 391 determines whether the object is at a predetermined position. If the object is at a predetermined position, the first acquisition unit 391 may generate position information of a reflective member corresponding to the position of the object. For example, the first acquisition unit 391 may generate position information of the first reflecting member R1 when the object is at a first position, and may generate position information of the second reflecting member R2 when the object is at a second position. The position information of the reflecting members may be generated by template matching or a machine learning method. When the imaging range of the wide-angle image is fixed, the position information of each reflecting member may be stored in advance in the memory 171.

[0138] The first acquisition unit 391 may generate identification information that identifies the reflective member in which the object is reflected. For example, the first acquisition unit 391 may generate first identification information when the object is at a first position and reflected on the first reflective member, and may generate second identification information when the object is at a second position and reflected on the second reflective member. The first identification information and the second identification information may include position information of the first reflective member and the second reflective member, respectively. The first identification information and the second identification information may include information on characteristics of the object reflected on the first reflective member and the second reflective member, respectively. The first identification information and the second identification information may include partial image data of the object reflected on the first reflective member and the second reflective member, respectively. The first identification information and the second identification information may include information on the time when wide-angle image data capturing the object was generated.

[0139] The drive control unit 392 controls the drive device 163 of the modification device 160 based on the wide-angle image data. For example, the drive control unit 392 may control the rotation angle of the reflecting member 161 based on position information of a portion of the object TG generated from the wide-angle image data and position information of another portion of the object TG at least partially different from the portion of the object TG so that an image of the portion of the object TG generated from the wide-angle image data and an image of the other portion of the object TG at least partially different from the portion of the object TG are sequentially formed on the imaging surface of the second imaging element 151 via the magnifying optical system 130. The relationship between the position information of the wide-angle image and the rotation angle of the reflecting member 161 may be stored in advance in the memory 171. Furthermore, for example, the drive control unit 392 may control the rotation angle of the reflecting member 161 based on position information of the reflecting member generated from the wide-angle image data so that an image of the object reflected on the reflecting member is formed on the imaging surface of the second imaging element 151 via the magnifying optical system 130. The relationship between the position information of the object and the rotation angle of the reflecting member 161 may be stored in advance in the memory 171 .

[0140] The second acquisition unit 393 controls the second imaging device 150 to generate image data at each predetermined imaging cycle. The second acquisition unit 393 may sequentially generate enlarged image data showing an enlarged image of a portion of the object TG and enlarged image data showing an enlarged image of another portion of the object TG that is at least partially different from the portion of the object TG. For example, the second acquisition unit 393 may alternately generate enlarged image data showing an enlarged image of a portion of the object TG and enlarged image data showing an enlarged image of another portion of the object TG that is at least partially different from the portion of the object TG. Note that the second imaging device 150 may sequentially capture an enlarged image of a portion of the object TG and an enlarged image of another portion of the object TG that is at least partially different from the portion of the object TG. For example, the second imaging device 150 may alternately capture an enlarged image of a portion of the object TG and an enlarged image of another portion of the object TG that is at least partially different from the portion of the object TG. In this case, the second acquisition unit 393 may simultaneously generate enlarged image data showing an enlarged image of a portion of the object TG and enlarged image data showing an enlarged image of another portion of the object TG that is at least partially different from the portion of the object TG. Note that the image data generated by the second imaging device 150 may be enlarged image data showing an enlarged image including the object reflected in the first reflecting member or an enlarged image including the object reflected in the second reflecting member. The second acquisition unit 393 acquires the enlarged image data generated by the second imaging device 150 and stores it in the memory 171.

[0141] The second acquisition unit 393 may associate identification information with the enlarged image data stored in the memory 171. The second acquisition unit 393 may associate the first identification information with the enlarged image data if the enlarged image includes an object reflected on the first reflecting member, and may associate the second identification information with the enlarged image data if the enlarged image includes an object reflected on the second reflecting member. Which object the enlarged image includes may be determined based on whether the drive control unit 392 controlled the drive device 163 based on the position information of the first reflecting member or the position information of the second reflecting member.

[0142] The generation unit 394 may generate a display signal that causes the display device 174 to display a wide-angle image that includes a portion of the object TG and another portion of the object TG that is at least partially different from the portion of the object TG, and output the display signal to the display device 174. Furthermore, in addition to or instead of generating a display signal that causes the display device 174 to display a wide-angle image that includes the other portion of the object TG that is at least partially different from the portion of the object TG, the generation unit 394 may generate a display signal that causes an enlarged image of the portion of the object TG and an enlarged image of the other portion of the object TG that is at least partially different from the portion of the object TG to be displayed in different display areas of the display device 174, and output the display signal to the display device 174. In this case, the generation unit 394 may generate a display signal that causes the enlarged image of the portion of the object TG and the enlarged image of the other portion of the object TG that is at least partially different from the portion of the object TG to be displayed sequentially in different display areas of the display device 174, and output the display signal to the display device 174. For example, the generation unit 394 may generate a display signal that alternately displays an enlarged image of a portion of the object TG and an enlarged image of another portion of the object TG that is at least partially different from the portion of the object TG in different display areas of the display device 174, and output the signal to the display device 174. The generation unit 394 may also generate a display signal that simultaneously displays an enlarged image of a portion of the object TG and an enlarged image of another portion of the object TG that is at least partially different from the portion of the object TG in different display areas of the display device 174, and output the signal to the display device 174. The generation unit 394 may also generate a display signal that displays an enlarged image of the object TG in a display area different from a display area of ​​the display device 174 that displays a wide-angle image including a portion of the object TG and another portion of the object TG that is at least partially different from the portion of the object TG. The generation unit 194 may generate a display signal that displays an image of the object reflected on a reflective member on the display device 174 for each predetermined imaging period. For example, the generating unit 194 may acquire the wide-angle image data stored in the memory 171 that has the most recent generation time. The generating unit 194 may acquire the enlarged image data stored in the memory 171 that is associated with the first identification information or the second identification information that has the most recent generation time.Based on the acquired data, the generation unit 194 may generate display signals for displaying the wide-angle image and the enlarged image including the object reflected on the first reflecting member or the second reflecting member in different display areas of the display device 174. The generation unit 194 may output the display signals to the display device 174 via the interface 172.

[0143] FIG. 19 is a flowchart showing an example of the flow of a display method executed by the imaging system 300.

[0144] First, the first acquisition unit 391 acquires wide-angle image data representing a wide-angle image including an object, a first reflecting member, and a second reflecting member from the first imaging device 140 (step S501). If the object is located at a predetermined position, the first acquisition unit 391 may generate position information of the reflecting member corresponding to the position of the object. The first acquisition unit 391 may also generate identification information identifying the reflecting member in which the object is reflected. Note that in step S501, the first acquisition unit 391 may acquire wide-angle image data representing a wide-angle image including a portion of the object TG and another portion of the object TG at least partially different from the portion of the object TG from the first imaging device 140. In this case, for example, the first acquisition unit 391 may generate position information of the portion of the object TG in the wide-angle image and position information of the other portion of the object TG at least partially different from the portion of the object TG in the wide-angle image.

[0145] Next, if the object is at the predetermined position (step S502-Yes), the drive control unit 392 determines the drive state of the drive device 163 (step S503).

[0146] If the driving device 163 has not yet been driven so that the image of the object reflected on the reflective member is formed on the imaging surface 152 of the second imaging element 151 (step S503-A), the driving control unit 392 starts driving the driving device 163 (step S504).

[0147] If the time required for driving has not elapsed since the drive device 163 began to be driven (step S503-B), the drive control unit 392 continues to drive the drive device 163 so that the image of the reflective member reflected on the object is formed on the imaging surface of the second imaging element 151 (step S505).

[0148] If the time required for driving the drive device 163 has elapsed since the drive device 163 started to be driven (step S503-C), the drive control unit 392 maintains the drive device 163 in a stationary state until the enlarged image data is stored (step S506). The drive control unit 392 may acquire the enlarged image data (step S507) without determining the drive state of the drive device (steps S503 to S506). In this case, the drive control unit 392 may drive the drive device 163 based on position information of a portion of the object TG so that an enlarged image of that portion of the object TG is formed on the imaging surface 152 of the second imaging element 151 via the magnifying optical system 130 based on position information of another portion of the object TG that is at least partially different from the portion of the object TG. The drive control unit 392 may also drive the drive device 163 based on position information of another portion of the object TG so that an enlarged image of that other portion of the object TG is formed on the imaging surface 152 of the second imaging element 151 via the magnifying optical system 130. For example, the drive control unit 392 may drive the drive device 163 so that an enlarged image of a portion of the object TG and an enlarged image of another portion of the object TG, at least a portion of which is different from the portion of the object TG, are formed sequentially on the imaging surface 152 of the second imaging element 151. For example, the drive control unit 392 may drive the drive device 163 so that an enlarged image of a portion of the object TG and an enlarged image of another portion of the object TG, at least a portion of which is different from the portion of the object TG, are alternately formed on the imaging surface 152 of the second imaging element 151.

[0149] Next, the second acquisition unit 393 acquires the enlarged image data from the second imaging device 150 (step S507).

[0150] Next, the second acquisition unit 393 determines whether the enlarged image data was generated after the time required for driving has elapsed since the time when driving of the driving device 163 started (step S508).

[0151] If the enlarged image data was generated before the time required for driving has elapsed (step S508-No), the second acquisition unit 193 deletes the acquired enlarged image data (step S509). Note that the second acquisition unit 193 may store the acquired enlarged image data in the memory 171 in association with identification information indicating that the image data does not contain an object, without deleting the acquired enlarged image data.

[0152] If the enlarged image data was generated after the time required for driving has elapsed (step S509—Yes), the second acquisition unit 193 stores the enlarged image data in the memory 171 (step S510). In addition, the second acquisition unit 193 associates the enlarged image data stored in the memory 171 with the first identification information or the second identification information.

[0153] If the object is not at the predetermined position in step S502 (No in step S502), or following step S510, generation unit 194 generates a display signal (step S511). Based on the wide-angle image data and enlarged image data stored in memory 171, generation unit 194 generates display signals for displaying the wide-angle image and the enlarged image including the object in different display areas of display device 174. Generation unit 194 outputs the display signals to display device 174 via interface 172. This completes the display method.

[0154] In this way, the generation unit 394 of the imaging system 300 generates a display signal that causes the display device 174 to display an enlarged image including the image of the object reflected on the multiple reflecting members according to the position of the object. Therefore, it becomes possible to grasp images of the object as seen from different directions.

[0155] In this embodiment, a third reflecting member R3 may be provided in addition to the first reflecting member R1 and the second reflecting member R2. In this case, the wide-angle image may include three different portions of the object TG. The generation unit 194 may generate and output a display signal that causes the display device 174 to sequentially display enlarged images including each of the three different portions of the object TG. For example, the generation unit 194 may generate and output a display signal that causes the display device 174 to alternately display enlarged images including each of the three different portions of the object TG. The generation unit 194 may also generate a display signal that causes the display device 174 to simultaneously display enlarged images including each of the three different portions of the object TG. In this embodiment, the number of reflecting members is not limited to the first reflecting member R1, the second reflecting member R2, and the third reflecting member R3, and four or more reflecting members may be provided. The wide-angle image may include a first reflecting member, a second reflecting member, and a third reflecting member. In this case, the generation unit 194 generates a display signal that causes the display device 174 to display an enlarged image including the object reflected on the first reflecting member, an enlarged image including the object reflected on the second reflecting member, and an enlarged image including the object reflected on the third reflecting member according to the position of the object. Similarly, enlarged images of the object reflected on four or more reflecting members may be displayed on the display device 174 according to the position of the object. Note that in this embodiment, an illumination device (not shown) may be provided. In this case, at least one of the first imaging device 140 and the second imaging device 150 may capture an image of the object TG illuminated by an illumination device (not shown).

[0156] 20 is a schematic diagram of an imaging system 400 according to a fourth embodiment. The imaging system 400 includes a first optical system 410, a second optical system 430, a first imaging device 440, a second imaging device 450, a change device 460, and a control device 170. The first optical system 410 and the first imaging device 440 are housed and held in a first housing 401. The second optical system 430 and the second imaging element 450 are housed and held in a second housing 402. In other words, the first optical system 410 and the second optical system 430 are provided separately. The imaging system 400 is also connected to a display device 174.

[0157] The first optical system 410 includes a front lens 411, a rear lens 412, and an aperture stop 413. The first optical system 410 may be telecentric on the side of the first image capture device 440. The front lens 411 is disposed farther from the first image capture device 440 than the aperture stop 413. The rear lens 412 is disposed closer to the first image capture device 440 than the aperture stop 413. The front lens 411 and the rear lens 412 may each be composed of a single lens or multiple lenses. The front lens 411 and the rear lens 412 may each include one or more lenses as well as optical components other than lenses. The front lens 411 may include a focusing lens that is movable along the optical axis during focusing.

[0158] The second optical system 430 includes a front lens 431, a rear lens 432, and an aperture stop 433. The second optical system 430 may be telecentric on the side of the second image capture device 450. The front lens 431 is disposed farther from the second image capture device 450 than the aperture stop 433. The rear lens 432 is disposed closer to the second image capture device 450 than the aperture stop 433. The front lens 431 and the rear lens 432 may each be composed of a single lens or multiple lenses. The front lens 431 and the rear lens 432 may each include one or more lenses as well as optical components other than lenses. The front lens 431 may include a focusing lens that is movable along the optical axis during focusing.

[0159] A first image Im1 is formed by the light LT1 that has passed through the first optical system 410. A second image Im2 is formed by the light LT2 that has passed through the second optical system 430.

[0160] The first imaging device 440 includes a first imaging element 441 that captures a first image Im1. The configuration of the first imaging element 441 is similar to that of the first imaging element 141.

[0161] The second imaging device 450 includes a second imaging element 451 that captures a second image Im2. The configuration of the first imaging element 451 is similar to that of the first imaging element 151.

[0162] The change device 460 includes a reflecting member 461 having a reflecting surface 462, and a driving device 463. The reflecting surface 462 of the reflecting member 461 may be disposed near the optical axis of the second magnifying optical system 431. The reflecting surface 462 of the reflecting member 461 reflects at least a portion of the light LT2 from the object toward the second optical system 430. The configuration of the driving device 463 is similar to the configuration of the driving device 163.

[0163] The display method performed by the imaging system 400 is similar to the display method performed by the imaging system 100 .

[0164] 20, the optical axis of the first optical system 410 and the optical axis of the second optical system 430 are parallel to each other, but this is not limiting. The direction of the optical axis of the first optical system 410 and the direction of the optical axis of the second optical system 430 may be different.

[0165] It should be understood by those skilled in the art that various changes, substitutions, and modifications can be made to the above-described embodiments within the scope of the present invention. For example, the above-described processes may be performed in a different order within the scope of the present invention. The above-described various embodiments and modifications may be combined as appropriate.

Claims

1. A first optical system, a first imaging device that generates first image data indicating a first image by imaging an image formed by the first optical system, a second optical system, a second imaging device that generates second image data indicating a second image by imaging an image formed by the second optical system, a changing device that changes at least one of the direction and position of at least a part of the second optical system and at least one of the second imaging devices, and a control device, wherein the first image includes a first object and a second object, and the control device controls the changing device based on the first image data so that an image of the first object is formed on an imaging surface of the second imaging device via the second optical system, and the image of the first object is imaged by the second imaging device, and an image of the second object is formed on the imaging surface via the second optical system based on the first image data, and the changing device is controlled so that the image of the second object is imaged by the second imaging device, and a display signal for displaying each of the second images of the first object and the second object in different display areas of a display device is output to the display device. An imaging system.

2. The second optical system is a magnifying optical system, the second image data of the first object is data generated by imaging an enlarged image of the first object formed via the magnifying optical system by the second imaging device, and the second image data of the second object is data generated by imaging an enlarged image of the second object formed via the magnifying optical system by the second imaging device. The imaging system according to claim 1.

3. The control device controls the changing device so that images of the first object and the second object are alternately formed on the imaging surface of the second imaging device, and the second imaging device alternately generates the second image data of the first object and the second image data of the second object. The imaging system according to claim 1 or 2.

4. The control device alternately generates: a display signal for causing the display device to display the second image of the first object based on the second image data of the first object; and a display signal for causing the display device to display the second image of the second object based on the second image data of the second object, and alternately outputs to the display device the display signal for causing the display device to display the second image of the first object and the display signal for causing the display device to display the second image of the second object. The imaging system according to claim 2.

5. The display signal alternately causes the display device to display the second image of the first object and the second image of the first object at a period shorter than a period corresponding to 30 fps. The imaging system according to claim 3 or 4.

6. The period at which the display signal for causing the display device to display the second image of the first object and the display signal for causing the display device to display the second image of the first object are alternately output is shorter than the screen update period of the display device. The imaging system according to any one of claims 3 to 5.

7. The period at which the display signal for causing the display device to display the second image of the first object and the display signal for causing the display device to display the second image of the first object are alternately output is shorter than a period corresponding to 30 fps. The imaging system according to any one of claims 3 to 6.

8. The display signal for causing the display device to display the second image of the first object and the display signal for causing the display device to display the second image of the first object are alternately generated at a predetermined time. The imaging system according to any one of claims 3 to 7.

9. The predetermined time is set according to a user operation. The imaging system according to claim 8.

10. The control device alternately generates the display signal for causing the display device to display the second image of the first object and the display signal for causing the display device to display the second image of the first object based on information regarding the operation of the changing device. The imaging system according to claim 3 or 4.

11. The control device controls the changing device based on the position information of the first object in the first image, and controls the changing device based on the position information of the second object in the first image. The imaging system according to any one of claims 1 to 10.

12. In the display area, a first display area for displaying the second image of the first object and a second display area for displaying the second image of the second object are provided. The imaging system according to any one of claims 1 to 11.

13. In the display area, a third display area for displaying the first image is provided. The imaging system according to any one of claims 1 to 12.

14. The control device generates first identification information for identifying the first object and second identification information for identifying the second object based on the first image data. The imaging system according to any one of claims 1 to 13.

15. The first identification information includes information regarding the characteristics of the first object. The imaging system according to claim 14.

16. The characteristics of the first object include at least one of the shape, color, size, and type of the first object. The imaging system according to claim 15.

17. The first identification information includes partial image data including the first object, which is generated from the first image data. The imaging system according to any one of claims 14 to 16.

18. The first identification information includes information regarding the position of the first object. The imaging system according to any one of claims 14 to 17.

19. The first identification information includes information regarding the time when the first image data in which the first object is imaged is generated. The imaging system according to any one of claims 14 to 18.

20. The first identification information includes information regarding the amount of change in the position of the first object. The imaging system according to any one of claims 14 to 19.

21. The control device generates the first identification information based on the first image data, controls the changing device based on the information regarding the position of the first object generated based on the first image data, and associates the first identification information with the second image data of the first object generated by being imaged by the second imaging element. The imaging system according to any one of claims 14 to 20.

22. The second identification information includes information regarding the characteristics of the second object. The imaging system according to any one of claims 14 to 21.

23. The imaging system according to claim 22, wherein the features of the second object include at least one of the shape, color, size, and type of the second object.

24. The imaging system according to any one of claims 14 to 23, wherein the second identification information includes partial image data including the second object, which is generated from the first image data.

25. The imaging system according to any one of claims 14 to 24, wherein the second identification information includes information regarding the position of the second object.

26. The imaging system according to any one of claims 14 to 25, wherein the second identification information includes information regarding the time when the first image data in which the second object is imaged is generated.

27. The imaging system according to any one of claims 14 to 26, wherein the second identification information includes information regarding the amount of change in the position of the second object.

28. The control device: generates the second identification information based on the first image data; controls the changing device based on the position information of the second object generated based on the first image data; associates the second identification information with the second image data of the second object generated by being imaged by the second imaging element. The imaging system according to any one of claims 14 to 27.

29. In the display area, a first display area for displaying the second image of the first object and a second display area for displaying the second image of the second object are provided. The first display area is an area for displaying the second image associated with the first identification information. The second display area is an area for displaying the second image associated with the second identification information. The imaging system according to claim 28.

30. The display signal includes a signal for displaying the second image data of the first object associated with the first identification information in the first display area and a signal for displaying the second image data of the second object associated with the second identification information in the second display area. The imaging system according to claim 29.

31. The first image includes at least one of the first identification information and the second identification information. The imaging system according to any one of claims 15 to 30.

32. The imaging system according to any one of claims 15 to 31, wherein the second image includes at least one of the first identification information and the second identification information.

33. The imaging system according to any one of claims 1 to 14, wherein the control device generates information including first identification information for identifying the first object based on the first image data.

34. The imaging system according to claim 33, wherein the control device generates information including second identification information for identifying the second object based on the first image data.

35. The imaging system according to claim 33 or 34, wherein the control device generates third identification information for identifying the first object or the second object based on the second image data.

36. The first identification information is information regarding the characteristics of the first object, the second identification information is information regarding the characteristics of the second object, and the third identification information is information regarding the characteristics of the first object or the second object. The imaging system according to claim 35.

37. Each of the characteristics of the first object, the characteristics of the second object, and the characteristics of the third object includes at least one of the shape, color, size, and type of each object. The imaging system according to claim 36.

38. The first identification information includes partial image data including the first object generated from the first image data, and / or the second identification information includes partial image data including the second object generated from the first image data, and / or the third identification information includes partial image data of the first object or the second object generated from the second image data. The imaging system according to any one of claims 35 to 37.

39. The first identification information includes information regarding the position of the first object, and / or the second identification information includes information regarding the position of the second object, and / or the third identification information includes information regarding the position of the first object or the second object. The imaging system according to any one of claims 35 to 38.

40. The first identification information includes information regarding the time when the first image data in which the first object is imaged is generated, and / or the second identification information includes information regarding the time when the first image data in which the second object is imaged is generated, and / or the third identification information includes information regarding the time when the second image data in which the first object or the second object is imaged is generated. The imaging system according to any one of claims 35 to 39.

41. The first identification information includes information regarding the amount of change in the position of the first object, and / or the second identification information includes information regarding the amount of change in the position of the second object, and / or the third identification information includes information regarding the amount of change in the position of the first object or the second object. The imaging system according to any one of claims 35 to 40.

42. The control device generates the first identification information based on the first image data, controls the changing device based on the position information of the first object generated based on the first image data, generates the third identification information based on the second image data, and associates the third identification information with the second image of the first object or the second object generated by being imaged by the second imaging element. The imaging system according to any one of claims 35 to 41.

43. The control device generates the second identification information based on the first image data, controls the changing device based on the position information of the second object generated based on the first image data, generates the third identification information based on the second image data, and associates the third identification information with the second image of the first object or the second object generated by being imaged by the second imaging element. The imaging system according to any one of claims 35 to 42.

44. The display area is provided with a first display area for displaying the second image of the first object and a second display area for displaying the second image of the second object. The first display area is an area for displaying the second image associated with the first identification information or the second image of the first object associated with the third identification information. The second display area is an area for displaying the second image associated with the second identification information or the second image of the second object associated with the third identification information. The imaging system according to any one of claims 35 to 43.

45. The control device compares the first identification information with the third identification information and, when it determines that the first identification information and the third identification information are the same, generates a signal for displaying the second image data in the first display area. The control device compares the second identification information with the third identification information and, when it determines that the first identification information and the third identification information are the same, generates a display signal for displaying the second image data in the second display area. The imaging system according to claim 44.

46. The control device, when it determines that the first object is not displayed in the second image data, generates a display signal for displaying a blank image in the first display area. The control device, when it determines that the second object is not displayed in the second image data, generates a display signal for displaying a blank image in the second display area. The imaging system according to claim 44 or 45.

47. The control device, when it determines that the first object is not displayed in the second image data, does not change the display of the first display area. The control device, when it determines that the second object is not displayed in the second image data, does not change the display of the second display area. The imaging system according to claim 44 or 45.

48. The first image includes the first identification information or the second identification information. The imaging system according to any one of claims 35 to 47.

49. The second image includes the third identification information. The imaging system according to any one of claims 35 to 48.

50. The control device sequentially obtains information regarding the movement of the first object acquired from the first image data, sequentially controls the changing device according to the information regarding the movement of the first object, and generates a display signal for causing the second image data in which an image of the first object is captured among the second image data generated by the second image capturing element to be displayed in the first display area according to the control of the changing device. The imaging system according to any one of claims 1 to 49.

51. The control device obtains first information regarding the movement of the first object acquired from the first image data generated at a first time, controls the changing device according to the first information regarding the movement of the first object, obtains second information regarding the movement of the first object acquired from the first image data generated at a second time different from the first time, and changes the control of the changing device based on the second information. The imaging system according to any one of claims 1 to 50.

52. The control device controls the changing device so that an image of the first object is displayed near the center of the second image. The imaging system according to any one of claims 1 to 51.

53. The control device sequentially obtains information regarding the movement of the second object acquired from the first image data, sequentially controls the changing device according to the information regarding the movement of the second object, and generates a display signal for causing the second image data in which an image of the second object is captured among the second image data generated by the second image capturing element to be displayed in the second display area according to the control of the changing device. The imaging system according to any one of claims 1 to 52.

54. The control device obtains first information regarding the movement of the second object acquired from the first image data generated at a first time, controls the changing device according to the first information regarding the movement of the second object, obtains second information regarding the movement of the second object acquired from the first image data generated at a second time different from the first time, and changes the control of the changing device based on the second information. The imaging system according to any one of claims 1 to 53.

55. The imaging system according to any one of claims 1 to 54, wherein the control device controls the changing device so that an image of the second object is displayed near the center of the second image.

56. The imaging system according to any one of claims 1 to 55, wherein the control device generates a display signal for displaying the first object and the second object represented in the first image in different modes, and for displaying each of the second image of the first object and the second image of the second object in a mode corresponding to the object represented in the second image.

57. The imaging system according to any one of claims 1 to 56, wherein the control device generates a display signal for enlarging and displaying a display area of the second image of the first object or the second image of the second object selected by a user operation.

58. The imaging system according to any one of claims 1 to 57, wherein the second imaging element continuously images the first object or the second object selected by a user operation to generate the second image of the object.

59. The imaging system according to any one of claims 1 to 58, wherein the control device adjusts a display period or an update frequency of display of the second image based on a position, a speed, or an appearance feature of the first object generated based on the first image data.

60. The imaging system according to any one of claims 1 to 59, further comprising a storage device that generates and stores at least one of the first image data and the second image data for a predetermined period.

61. The imaging system according to claim 60, wherein the control device generates a display signal for arranging a plurality of first images generated based on the first image data stored in the storage device in chronological order and displaying them on the display device.

62. The imaging system according to claim 60 or 61, wherein the control device generates a display signal for arranging a plurality of second images generated based on the second image data stored in the storage device in chronological order and displaying them on the display device.

63. The control device detects an abnormality of the first object based on the first image data or the second image data, sets identification information indicating the abnormality in the second image representing the first object in which the abnormality is detected, and generates a display signal for displaying the second image in which the identification information indicating the abnormality is set. The imaging system according to any one of claims 1 to 62.

64. The changing device includes a reflecting member that reflects at least a part of the light passing through the second optical system and guides it to the imaging surface, and a driving device that drives the reflecting member so as to change the orientation of the reflecting surface of the reflecting member with respect to the optical axis of the second optical system. The imaging system according to any one of claims 1 to 63.

65. The reflecting member is provided in at least a part of the optical path of the second optical system between the second optical system and the second imaging device. The imaging system according to claim 64.

66. The changing device moves the imaging surface of the second imaging device in a direction intersecting the optical axis of the second optical system. The imaging system according to any one of claims 1 to 63.

67. The changing device includes a driving device that moves the second imaging device in a direction parallel to the imaging surface of the second imaging device. The imaging system according to any one of claims 1 to 63.

68. The imaging system further includes an optical splitting member that splits the light passing through the first optical system and guides one light to the first imaging device and the other light to the second optical system. The second optical system re-images at least a part of the image formed from the other light split by the optical splitting member on the imaging surface of the second imaging device. The imaging system according to any one of claims 1 to 67.

69. The direction of the optical axis of the first optical system is different from the direction of the optical axis of the second optical system. The imaging system according to any one of claims 1 to 67.

70. The control device controls the changing device so that an image of a third object is formed on the imaging surface of the second imaging device based on the first image data, and outputs a display signal for further displaying the second image of the third object generated by imaging the second imaging device in a display area different from the second image of the first object and the second image of the second object on the display device. The imaging system according to any one of claims 1 to 69.

71. A first optical system, a first imaging device that generates first image data indicating a first image by imaging an image formed by the first optical system, a second optical system, a second imaging device that generates second image data indicating a second image by imaging an image formed by the second optical system, a changing device that changes at least one of the direction and position of at least a part of the second optical system and at least a part of the second imaging device, and a control device, wherein the first image includes an object, and the control device controls the changing device based on the first image data so that a first image of a part of the object is formed on the imaging surface of the second imaging device via a first reflecting member and the second optical system, and causes the second imaging device to image the first image, and outputs a display signal for displaying each of the second images generated by causing the second imaging device to image the second image of another part of the object, which is at least partially different from the part of the object, via a second reflecting member disposed at a position different from the first reflecting member and the second optical system, in different display areas of a display device. An imaging system.

72. The imaging system according to claim 71, wherein the first image includes the part of the object and the other part of the object as the object.

73. The first reflecting member reflects light from the part of the object, the second reflecting member reflects light from the other part of the object, and the first imaging device images an image formed via the first optical system by the light from the part of the object reflected by the first reflecting member and an image formed via the first optical system by the light from the other part of the object reflected by the second reflecting member to generate the first image data. The imaging system according to claim 71 or 72.

74. The imaging system according to any one of claims 71 to 73, wherein the control device inspects the object based on at least one of the second image generated by causing the second imaging device to image the first image and the second image generated by causing the second imaging device to image the second image.

75. The imaging system according to any one of claims 71 to 74, further comprising the first reflecting member and the second reflecting member.

76. A first optical system, a first imaging device that generates first image data by imaging an image formed by the first optical system, a second optical system, a second imaging device that generates second image data by imaging an image formed by the second optical system, a changing device that changes at least part of the direction or position of the second optical system and the second imaging device, and a control device, wherein the first image includes a first object and a second object, and the control device controls the changing device based on the first image data so that an image of the first object is formed on the imaging surface via the second optical system, and causes the second imaging device to image the image of the first object, and controls the changing device based on the first image data so that an image of the second object is formed on the imaging surface of the second imaging device, and causes the second imaging device to image the image of the second object, and outputs a display signal to the display device to alternately display each of the second images of the first object and the second images of the second object in a display area of the display device.

77. A display method executed by an imaging system having a first optical system, a second optical system, a first imaging device, a second imaging device, and a changing device, the method including: generating first image data including a first object and a second object by causing the first imaging device to image an image formed by the first optical system; generating second image data by causing the second imaging device to image an image formed by the second optical system; controlling the changing device so that an image of the first object is formed on the imaging surface via the second optical system based on the first image data, and causing the second imaging device to image the image of the first object, thereby generating a second image of the first object; controlling the changing device so that an image of the second object is formed on the imaging surface of the second imaging device based on the first image data, and causing the second imaging device to image the image of the second object, thereby generating a second image of the second object; and outputting, to the display device, a display signal for displaying each of the second image of the first object and the second image of the second object in different display areas of the display device.

Images