Imaging system, imaging method, and program

Abstract

This imaging system includes an image sensor and a control unit. The control unit acquires, from the image sensor, four or more images including two or more first images generated under first imaging conditions and two or more second images generated under second imaging conditions. The control unit calculates the amount of motion that occurs between a reference image included among the four or more images and an image which is not the reference image and is included in the four or more images, and sets, on the basis of the amount of motion, an image group including three or more images from among the four or more images. The control unit generates motion compensation information used to compensate for motion in the image group. The control unit generates, on the basis of the motion compensation information and the images included in the image group, three-dimensional information of a subject.

Description

Imaging system, imaging method, and program

[0001] This application claims priority to Japanese Patent Application No. 2024-016215, filed February 6, 2024, the contents of which are incorporated herein by reference.

[0002] In recent years, industrial endoscopes have been widely used to observe internal damage and corrosion in the inspection of boilers, turbines, engines, chemical plants, and the like. Patent Document 1 discloses a measuring endoscope device as an example of an industrial endoscope. In the measuring endoscope device, a camera including two optical systems and an image sensor is provided at the tip of the endoscope. The measuring endoscope device performs three-dimensional measurement based on stereo measurement using the principle of triangulation.

[0003] When a defect such as a flaw is found inside the object during inspection, the user uses such a measuring endoscope device to measure the size of the flaw, etc. This allows the user to determine whether the object needs to be disassembled and repaired.

[0004] The above-described measuring endoscope device has an optical system that forms two optical images of a subject in a common area of ​​an imaging element. The two optical images are formed by light that passes through two optical paths corresponding to two different viewpoints. Hereinafter, the two optical paths will be referred to as the first optical path and the second optical path. The measuring endoscope device also has an optical path switching means that switches between the two optical paths. The measuring endoscope device captures an optical image formed by only light that passes through one of the two optical paths.

[0005] When observing a subject, the measuring endoscope device captures a first optical image of the subject formed by light that has passed through a first optical path, and generates an image. The user observes the image. When the user measures the size of a flaw or the like discovered during observation, image acquisition and measurement are performed as follows: First, the measuring endoscope device captures a first optical image of the subject, and generates an image (referred to as a first image).

[0006] The measuring endoscope device then switches the optical path and captures a second optical image of the subject formed by the light that has passed through the second optical path, generating an image (referred to as the second image). The measuring endoscope device measures the shape of the subject by using the principle of triangulation based on the parallax between the first image and the second image acquired by switching the imaging conditions in a time-division manner. The measuring endoscope device switches the imaging conditions by switching the optical path used between the two optical paths. The measuring endoscope device performs triangulation calculations using camera parameters. The camera parameters (such as baseline length) indicate the characteristics of the camera and are acquired, for example, during the manufacture of the measuring endoscope device.

[0007] If the object or the camera moves between the time the first image and the time the second image are acquired, the camera parameters will deviate from the actual situation, and the measuring endoscope device will not be able to accurately measure the shape of the object.

[0008] The measuring endoscope device disclosed in Patent Document 1 detects movement (blur) of the object or the camera and performs a two-stage determination of whether or not measurement is possible in order to accurately measure the shape of the object. In the first stage, the measuring endoscope device detects blur from images acquired before switching the optical path, and in the second stage, it detects blur from images acquired before and after switching the optical path. If no movement is detected in either of the two stages, the measuring endoscope device allows measurement to be performed. This prevents an increase in measurement error.

[0009] The measurement device disclosed in Patent Document 2 calculates the amount of positional shift between images and determines whether measurement is possible based on that amount of positional shift. This measurement device determines that measurement is possible in some of the cases where measurement is determined to be impossible in Patent Document 1. This measurement device also corrects the positional shift of the images to achieve motion compensation, and measures the shape of the subject by using the corrected images. Even when movement is detected, the measurement device determines that measurement is possible to some extent. This suppresses an increase in measurement error and improves usability.

[0010] JP 2013-105078 A JP 2018-112447 A

[0011] In Patent Document 1, when movement is detected from an image, the measuring endoscope device stops starting the measurement process. In situations where movement occurs continuously, it is difficult for the measuring endoscope device to start the measurement process. In such situations, the measuring endoscope device repeats image capture until it determines that measurement can be performed. This makes it difficult to use.

[0012] In Patent Document 2, the situation where the motion in the image is not uniform within the field of view of the camera is not considered, and the process that generates the motion compensation is applied uniformly throughout the entire field of view. Therefore, for example, in the following situations, the performance of suppressing the increase in measurement error is reduced.

[0013] There are situations where both nearby and distant objects are present in the field of view while the camera is moving. For example, when inspecting turbine engine turbine blades, the turbine blades may be lined up from the front to the back of the field of view. In such situations, even if the camera is moving in a translational manner, the nearby object will move significantly in the image, while the distant object will move slightly.

[0014] Alternatively, there are situations where the motion of the object involves rotation, for example, when observing the turbine blades of a turbine engine as the turbine section rotates slowly, in which case the root region of the turbine blade moves less in the image and the tip region of the turbine blade moves more in the image.

[0015] An object of the present invention is to provide an imaging system, an imaging method, and a program that can improve the accuracy of three-dimensional information of a subject without sacrificing ease of use.

[0016] According to an aspect of the present invention, an imaging system includes an image sensor and a control unit. The image sensor generates images of a subject under one of two or more imaging conditions, including a first imaging condition and a second imaging condition different from the first imaging condition. The control unit continuously switches between the two or more imaging conditions and acquires four or more images from the image sensor, including two or more first images generated under the first imaging condition and two or more second images generated under the second imaging condition. The control unit calculates an amount of motion between a reference image included in the four or more images and an image other than the reference image included in the four or more images, and sets an image group including three or more images of the four or more images based on the amount of motion. The image group includes at least one first image of the two or more first images and at least one second image of the two or more second images. The control unit generates motion compensation information for compensating for motion in the image group based on the at least one first image and the at least one second image. The control unit generates three-dimensional information of the subject based on the images included in the image group and the motion compensation information.

[0017] In an imaging system according to an aspect of the present invention, the control unit may generate disparity information based on the at least one first image and the at least one second image, and may generate the motion compensation information based on the disparity information.

[0018] In an imaging system according to an aspect of the present invention, the control unit may generate two or more sets of disparity information based on at least two first images including the at least one first image and the at least one second image.

[0019] In the imaging system according to an aspect of the present invention, the control unit may generate the two or more sets of disparity information by executing a first process and a second process. The control unit may execute the first process based on a second image included in the at least one second image and a first image included in the at least two first images. The control unit may execute the second process based on a second image included in the at least one second image and a first image included in the at least two first images and different from the first image used in the first process.

[0020] In the imaging system according to an aspect of the present invention, the control unit may generate the two or more sets of disparity information by executing a first process and a second process. The control unit may execute the first process based on a second image included in the at least one second image and a first image included in the at least two first images. The control unit may execute the second process based on the at least two first images.

[0021] In the imaging system according to an aspect of the present invention, the image sensor may further generate images of the subject under a third imaging condition different from both the first imaging condition and the second imaging condition. The control unit may continuously switch between the first imaging condition, the second imaging condition, and the third imaging condition. The control unit may acquire six or more images from the image sensor, including the two or more first images, the two or more second images, and two or more third images generated under the third imaging condition. The control unit may calculate the amount of motion between an image included in the six or more images and the reference image based on the amount of motion. The control unit may set an image group including three or more images of the six or more images, the image group including at least one first image of the two or more first images, at least one second image of the two or more second images, and at least one third image of the two or more third images. The control unit may generate the motion compensation information based on at least two images among the at least one first image, the at least one second image, and the at least one third image, which images are generated under different imaging conditions.

[0022] In the imaging system according to an aspect of the present invention, the control unit may acquire an image from the image sensor during a readout period shorter than one frame period. The control unit may turn off a light source that generates illumination light to be irradiated onto the subject during the readout period. The control unit may turn on the light source during at least a portion of a period excluding the readout period. The control unit may switch between the first imaging condition and the second imaging condition during the readout period.

[0023] In the imaging system according to an aspect of the present invention, the control unit may calculate a time difference between a reference time and a time at which the image sensor generates an image included in the image group after the reference time, and the control unit may set the image group so that the time difference is equal to or less than a predetermined threshold.

[0024] In the imaging system according to an aspect of the present invention, the control unit may set a first provisional image group and a second provisional image group based on the amount of motion. Each of the first provisional image group and the second provisional image group includes three or more images from the four or more images. At least one image included in the first provisional image group is different from at least one image included in the second provisional image group. The control unit may calculate a first time difference between a reference time and a timing at which the image sensor generates images included in the first provisional image group after the reference time. The control unit may calculate a second time difference between the reference time and a timing at which the image sensor generates images included in the second provisional image group after the reference time. When the first time difference is smaller than the second time difference, the control unit may set the first provisional image group as the image group. When the second time difference is smaller than the first time difference, the control unit may set the second provisional image group as the image group.

[0025] An imaging system according to an aspect of the present invention may include a first objective optical system and a second objective optical system having parallax with respect to the first objective optical system. The first imaging condition may indicate that the image sensor generates an image at a first timing based on an optical image of the subject formed through the first objective optical system. The second imaging condition may indicate that the image sensor generates an image at a second timing different from the first timing based on the optical image of the subject formed through the second objective optical system.

[0026] In the imaging system according to an aspect of the present invention, the control unit may calculate a first amount of motion between a first reference image included in the two or more first images and a first image that is different from the first reference image and is included in the two or more first images. The control unit may calculate a second amount of motion between a second reference image included in the two or more second images and a second image that is different from the second reference image and is included in the two or more second images. The control unit may set the image group based on the first amount of motion and the second amount of motion.

[0027] In the imaging system according to this aspect of the present invention, the control unit may determine the linearity of the movement based on the amount of movement, and may set the image group based on the linearity.

[0028] In the imaging system according to an aspect of the present invention, the control unit may set a first provisional image group and a second provisional image group. Each of the first provisional image group and the second provisional image group includes three or more images from the four or more images. At least one image included in the first provisional image group and at least one image included in the second provisional image group are different from each other. The control unit may determine the linearity of the motion of each of the first provisional image group and the second provisional image group based on the amount of motion. When the linearity of the first provisional image group is higher than the linearity of the second provisional image group, the control unit may set the first provisional image group as the image group. When the linearity of the second provisional image group is higher than the linearity of the first provisional image group, the control unit may set the second provisional image group as the image group.

[0029] In an imaging system according to an aspect of the present invention, the group of images may include at least two first images of the two or more first images and at least one second image of the two or more second images.

[0030] In the imaging system according to an aspect of the present invention, the control unit may generate disparity information based on a first image included in the image group and a second image included in the image group. The control unit may generate the motion compensation information by correcting the disparity information. The control unit may generate, as the 3D information, a 3D point cloud including two or more points on the subject by using the first image or the second image used to generate the disparity information and the motion compensation information.

[0031] In an imaging system according to an aspect of the present invention, the image sensor may generate the reference image by generating an image of the subject under a third imaging condition that is different from both the first imaging condition and the second imaging condition.

[0032] The imaging system according to an aspect of the present invention may include a stripe pattern generator that generates light having a stripe pattern and that is projected onto the subject. The first imaging condition may indicate a first stripe pattern that the light has. The second imaging condition may indicate a second stripe pattern that the light has. The second stripe pattern may be different from the first stripe pattern.

[0033] In the imaging system according to an aspect of the present invention, the image sensor may further generate an image of the subject under a third imaging condition different from either the first imaging condition or the second imaging condition. The third imaging condition indicates a third stripe pattern of the light. The third stripe pattern is different from either the first stripe pattern or the second stripe pattern. The image sensor may further generate an image of the subject under a fourth imaging condition different from either the first imaging condition, the second imaging condition, or the third imaging condition. The control unit may continuously switch between the first imaging condition, the second imaging condition, the third imaging condition, and the fourth imaging condition. The control unit may acquire six or more images from the image sensor, including the two or more first images, the two or more second images, and two or more third images generated under the third imaging condition. The control unit may calculate the amount of motion between an image included in the six or more images and the reference image. The control unit may set an image group including three or more images of the six or more images based on the amount of motion. The image group may include at least one first image of the two or more first images, at least one second image of the two or more second images, and at least one third image of the two or more third images. The control unit may generate the motion compensation information based on at least one image of the at least one first image, the at least one second image, and the at least one third image, and a fourth image generated under the fourth imaging condition.

[0034] In the imaging system according to an aspect of the present invention, the control unit may calculate a first amount of motion between a first image included in the two or more first images and the reference image. The control unit may calculate a second amount of motion between a second image included in the two or more second images and the reference image. The control unit may calculate a third amount of motion between a third image included in the two or more third images and the reference image. The control unit may set the image group based on the first amount of motion, the second amount of motion, and the third amount of motion.

[0035] In the imaging system according to an aspect of the present invention, the control unit may generate a three-dimensional point cloud including two or more points on the object. The control unit may shift pixels of the fourth image based on the motion compensation information. The control unit may generate the three-dimensional information by correlating colors of the shifted pixels with points included in the two or more points.

[0036] In the imaging system according to this aspect of the present invention, the control unit may generate a three-dimensional point cloud including two or more points on the object. The control unit may shift pixels of the fourth image based on the motion compensation information. The control unit may generate a texture image based on the fourth image having the shifted pixels. The control unit may convert the three-dimensional point cloud into two or more meshes. The control unit may generate a mesh polygon as the three-dimensional information by combining the two or more meshes and the texture image.

[0037] The imaging system according to this aspect of the present invention may further include an imaging device having the image sensor and the control unit.

[0038] The imaging system according to the aspect of the present invention may further include an imaging device having the image sensor. The control unit may be included in a device different from the imaging device.

[0039] According to an aspect of the present invention, an imaging method includes the following processes: a control unit continuously switches between two or more imaging conditions, including a first imaging condition and a second imaging condition different from the first imaging condition, to cause an image sensor to generate images of a subject under any one of the two or more imaging conditions; the control unit acquires four or more images from the image sensor, including two or more first images generated under the first imaging condition and two or more second images generated under the second imaging condition; the control unit calculates an amount of motion between a reference image included in the four or more images and an image other than the reference image included in the four or more images, based on the amount of motion; the control unit sets an image group including three or more images of the four or more images, based on the amount of motion; the image group includes at least one first image of the two or more first images and at least one second image of the two or more second images; and the control unit generates motion compensation information for compensating for motion in the image group, based on the at least one first image and the at least one second image. The control unit generates three-dimensional information of the subject based on the images included in the image group and the motion compensation information.

[0040] According to an aspect of the present invention, a program causes a computer to perform the following processes. The computer continuously switches between two or more imaging conditions, including a first imaging condition and a second imaging condition different from the first imaging condition, to generate images of a subject under any one of the two or more imaging conditions. The computer acquires four or more images from the image sensor, including two or more first images generated under the first imaging condition and two or more second images generated under the second imaging condition. The computer calculates an amount of motion between a reference image included in the four or more images and an image other than the reference image included in the four or more images, based on the amount of motion. The computer sets an image group including three or more images of the four or more images, based on the amount of motion. The image group includes at least one first image of the two or more first images and at least one second image of the two or more second images. The computer generates motion compensation information for compensating for motion in the image group, based on the at least one first image and the at least one second image. The computer generates three-dimensional information of the subject based on the images included in the image group and the motion compensation information.

[0041] According to the above aspects, the imaging system, imaging method, and program can improve the accuracy of three-dimensional information of a subject without sacrificing ease of use.

[0042] 1 is a perspective view showing the overall configuration of an endoscopic device according to a first embodiment of the present invention. FIG. 2 is a block diagram showing the internal configuration of an endoscopic device according to the first embodiment of the present invention. FIG. 3 is a block diagram showing the functional configuration of a CPU (Central Processing Unit) in the endoscopic device according to the first embodiment of the present invention. FIG. 4 is a timing chart showing the operation of the endoscopic device according to the first embodiment of the present invention. FIG. 5 is a flowchart showing the operation procedure of the endoscopic device according to the first embodiment of the present invention. FIG. 6 is a flowchart showing the operation procedure of the endoscopic device in image acquisition processing according to the first embodiment of the present invention. FIG. 7 is a flowchart showing the operation procedure of the endoscopic device in image selection processing according to the first embodiment of the present invention. FIG. 8 is a diagram showing an example of an image set according to the first embodiment of the present invention. FIG. 9 is a diagram showing an example of a combination of images used to calculate the amount of motion of an image according to the first embodiment of the present invention. FIG. 10 is a flowchart showing the operation procedure of the endoscopic device in motion detection processing according to the first embodiment of the present invention. FIG. 11 is a diagram showing an example of images used in motion detection processing according to the first embodiment of the present invention. FIG. 12 is a diagram showing an example of images used in motion detection processing according to the first embodiment of the present invention. FIG. 13 is a flowchart showing the operation procedure of the endoscopic device in measurement processing according to the first embodiment of the present invention. FIG. 14 is a diagram showing an example of images used to generate motion compensation information in the first embodiment of the present invention. FIG. 15 is a block diagram showing the internal configuration of an endoscopic device according to a second embodiment of the present invention. FIG. 16 is a block diagram showing the configuration of a stripe generation unit included in the endoscopic device according to the second embodiment of the present invention. FIG. 17 is a timing chart showing the operation of the endoscopic device according to the second embodiment of the present invention. Fig. 10 is a flowchart showing the operation procedure of an endoscopic device according to a second embodiment of the present invention. Fig. 11 is a flowchart showing the operation procedure of an endoscopic device in image acquisition processing of the second embodiment of the present invention. Fig. 12 is a diagram showing an example of an image acquired in image acquisition processing of the second embodiment of the present invention. Fig. 13 is a diagram showing the distribution of motion amounts calculated in motion detection processing of the second embodiment of the present invention. Fig. 14 is a flowchart showing the operation procedure of an endoscopic device in image acquisition processing of a modified example of the second embodiment of the present invention. Fig. 15 is a flowchart showing the operation procedure of an endoscopic device according to a third embodiment of the present invention.10 is a diagram showing an example of a combination of images used to calculate the amount of motion of an image in a third embodiment of the present invention. FIG. 11 is a perspective view showing the overall configuration of an endoscope system according to a fourth embodiment of the present invention. FIG. 12 is a block diagram showing the functional configuration of an external device according to the fourth embodiment of the present invention.

[0043] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0044] (First embodiment) Fig. 1 shows the overall configuration of an endoscopic device 1 according to a first embodiment of the present invention. Fig. 2 shows the internal configuration of the endoscopic device 1. The endoscopic device 1 is an example of an imaging system. The endoscopic device 1 shown in Fig. 1 has an endoscope 2, a main body 3, an operation unit 4, and a display unit 5. The endoscope 2 has an elongated insertion section 20. The endoscope 2 and the main body 3 constitute an imaging device.

[0045] The insertion section 20 is inserted into the interior of the object to be examined. An optical adapter 21 can be attached to the tip of the insertion section 20. The optical adapter 21 has an optical system for capturing light from the object into the tip of the insertion section 20. For example, a stereo optical adapter is attached to the tip of the insertion section 20. In this case, the endoscope device 1 can acquire two optical images corresponding to two different viewpoints. By using the two optical images, the endoscope device 1 can measure the dimensions (size) of the object based on the principle of triangulation. The main body 3 has a configuration for controlling the endoscope device 1. The operation section 4 accepts operations performed by the user. The display section 5 displays images acquired by the endoscope device 1, processing menus, etc.

[0046] Fig. 2 shows a detailed configuration of the endoscope device 1. An optical adapter 21 is attached to the tip of the insertion section 20 shown in Fig. 2. The optical adapter 21 of the first embodiment is a stereo optical adapter that forms multiple optical images corresponding to multiple viewpoints. The optical adapter 21 has an observation optical system 70 and a part of an illumination optical system 80.

[0047] The observation optical system 70 has concave lenses 23a, 23b, convex lenses 24a, 24b, a switching unit 25, and an imaging optical system 26. The illumination optical system 80 has a condenser lens 81, a light guide 82, a rod lens 83, and a diffusing lens 84. The condenser lens 81 is disposed in the main body 3. The light guide 82 is disposed in the main body 3 and the insertion unit 20. The rod lens 83 and the diffusing lens 84 are disposed in the optical adapter 21.

[0048] The insertion section 20 has an imaging element 22. The imaging element 22 is an image sensor and is disposed at the tip of the insertion section 20. The main body section 3 has an imaging control section 30, an image processing section 31, a light source section 32, an illumination control section 33, a CPU (Central Processing Unit) 34, a memory 35, and a switching control section 36. The light source section 32 has a white light source 37.

[0049] The white light source 37 converts the power supplied from the illumination control unit 33 into white light. This allows the white light source 37 to generate illumination light for illuminating the subject. For example, the white light source is a combination of a semiconductor light-emitting element and a phosphor. The semiconductor light-emitting element is a light-emitting diode (LED) or a laser diode (LD), and emits blue light. The phosphor converts the blue light, which is excitation light, into white light. The semiconductor light-emitting element can quickly switch the light on and off. The illumination control unit 33 can adjust the exposure time for capturing an image of the subject by adjusting the lighting period of the semiconductor light-emitting element. Compared to other light sources such as halogen lamps, the semiconductor light-emitting element has a higher light-emitting efficiency. Therefore, it consumes less power than other light sources with the same brightness, and the endoscope device 1 can be made more compact.

[0050] The illumination control unit 33 supplies power to the light source unit 32. The illumination control unit 33 controls the timing of turning on the white light source 37, the timing of turning off the white light source 37, and the amount of light emitted by the white light source 37 based on the light source control parameters output from the video processing unit 31. Control modes for the white light source 37 include a continuous illumination mode and a pulse illumination mode. In the continuous illumination mode, the amount of light is controlled according to the magnitude of the direct current supplied to the light source unit 32. In the pulse illumination mode, the amount of light is controlled according to the width and height of the current pulse supplied to the light source unit 32. The white light source 37 may be disposed in the optical adapter 21.

[0051] White light emitted from the white light source 37 is condensed by a condenser lens 81 and transmitted to the tip of the insertion section 20 via a light guide 82. The light guide 82 is an optical fiber bundle formed by bundling optical fiber strands. The white light emitted from the light guide 82 is transmitted to a diffusing lens 84 via a rod lens 83, and is irradiated onto the subject by the diffusing lens 84.

[0052] The observation optical system 70 captures light reflected from the surface of an object illuminated with white light. The light captured by the observation optical system 70 is incident on the image sensor 22. In other words, the observation optical system 70 forms an optical image of the object illuminated with the illumination light on the image sensor 22.

[0053] The observation optical system 70 includes a first optical system and a second optical system. The first optical system and the second optical system are disposed optically in front of (on the subject side of) the image sensor 22. The first optical system forms a first optical image of a subject corresponding to a first viewpoint on the image sensor 22. The second optical system forms a second optical image of a subject corresponding to a second viewpoint different from the first viewpoint on the image sensor 22.

[0054] The concave lens 23a, the convex lens 24a, and the imaging optical system 26 constitute a first optical system (first objective optical system). Light from a subject enters the first optical system and travels along a first optical path L1 within the first optical system. The first optical system forms a first optical image in an imaging region S1 of the image sensor 22. The concave lens 23b, the convex lens 24b, and the imaging optical system 26 constitute a second optical system (second objective optical system). Light from a subject enters the second optical system and travels along a second optical path L2 within the second optical system. The second optical system forms a second optical image in an imaging region S1 of the image sensor 22.

[0055] The image sensor 22 generates an image of a subject under one of a first imaging condition and a second imaging condition, which are different from each other. The first imaging condition indicates that the image sensor 22 generates an image at a first timing based on a first optical image. The second imaging condition indicates that the image sensor 22 generates an image at a second timing different from the first timing based on a second optical image.

[0056] Under the first imaging condition, the switching unit 25 allows light passing through the first optical system to be incident on the image sensor 22 and blocks light passing through the second optical system. Under the second imaging condition, the switching unit 25 allows light passing through the second optical system to be incident on the image sensor 22 and blocks light passing through the first optical system. The switching control unit 36 ​​controls the switching unit 25 to switch the optical image formed on the image sensor 22 between the first optical image and the second optical image.

[0057] The switching unit 25 sets either the first optical path L1 or the second optical path L2, which are different from each other, as an imaging optical path, thereby forming only one of the first optical image or the second optical image in the imaging region S1 of the image sensor 22. The first optical image is formed by light that has passed through the first optical path L1. The second optical image is formed by light that has passed through the second optical path L2. Under a first imaging condition, the first optical path L1 is set as the imaging optical path. Under a second imaging condition, the second optical path L2 is set as the imaging optical path.

[0058] The optical axis of the second optical system on the subject side is approximately parallel to the optical axis of the first optical system on the subject side. The second optical system has parallax with respect to the first optical system. That is, the first optical system and the second optical system are spaced apart in the parallax direction. The parallax direction is the direction of a straight line passing through the optical center (principal point) of the first optical system and the optical center (principal point) of the second optical system. The parallax direction is approximately perpendicular to the optical axis of each optical system.

[0059] The switching unit 25 switches the imaging optical path between the first optical path L1 and the second optical path L2. The switching unit 25 transmits only light passing through either the first optical path L1 or the second optical path L2 and blocks light passing through the other. For example, the switching unit 25 includes a shutter (a shielding plate) that is inserted into only one of the first optical path L1 or the second optical path L2.

[0060] When the switching unit 25 transmits light in the first optical path L1, the shutter is inserted into the second optical path L2 and blocks the light in the second optical path L2. When the switching unit 25 transmits light in the second optical path L2, the shutter is inserted into the first optical path L1 and blocks the light in the first optical path L1. The operation of the switching unit 25 is controlled based on a control signal output from the switching control unit 36. The switching unit 25 may be a liquid crystal shutter including a polarizing plate and a liquid crystal cell. The switching unit 25 is not limited to the above configuration.

[0061] When the first optical path L1 is set as the imaging optical path, light passing through the first optical path L1 passes through the switching unit 25 and enters the imaging optical system 26. At this time, the imaging optical system 26 forms a first optical image in the imaging region S1 of the image sensor 22. When the second optical path L2 is set as the imaging optical path, light passing through the second optical path L2 passes through the switching unit 25 and enters the imaging optical system 26. At this time, the imaging optical system 26 forms a second optical image in the imaging region S1 of the image sensor 22.

[0062] The switching control unit 36 ​​causes the switching unit 25 to set a first imaging condition for two or more first frame periods. The switching control unit 36 ​​causes the switching unit 25 to set a second imaging condition for two or more second frame periods. Each of the two or more second frame periods is different from each of the two or more first frame periods.

[0063] The optical adapter 21 and the insertion section 20 may be integrated. That is, the components within the optical adapter 21 may be disposed at the tip of the insertion section 20.

[0064] The imaging element 22 has two or more cells (pixels) arranged in a matrix. The two or more cells are arranged in an imaging region S1 of the imaging element 22. The number of rows and the number of columns in the arrangement of the two or more cells are each two or more. The number of rows and the number of columns do not have to be the same. Each of the two or more cells generates a pixel signal according to the amount of light incident on the cell.

[0065] The image sensor 22 sequentially reads out pixel signals from at least some of the two or more cells, row by row. The pixel signals are generated based on an optical image of a subject. The image sensor 22 generates an image of the subject in each of two or more frame periods based on at least some of the pixel signals (pixel signals of effective pixels) read out from at least some of the two or more cells.

[0066] A frame is a set of pixel signals included in one image. One image (one frame) is generated in one frame period. The imaging element 22 generates one image based on the pixel signals of one frame.

[0067] The imaging element 22 continuously scans two or more rows in an array of two or more cells, row by row, during each of two or more frame periods. As a result, the imaging element 22 reads pixel signals from the cells in the two or more rows. The imaging element 22 generates a first image based on a first optical image formed in the imaging region S1, and generates a second image based on a second optical image formed in the imaging region S1. The first image and the second image are image data including pixel values ​​of pixel signals read from the cells in the two or more rows. The imaging element 22 outputs the first image and the second image to the video processing unit 31. The operation of the imaging element 22 is controlled based on a control signal output from the imaging control unit 30.

[0068] The first imaging condition and the second imaging condition are different from each other. Under the first imaging condition, the first optical path L1 is set as the imaging optical path. The imaging element 22 generates a first image of the subject by capturing an image of the subject under the first imaging condition. Under the second imaging condition, the second optical path L2 is set as the imaging optical path. The imaging element 22 generates a second image of the subject by capturing an image of the subject under the second imaging condition.

[0069] For example, a line exposure type CMOS imager is used for the imaging element 22. By employing a CMOS imager, the configuration of the endoscope device 1 can be simplified and the power consumption of the endoscope device 1 can be reduced.

[0070] A signal line 90 is arranged inside the insertion section 20 and inside the main body section 3. The signal line 90 is a composite coaxial line formed by bundling a plurality of coaxial cables. The distal end of the signal line 90 is connected to the imaging element 22, and some of the coaxial cables on the proximal end of the signal line 90 are connected to the imaging control unit 30. The imaging control unit 30 supplies driving power to the imaging element 22 via the signal line 90. The imaging control unit 30 also outputs imaging parameters received from the video processing unit 31 to the imaging element 22. In this way, the imaging control unit 30 controls the imaging element 22.

[0071] The remaining coaxial cable on the base end of the signal line 90 is connected to the image processing unit 31. The image generated by the imaging element 22 is transmitted to the image processing unit 31. The image processing unit 31 performs various types of image processing on the image output from the imaging element 22. For example, the image processing performed by the image processing unit 31 is at least one of demosaicing, digital gain adjustment, noise reduction, white balance adjustment, contour correction, and gamma correction. The image processing unit 31 combines the image that has undergone the image processing with graphic data generated by the CPU 34. As a result, the image processing unit 31 generates a video signal for display. The video processing unit 31 outputs the generated video signal to the display unit 5.

[0072] Furthermore, the image processing unit 31 generates control parameters for capturing an image at an appropriate brightness. The image processing unit 31 generates imaging control parameters and lighting control parameters based on an input image or an image that has undergone image processing. The imaging control parameters include the line readout cycle, frame rate, and analog gain of the image sensor 22. The lighting control parameters include the timing to turn on the illumination light, the timing to turn off the illumination light, and lighting intensity. The image processing unit 31 outputs the imaging control parameters to the imaging control unit 30. The imaging control unit 30 controls the image sensor 22 based on the imaging control parameters. The image processing unit 31 outputs the lighting control parameters to the lighting control unit 33. The lighting control unit 33 controls the white light source 37 based on the lighting control parameters.

[0073] At least two of the imaging control unit 30, the image processing unit 31, the lighting control unit 33, and the switching control unit 36 ​​may be integrated. The imaging control unit 30, the image processing unit 31, the lighting control unit 33, and the switching control unit 36 ​​may be configured as a control unit using at least one of a processor and a logic circuit. For example, the processor is at least one of a CPU, a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). For example, the logic circuit is at least one of an ASIC (Application Specific Integrated Circuit) and an FPGA (Field-Programmable Gate Array). The imaging control unit 30, the image processing unit 31, the lighting control unit 33, and the switching control unit 36 ​​may include one or more processors. The imaging control unit 30, the video processing unit 31, the illumination control unit 33, and the switching control unit 36 ​​may each include one or more logic circuits.

[0074] The computer of the endoscope device 1 may load a program and execute the loaded program. The program includes instructions that define the operations of the imaging control unit 30, the illumination control unit 33, and the switching control unit 36. In other words, the functions of the imaging control unit 30, the illumination control unit 33, and the switching control unit 36 ​​may be realized by software. The program may be provided by a "computer-readable recording medium" such as a flash memory. The program may be transmitted from a computer storing the program to the endoscope device 1 via a transmission medium or by transmission waves in the transmission medium. A "transmission medium" that transmits the program is a medium capable of transmitting information. Examples of media capable of transmitting information include networks (communication networks) such as the Internet and communication lines (communication lines) such as telephone lines. The above-mentioned program may realize some of the above-mentioned functions. Furthermore, the above-mentioned program may be a difference file (difference program). The above-mentioned functions may be realized by combining a program already stored in the computer with the difference program.

[0075] The CPU 34 controls each unit within the endoscope device 1. The CPU 34 also monitors the state of the operation unit 4. As a result, the CPU 34 detects operations related to measurement, etc. The CPU 34 may be a DSP or a GPU. The CPU 34 may be an ASIC or an FPGA. The CPU 34 may be an SoC (System On Chip) incorporating a DSP, a GPU, or both.

[0076] The memory 35 stores images processed by the image processing unit 31. The memory 35 may store images output from the image sensor 22. The memory 35 may be removable from the endoscope device 1. The memory 35 is configured as a volatile or non-volatile memory. For example, the memory 35 is any one of RAM (Random Access Memory), DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), and flash memory. Alternatively, the memory 35 may be a combination of at least two of the above memories. The endoscope device 1 may have a hard disk drive for storing images. Images stored in the memory 35 may be recorded on an external storage medium or online storage not shown in FIG. 2.

[0077] The operation unit 4 is a user interface that accepts instructions from a user. The user operates the operation unit 4 to input instructions necessary for controlling various operations of the entire endoscope device 1. The operation unit 4 outputs a signal indicating the instruction accepted from the user to the CPU 34. For example, the operation unit 4 is at least one of a button, a switch, a key, a mouse, a joystick, a touchpad, a trackball, and a touch panel.

[0078] The display unit 5 displays an image of the subject based on the video signal output from the video processing unit 31. The display unit 5 also displays operation control details, measurement results, etc. For example, the operation control details are displayed as a menu. For example, the display unit 5 is at least one of a liquid crystal display and an organic EL (Electro Luminescence) display. The display unit 5 may also be a touch panel display. In this case, the operation unit 4 and the display unit 5 are integrated.

[0079] 3 shows the functional configuration of the CPU 34. The CPU 34 functions as a main control unit 340, a display processing unit 341, an image processing unit 342, a visualization processing unit 343, and a recording unit 344. At least one of the blocks shown in FIG. 3 may be configured by a circuit (e.g., a GPU) different from the CPU 34.

[0080] The main control unit 340 controls the processing executed by each unit. When the user operates the operation unit 4, the main control unit 340 accepts the user's operation. The main control unit 340 acquires two or more first images generated under first imaging conditions and two or more second images generated under second imaging conditions from the image sensor 22 via the video processing unit 31. The main control unit 340 outputs the first and second images to the image processing unit 342. Hereinafter, when there is no need to distinguish between the first and second images, the first and second images will simply be referred to as images. The main control unit 340 also executes processing related to switching imaging conditions and turning illumination light on and off. In other words, the main control unit 340 controls the switching control unit 36 ​​and the illumination control unit 33.

[0081] The first image is an image generated based on a first optical image formed in the imaging region S1. The second image is an image generated based on a second optical image formed in the imaging region S1. The first image may be an image generated based on the second optical image, and the second image may be an image generated based on the first optical image.

[0082] The display processing unit 341 generates graphic data for displaying menus and the like. The graphic data generated by the display processing unit 341 is output to the video processing unit 31. The display processing unit 341 also controls the video processing unit 31 to control the state of the image displayed on the display unit 5.

[0083] The image processing unit 342 performs image processing based on the image of the subject. The image processing unit 342 includes a motion detection unit 345, an image selection unit 346, and a measurement processing unit 347.

[0084] The motion detection unit 345 calculates the amount of motion between the image output from the main control unit 340 and a reference image, and based on the amount of motion, detects the motion of the subject or the image sensor 22. For example, the reference image is the first image or the second image.

[0085] The image selection unit 346 selects three or more images based on the amount of motion, and the selected three or more images constitute a set of images that are used to generate three-dimensional (3D) information of the subject.

[0086] The measurement processing unit 347 uses an image group containing three or more images selected by the image selection unit 346 to measure at least one of the shape, dimensions, and distance to the subject (subject distance). For example, the shape of the subject is measured as a 3D point cloud, mesh polygon, or surface. A 3D point cloud is a collection of 3D coordinates of multiple points on the surface of the subject. A mesh polygon is a collection of triangles whose vertices are points included in the 3D point cloud. A surface represents a shape by combining mathematically and geometrically described curved surfaces. The dimensions of the subject are the distance between any two points on the subject and the area of ​​a region on the subject consisting of three or more points. The subject distance is the distance from the tip of the insertion unit 20, in which the image sensor 22 is located, to the subject. Specifically, the subject distance is the distance from the tip surface of the optical adapter 21 to the subject. The subject distance may be the distance from the principal point of the first optical system or the principal point of the second optical system to the subject. The subject distance may also be the distance from the subject-side surface of the lens to the subject.

[0087] The measurement processing unit 347 performs stereo measurement by triangulation using the parallax between the two images. Specifically, the measurement processing unit 347 detects points on the second image that correspond to measurement points set on the first image. This process is called template matching. The measurement processing unit 347 calculates the 3D coordinates of points on the object that correspond to the measurement point coordinates based on the coordinates of the detected points (corresponding point coordinates) and the coordinates of the measurement points (measurement point coordinates). The measurement processing unit 347 may calculate the 3D coordinates of only one point on the surface of the object.

[0088] The results of the image processing executed by the image processing unit 342 are output to the visualization processing unit 343. The visualization processing unit 343 is a data generation unit (data generator). The visualization processing unit 343 generates graphic data that visualizes the image processing results. As a result, the visualization processing unit 343 generates graphic data that expresses the results of reconstructing the 3D shape of the subject as an image.

[0089] In the first embodiment, the illumination light is white light. The image sensor 22 has a red color filter, a green color filter, and a blue color filter. The image sensor 22 generates a color image as an image of a subject. The color image includes two or more pixels. Each pixel has information indicating the brightness of red, green, and blue as a pixel value.

[0090] The image processing unit 342 generates 3D shape data (color 3D point cloud or color textured mesh polygon) in which the 3D coordinates of two or more points on the surface of the subject (3D point cloud) are associated with pixel values ​​corresponding to each of the two or more points. The visualization processing unit 343 generates graphic data that visualizes the 3D shape data. The graphic data corresponds to an image generated by placing the 3D shape data in a virtual space and capturing the 3D shape data with a virtual camera placed in the virtual space.

[0091] The 3D coordinates of each point on the surface of the subject are associated with the pixel value of each point in the 3D shape data generated by the image processing unit 342. At least one of the 3D shape data arranged in the virtual space by the visualization processing unit 343, the position of the virtual camera, the attitude of the virtual camera, and the angle of view of the virtual camera can be changed by the user operating the operation unit 4.

[0092] The graphic data generated by the visualization processing unit 343 is output to the video processing unit 31. The recording unit 344 records the image of the subject in the memory .

[0093] Fig. 4 shows the relationship between the operation of the image sensor 22 in measurement mode, the illumination state, and the imaging condition state. The operation of the endoscope device 1 will be described with reference to Fig. 4. In the example shown in Fig. 4, the imaging region S1 of the image sensor 22 has eight rows. The number of rows in the imaging region S1 is not limited to eight.

[0094] Timing chart TC10 shows the operation of the image sensor 22. In timing chart TC10, the horizontal direction indicates time, and the vertical direction indicates the row position of the cells in the image sensor 22. The top row is row 1, and the bottom row is row 8.

[0095] Timing chart TC10 shows the illumination state, i.e., the state of the white light source 37. The state of the white light source 37 is either on (ON) or off (OFF). Timing chart TC10 also shows the imaging condition, i.e., the imaging optical path. The imaging optical path is either the first optical path L1 or the second optical path L2. Timing chart TC10 also shows the operation of the switching unit 25. The operation of the switching unit 25 is either switching from the first optical path L1 to the second optical path L2 or switching from the second optical path L2 to the first optical path L1.

[0096] In timing chart TC10, the frame rate is 60 fps. The length of each frame period is 1 / 60 seconds. Each frame period includes an exposure period for one row of cells and a readout period for one row of cells. During the readout period for one row, pixel signals from the cells in that row are read out. The readout of pixel signals includes charge transfer and signal readout.

[0097] In timing chart TC10, the frame period is shown based on the start timing of the exposure period for the cells in the first row. The frame periods for the second to eighth rows start a predetermined time later than the frame period for the row immediately preceding each row. Pixel signals accumulated in the cells during the exposure period of frame period n are read out from the cells during the readout period of frame period n. The exposure period of frame period n is the exposure period for each of the first to eighth rows in frame period n. The readout period of frame period n is the period from the start of the readout period for the first row in frame period n to the end of the readout period for the eighth row in frame period n.

[0098] The dashed line L10 in Fig. 4 indicates the start timing of the frame period for each row. Although the dashed line L10 is omitted in timing charts other than timing chart TC10, the meaning of the dashed line L10 is the same throughout this specification. The symbol M10 in Fig. 4 indicates a readout period. The symbol M11 in Fig. 4 indicates an exposure period.

[0099] The following is an outline of the operation of the endoscope device 1. During a first period, the image sensor 22 sequentially reads out pixel signals from at least some of the two or more cells, row by row. The image sensor 22 generates an image of the subject during each of two or more frame periods based on at least some of the pixel signals (pixel signals of effective pixels) read out from at least some of the two or more cells.

[0100] The illumination control unit 33 causes the white light source 37 to generate illumination light during a second period. The second period is at least a part of the period excluding the first period. The illumination control unit 33 causes the white light source 37 to stop generating illumination light during a third period. The third period is the entire period excluding the second period and includes the first period. The second period and the third period are repeated alternately. The switching control unit 36 ​​causes the switching unit 25 to start switching the imaging conditions during the third period and complete the switching of the imaging conditions during the third period. The operations of the illumination control unit 33 and the switching control unit 36 ​​are controlled by the main control unit 340 of the CPU 34.

[0101] The operation of the endoscope device 1 will now be described in detail. At the start of the exposure period of frame period i, the cells in the first row are reset. This starts the exposure period for the cells in the first row. During the exposure period, signals are accumulated in the cells based on light incident on the cells. After the exposure period for the cells in the first row starts, the exposure period for the cells in the second row starts. Similarly, the exposure periods for the cells in the third to eighth rows start sequentially.

[0102] The image sensor 22 sequentially resets the cells of two or more rows one by one in succession, and by this resetting, the image sensor 22 sequentially starts the exposure period of the cells of two or more rows.

[0103] In the frame period (i-1) immediately preceding the frame period i, the imaging optical path is the second optical path L2. When the exposure period for the cells in the first row in frame period i begins, switching of the imaging optical path begins. The switching control unit 36 ​​outputs a control signal for switching the imaging optical path to the switching unit 25. In response to this, the switching control unit 36 ​​causes the switching unit 25 to switch the imaging optical path. The switching unit 25 begins switching from the second optical path L2 to the first optical path L1 based on the control signal output from the switching control unit 36.

[0104] The start of switching from the second optical path L2 to the first optical path L1 need only be simultaneous with or after the start of readout of pixel signals from the cells in the first row in frame period (i-1) (the start of the readout period of frame period (i-1)). Furthermore, the timing of outputting a control signal from the imaging control unit 30 may be earlier than the start of readout of pixel signals from the cells in the first row by a time that is equal to or shorter than the time difference between the output of the control signal and the start of switching of the imaging optical path. This allows the start of switching of the imaging optical path to be simultaneous with or after the start of readout of pixel signals from the cells in the first row.

[0105] When the exposure period for the cells in the third row in frame period i starts, the switching unit 25 completes switching from the second optical path L2 to the first optical path L1. The imaging optical path for the image acquired in frame period i is the first optical path L1. Completion of switching from the second optical path L2 to the first optical path L1 need only be simultaneous with or before the start of the exposure period for the cells in the eighth row in frame period i (the end of the readout period of frame period (i-1)).

[0106] When the exposure period of the cells in the first row in frame period i starts, the white light source 37 is turned off. When the exposure period of the cells in the eighth row in frame period i starts, the white light source 37 starts to be turned on. The illumination control unit 33 outputs a control signal for turning on the white light source 37 to the white light source 37, thereby turning on the white light source 37. The white light source 37 starts to be turned on based on the control signal output from the illumination control unit 33. The turning on of the white light source 37 may start after the start of the exposure period of the cells in the eighth row in frame period i (the end of the readout period of frame period (i-1)).

[0107] When the exposure period of the cells in the first row in frame period i ends, the readout period of the cells in the first row in frame period i begins. The cells in the first row output pixel signals. When a predetermined time has elapsed since the start of the readout period of the cells in the first row, the readout period of the cells in the first row ends. At this time, the cells in the first row are reset, and the exposure period of the cells in the first row in frame period (i+1) begins.

[0108] When the readout period for the cells in the first row in frame period i starts, the white light source 37 stops being lit. The illumination control unit 33 outputs a control signal for turning off the white light source 37 to the white light source 37, thereby turning off the white light source 37. The white light source 37 turns off based on the control signal output from the illumination control unit 33. The white light source 37 may stop being lit before the readout period for the cells in the first row in frame period i starts (the readout period of frame period i starts).

[0109] In timing chart TC10 shown in FIG. 4 , the illumination control unit 33 intermittently turns on the white light source 37. The white light source 37 turns on intermittently. The period during which the white light source 37 remains on overlaps with the exposure periods of two or more rows of cells. During the period during which the white light source 37 remains on, the cells of all rows are simultaneously exposed. The period (image capture period) during which light from the subject enters the cells due to the white light source 37 being turned on is common to all rows of cells. In other words, global exposure (simultaneous exposure) of the cells of two or more rows is performed based on the control of the timing of turning on and off the white light source 37. The period during which the white light source 37 remains on does not include the cell readout period.

[0110] When the readout period for the cells in the first row in frame period i ends, the readout period for the cells in the second row in frame period i begins. The cells in the second row output pixel signals. When a predetermined time has elapsed since the readout period for the cells in the second row began, the readout period for the cells in the second row ends. At this time, the cells in the second row are reset, and an exposure period for the cells in the second row in frame period (i+1) begins. Similarly, readout periods for the cells in the third to eighth rows begin sequentially, and pixel signals from the cells in the third to eighth rows are read out sequentially. The cells in the third to eighth rows are reset sequentially, and an exposure period for the cells in the third to eighth rows in frame period (i+1) begins sequentially.

[0111] The image sensor 22 continuously scans the cells of two or more rows one by one, and sequentially reads out pixel signals from the cells of two or more rows through this scanning.

[0112] The driving method of the image sensor 22 is the rolling shutter method. In the rolling shutter method, the row to be read out is changed row by row during the readout of one frame period, and pixel signals are read out continuously from the cells of each row. In the rolling shutter method, an exposure period is started sequentially for each row, and pixel signals are read out sequentially for each row. The cells of a row whose pixel signals have been read out are reset, and the exposure period for the next frame period begins.

[0113] When the readout period for the cells in the first row in frame period i ends, switching of the imaging optical path begins. The switching control unit 36 ​​outputs a control signal for switching the imaging optical path to the switching unit 25. As a result, the switching control unit 36 ​​causes the switching unit 25 to switch the imaging optical path. The switching unit 25 begins switching from the first optical path L1 to the second optical path L2 based on the control signal output from the switching control unit 36. The start of switching from the first optical path L1 to the second optical path L2 need only be simultaneous with or after the start of readout of the cells in the first row in frame period i (the start of the readout period of frame period i).

[0114] When the readout period for the cells in the third row in frame period i ends, the switching unit 25 completes switching from the first optical path L1 to the second optical path L2. The imaging optical path for the image acquired in frame period (i+1) is the second optical path L2. Completion of switching from the first optical path L1 to the second optical path L2 need only be simultaneous with or before the start of the exposure period for the cells in the eighth row in frame period (i+1) (the end of the readout period for frame period i).

[0115] When the readout period for the cells in the eighth row in frame period i ends, the white light source 37 starts to light up. The illumination control unit 33 outputs a control signal for lighting to the white light source 37, thereby turning on the white light source 37. The white light source 37 starts to light up based on the control signal output from the illumination control unit 33. The white light source 37 may start to light up after the readout period for the cells in the eighth row in frame period i ends (the readout period of frame period i ends).

[0116] In each frame period before frame period i, the image sensor 22 performs the same operation as in frame period i. In each frame period before frame period i, the white light source 37 is intermittently turned on. In each frame period before frame period i, the switching unit 25 switches the image pickup optical path to the first optical path L1 or the second optical path L2.

[0117] When the exposure period of the cells in the first row in frame period (i+1) ends, the readout period of the cells in the first row in frame period (i+1) begins. Similar to the operation in frame period i, the readout periods of the cells in rows 1 to 8 begin sequentially, and pixel signals from the cells in rows 1 to 8 are read out sequentially. The cells in rows 1 to 8 are reset sequentially, and the exposure period of the cells in rows 1 to 8 in frame period (i+2) begins sequentially.

[0118] When the readout period for the cells in the first row in frame period (i+1) starts, the white light source 37 stops being lit. The illumination control unit 33 outputs a control signal for turning off the white light source 37 to the white light source 37, thereby turning off the white light source 37. The white light source 37 turns off based on the control signal output from the illumination control unit 33. The white light source 37 may stop being lit before the readout period for the cells in the first row in frame period (i+1) starts (the readout period of frame period (i+1) starts).

[0119] When the readout period for the cells in the first row in frame period (i+1) ends, switching of the imaging optical path begins. The switching control unit 36 ​​outputs a control signal for switching the imaging optical path to the switching unit 25. As a result, the switching control unit 36 ​​causes the switching unit 25 to switch the imaging optical path. The switching unit 25 begins switching from the second optical path L2 to the first optical path L1 based on the control signal output from the switching control unit 36. The start of switching from the second optical path L2 to the first optical path L1 need only be simultaneous with or after the start of readout of the cells in the first row in frame period (i+1) (the start of the readout period of frame period (i+1)).

[0120] When the readout period for the cells in the third row in frame period (i+1) ends, the switching unit 25 completes switching from the second optical path L2 to the first optical path L1. The imaging optical path for the image acquired in frame period (i+2) is the first optical path L1. Completion of switching from the second optical path L2 to the first optical path L1 need only be simultaneous with or before the start of the exposure period for the cells in the eighth row in frame period (i+2) (the end of the readout period for frame period (i+1)).

[0121] The imaging device 22 performs the same operations in frame period (i+2) and frame period (i+3) as in frame period i.

[0122] The white light source 37 is turned on during the exposure periods of each of frame periods (i+2) and (i+3). The white light source 37 is repeatedly turned on and off. The white light source 37 is turned on for each frame period and turned off for each frame period.

[0123] The switching unit 25 switches the imaging optical path during the readout period in each of frame periods (i+2) and (i+3). The switching unit 25 repeatedly switches from the first optical path L1 to the second optical path L2 and from the second optical path L2 to the first optical path L1. The switching unit 25 switches the imaging optical path for each frame period.

[0124] While the endoscope device 1 is operating in the measurement mode, the endoscope device 1 continuously or intermittently repeats the operations from frame period i to frame period (i+3).

[0125] In the timing chart TC10 shown in FIG. 4 , pixel signals are read from cells in rows 1 to 8. That is, pixel signals are read from all of two or more cells. Pixel signals may be read from cells in only some rows. For example, pixel signals may be read from cells in rows excluding rows consisting of only optical black pixels, or from cells in only effective rows that include effective pixels. When pixel signals are read from cells in only some rows, the period during which the white light source 37 remains lit may overlap with the period during which cells in only some of the rows are simultaneously exposed.

[0126] The numbers in the first to eighth rows indicate the readout order. These numbers may differ from the physical order on the imaging surface. For example, the cells may be physically arranged in descending order starting from the eighth row. Alternatively, the cells may be arranged in the order of the first row, fifth row, second row, sixth row, third row, seventh row, fourth row, and eighth row in a predetermined direction.

[0127] The image sensor 22 sequentially reads out pixel signals row by row from at least some of the two or more cells during the first period T1. The image sensor 22 sequentially reads out pixel signals row by row from cells in a simultaneously exposed row during the first period T1. The simultaneously exposed row is at least some of two or more rows in an array of two or more cells. The simultaneously exposed row includes cells that are simultaneously exposed by lighting the white light source 37.

[0128] 4, all rows are simultaneously exposed. The first period T1 of the frame period n is the period from the start of readout of pixel signals from the cells in the first row of the frame period n to the completion of readout of pixel signals from the cells in the eighth row (the readout period of the frame period n).

[0129] In the timing chart TC10 shown in Fig. 4, the white light source 37 is turned on simultaneously with the completion of the readout of pixel signals in one frame period. In other words, the white light source 37 is turned on simultaneously with the completion of the first period T1. The white light source 37 is turned off simultaneously with the start of the readout of pixel signals in one frame period. In other words, the white light source 37 is turned off simultaneously with the start of the first period T1.

[0130] The illumination control unit 33 causes the white light source 37 to generate illumination light during the second period T2. The white light source 37 continues to be lit during the second period T2. The second period T2 is at least a part of the period excluding the first period T1. In the timing chart TC10 shown in FIG. 4 , the second period T2 is the entire period excluding the first period T1.

[0131] For example, when there is a margin for power consumption, such as when the endoscope device 1 is connected to an AC adapter, the second period T2 can be lengthened by shortening the first period T1. In other words, the exposure time is lengthened, thereby improving the imaging sensitivity.

[0132] The illumination control unit 33 causes the white light source 37 to stop generating illumination light during the third period T3. The white light source 37 remains off during the third period T3. The third period T3 is the entire period excluding the second period T2. The third period T3 is the period including the entire first period T1. In the timing chart TC10 shown in FIG. 4, the third period T3 is the same as the first period T1.

[0133] The second period T2 overlaps with the exposure periods of two or more rows in the array of two or more cells. The second period T2 begins when the readout period of the row scanned last by the image sensor 22 in one frame period is completed. The second period T2 ends when the readout period of the row scanned first by the image sensor 22 in one frame period is completed.

[0134] The switching control unit 36 ​​causes the switching unit 25 to start switching the imaging optical path in the third period T3 and complete switching of the imaging optical path in the third period T3. That is, the switching control unit 36 ​​causes the switching unit 25 to complete switching of the imaging optical path before the start of the next second period T2. In the timing chart TC10 shown in FIG. 4 , the third period T3 is the same as the first period T1, so the switching unit 25 starts switching the imaging optical path in the first period T1 and completes switching of the imaging optical path in the first period T1.

[0135] During the first period T1, the image sensor 22 reads out pixel signals for one frame in a period equal to or less than half the sum of the exposure and readout periods for the cells in each row of two or more frame periods. In other words, the image sensor 22 reads out pixel signals for one frame in a period equal to or less than half the frame period. By shortening the time required to read out pixel signals from the cells, the image sensor 22 can lengthen the period during which at least a portion of two or more cells can be simultaneously exposed (the simultaneous exposure period). This improves imaging sensitivity. In the timing chart TC10 shown in FIG. 4, the frame period is approximately 1 / 60 seconds, and the simultaneous exposure period for all cells is approximately 1 / 120 seconds.

[0136] The length of the period during which the image sensor 22 reads out pixel signals from the cells of one row is equal to or less than Tr. The time Tr is expressed by the following formula (1). The number m is an integer equal to or greater than 2 and is the number of simultaneously exposed rows of the image sensor 22. The simultaneously exposed period of the m rows of cells includes all of the second period T2 during which the white light source 37 remains lit. In formula (1), the time Tf is the length of the frame period. Tr=Tf / (m×2) (1)

[0137] The second period T2 does not overlap with the third period in which the imaging optical path is switched. The imaging optical path is fixed while the white light source 37 is on. That is, the imaging optical path is fixed while two or more cells are exposed. The image output from the imaging element 22 is generated based on only one of the first optical image and the second optical image.

[0138] The measurement processing unit 347 performs measurement processing based on the first image and the second image. The first image is acquired in a first frame period, and the second image is acquired in a second frame period. The first frame period and the second frame period are two consecutive frame periods. The interval between the first acquisition timing and the second acquisition timing is the same as one frame period. The first acquisition timing is the timing at which the image sensor 22 acquires the first image. The second acquisition timing is the timing at which the image sensor 22 acquires the second image.

[0139] 5 shows the procedure of the operation of the endoscope device 1. The operation of the endoscope device 1 will be described with reference to FIG.

[0140] When the endoscope device 1 is started up, the endoscope device 1 operates in an observation mode. When the optical adapter 21 is attached to the tip of the insertion section 20, the endoscope device 1 operates in a measurement mode.

[0141] When the operation mode of the endoscope device 1 is switched to the measurement mode, initial settings are performed. In the initial settings, the imaging control unit 30 sets simultaneous exposure rows in the image sensor 22. The simultaneous exposure rows include cells that are simultaneously exposed by turning on the white light source 37. In the timing chart TC10 shown in FIG. 4, all rows in an array of two or more cells are simultaneously exposed. In the initial settings, the imaging control unit 30 sets the length of a frame period in the image sensor 22. In the initial settings, the imaging control unit 30 sets the length of a first period T1 in which the image sensor 22 reads pixel signals in the image sensor 22. The illumination control unit 33 turns on the white light source 37 (step S100).

[0142] After step S100, the switching control unit 36 ​​outputs a control signal for switching the imaging optical path to the switching unit 25. As a result, the switching control unit 36 ​​causes the switching unit 25 to start switching the imaging optical path. The switching unit 25 starts switching from the second optical path L2 to the first optical path L1 based on the control signal output from the switching control unit 36. Thereafter, the switching control unit 36 ​​causes the switching unit 25 to complete switching of the imaging optical path (step S105). If the imaging optical path is already the first optical path L1 in step S105, the processing in step S105 is not necessary.

[0143] After step S105, the image sensor 22 generates one frame of image and outputs the generated image. The image sensor 22 continuously scans two or more rows of cells one by one during each frame period, and sequentially reads out pixel signals from the two or more rows of cells during each frame period. The image sensor 22 outputs an image based on the pixel signals of the two or more rows of cells during each frame period. Because the imaging optical path is the first optical path L1, the image sensor 22 outputs the first image. The image processor 31 processes the first image output from the image sensor 22. The first image processed by the image processor 31 is output to the CPU 34. The main controller 340 acquires the first image (step S110).

[0144] After step S110, the display processing unit 341 displays the first image generated in step S110 on the display unit 5. The display unit 5 displays the first image (step S115).

[0145] After step S115, the main control unit 340 determines whether or not to perform measurement (step S120). For example, when a user inputs a measurement instruction by operating the operation unit 4, the main control unit 340 determines to perform measurement in step S120. Measurement may be performed at predetermined time intervals. For example, the main control unit 340 may determine to perform measurement when a predetermined time has elapsed since the endoscope device 1 was started. Alternatively, the main control unit 340 may determine to perform measurement when a predetermined time has elapsed since the previous measurement was performed.

[0146] When the main control unit 340 determines in step S120 that measurement is not to be performed, the process proceeds to step S105. The imaging optical path is maintained as the first optical path L1 until a measurement instruction is input. Until a measurement instruction is input, the imaging element 22 sequentially outputs first images, and the display unit 5 sequentially updates and displays the first images.

[0147] When the main control unit 340 determines in step S120 that measurement is to be performed, the CPU 34 executes an image acquisition process to acquire candidates for images to be used in the measurement process (step S125).

[0148] After step S125, the CPU 34 executes an image selection process to select an image to be used in the measurement process (step S130).

[0149] After step S130, the main control unit 340 determines whether or not the image selection was successful based on the result of the processing in step S130 (step S135). If the main control unit 340 determines in step S135 that the image selection was unsuccessful, step S125 is executed.

[0150] When the main control unit 340 determines in step S135 that the image selection has been successful, the CPU 34 executes a measurement process including a 3D reconstruction process for reconstructing the 3D shape of the subject and a dimension measurement process for measuring the dimensions of the subject (step S140). After step S140, step S105 is executed.

[0151] 6 shows the procedure of the operation of the endoscope device 1 in the image acquisition process (step S125). The operation of the endoscope device 1 in the image acquisition process will be described with reference to FIG.

[0152] When the main control unit 340 determines in step S120 that measurement is to be performed, the imaging optical path is the first optical path L1. The imaging element 22 generates one frame of image and outputs the generated image. Because the imaging optical path is the first optical path L1, the imaging element 22 outputs the first image. The illumination control unit 33 turns on the white light source 37 during the second period. The illumination control unit 33 turns off the white light source 37 during the third period (step S1250).

[0153] After step S1250, the switching control unit 36 ​​outputs a control signal for switching the imaging optical path to the switching unit 25. As a result, the switching control unit 36 ​​causes the switching unit 25 to start switching the imaging optical path. The switching unit 25 starts switching from the first optical path L1 to the second optical path L2 based on the control signal from the switching control unit 36. Thereafter, the switching control unit 36 ​​causes the switching unit 25 to complete switching of the imaging optical path (step S1251). In reality, step S1251 is executed while the image sensor 22 is reading out pixel signals in step S1250.

[0154] After step S1251, the display processing unit 341 displays the first image generated in step S1250 on the display unit 5. The display unit 5 displays the first image (step S1252). Step S1252 may be omitted.

[0155] After step S1252, the image sensor 22 generates one frame of image and outputs the generated image. Because the imaging optical path is the second optical path L2, the image sensor 22 outputs the second image. The image processing unit 31 processes the second image output from the image sensor 22. The second image processed by the image processing unit 31 is output to the CPU 34. The main control unit 340 acquires the second image. The illumination control unit 33 turns on the white light source 37 during the second period. The illumination control unit 33 turns off the white light source 37 during the third period (step S1253).

[0156] After step S1253, the main control unit 340 determines whether or not the acquisition of a predetermined number of images has been completed (step S1254). The predetermined number is (2n+1), where n is an integer of 2 or greater.

[0157] In the example described below, the motion detection unit 345 uses two or more image sets to determine whether measurement is possible. Each image set includes four or more images, and the four or more images include two or more first images and two or more second images. In the example described below, each image set includes five images. For example, each image set includes three first images and two second images. Alternatively, each image set includes three second images and two first images. It is desirable that the predetermined number (2n+1) is sufficiently greater than 5. For example, the predetermined number is 15.

[0158] When the main control unit 340 determines in step S1254 that acquisition of the predetermined number of images has not been completed, step S1255 is executed. In step S1255, the same processing as step S105 is executed. After step S1255, step S1250 is executed. The imaging optical path repeatedly switches between the first optical path L1 and the second optical path L2 until the predetermined number of images is acquired. The image sensor 22 repeatedly outputs the first image and the second image. Visibility deteriorates when the first image and the second image, which have parallax, are alternately displayed. Therefore, the display unit 5 continues to display the first image by sequentially updating it without displaying the second image. When the main control unit 340 determines in step S1254 that acquisition of the predetermined number of images has been completed, the image acquisition process ends.

[0159] 7 shows the procedure of the operation of the endoscope device 1 in the image selection process (step S130). The operation of the endoscope device 1 in the image selection process will be described with reference to FIG.

[0160] The motion detection unit 345 uses a predetermined number of images acquired in the image acquisition process to perform motion detection processing (step S1300).

[0161] The motion detection process in step S1300 will be described in detail. In the image acquisition process, (k-4) image sets are acquired. The number k is the number of images acquired in the image acquisition process (2n+1). In an example where 15 images are acquired, 11 image sets are acquired. The motion detection unit 345 uses each of the 11 image sets to detect motion in the images. The motion in the images is the motion of the subject or the image sensor 22.

[0162] 8 shows examples of image sets used in the image selection process. The image sensor 22 outputs first images L(t) and L(t+2), etc., and second images R(t+1) and R(t+3), etc. For example, the main control unit 340 acquires image sets SET1 to SET11.

[0163] Image set SET1 includes first images L(t), L(t+2), and L(t+4) and second images R(t+1) and R(t+3). Image set SET2 includes first images L(t+2) and L(t+4) and second images R(t+1), R(t+3), and R(t+5). Image set SET3 includes first images L(t+2), L(t+4), and L(t+6) and second images R(t+3) and R(t+5). Image sets SET4 to SET11 also include five images, like image sets SET1 to SET3.

[0164] For example, the motion detection unit 345 uses the image set SET1 to calculate a first motion amount MA1, a second motion amount MA2, and a third motion amount MA3 shown in Fig. 9. Fig. 9 shows an example of a combination of images used to calculate the motion amount of an image.

[0165] Specifically, the motion detection unit 345 uses the first image L(t) as a reference image and calculates a first motion amount MA1 that indicates the amount of motion of the first image L(t+2) relative to the first image L(t). The motion detection unit 345 uses the second image R(t+1) as a reference image and calculates a second motion amount MA2 that indicates the amount of motion of the second image R(t+3) relative to the second image R(t+1). The motion detection unit 345 uses the first image L(t+2) as a reference image and calculates a third motion amount MA3 that indicates the amount of motion of the first image L(t+4) relative to the first image L(t+2).

[0166] Furthermore, the motion detection unit 345 uses the image set SET2 to calculate a first motion amount, a second motion amount, and a third motion amount. Specifically, the motion detection unit 345 uses the second image R(t+1) as a reference image and calculates a first motion amount indicating the motion amount of the second image R(t+3) relative to the second image R(t+1). The motion detection unit 345 uses the first image L(t+2) as a reference image and calculates a second motion amount indicating the motion amount of the first image L(t+4) relative to the first image L(t+2). The motion detection unit 345 uses the second image R(t+3) as a reference image and calculates a third motion amount indicating the motion amount of the second image R(t+5) relative to the second image R(t+3).

[0167] The motion detection unit 345 uses each of the image sets SET3 to SET11 and performs the same process as described above to calculate the first motion amount, the second motion amount, and the third motion amount for each image set. The motion detection unit 345 generates motion information corresponding to each motion amount.

[0168] Fig. 10 shows the procedure for a process of generating motion information. The operation of the motion detection unit 345 will be described with reference to Fig. 10. In the following example, the motion detection unit 345 uses two first images to generate motion information. When the motion detection unit 345 uses two second images to generate motion information, the motion detection unit 345 performs a process similar to the process shown in Fig. 10.

[0169] The motion detection unit 345 uses the two first images to perform template matching for each region (step S1300a).

[0170] Step S1300a will be described in detail with reference to Figures 11A and 11B. The motion detection unit 345 uses a reference image 500 shown in Figure 11A and an image 501 shown in Figure 11B. The reference image 500 is generated before the image 501 is generated. The reference image 500 may also be generated after the image 501 is generated.

[0171] Template regions 502 are arranged at regular intervals in reference image 500. Motion detection unit 345 searches for a region in image 501 that matches an image in template region 502. A matching search range 503 of a certain range is set in image 501, with the position of template region 502 in reference image 500 as its center.

[0172] The motion detection unit 345 calculates a matching position 504 within the matching search range 503 at which the degree of match between the matching search range 503 of the image 501 and the template region 502 of the reference image 500 is highest. As an index indicating this degree of match, known indices such as SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), NCC (Normalized Cross Correlation), and ZNCC (Zero Means Normalized Cross Correlation) can be used.

[0173] In step S1300a, the motion detection unit 345 calculates the amount of positional deviation between the matching position 504 in the image 501 and the position of each template region 502 in the reference image 500 as a movement vector of each template region 502.

[0174] In the template matching process, if there is a clue that uniquely identifies corresponding points, a highly reliable motion vector can be calculated. However, in low-contrast areas and areas with repeated patterns where there is no clue for alignment, the reliability of the motion vector is low. After step S1300a, the motion detection unit 345 determines the reliability of the motion vector for each template area 502 (step S1300b).

[0175] A known method can be used to determine the reliability of a motion vector. For example, the reliability of motion vector 505 in FIG. 11B is high, and the reliability of motion vector 506 in FIG. 11B is low. After step S1300b, motion detection unit 345 calculates a global vector by averaging the motion vectors with high reliability calculated in step S1300b among the motion vectors calculated in step S1300a (step S1300c). The global vector corresponds to the motion information.

[0176] In this case, the motion detection unit 345 may calculate the global vector from only the motion vectors in the central region of the image. Alternatively, the motion detection unit 345 may calculate the global vector by using the reliability value as a weight and calculating a weighted average. The magnitude of the global vector indicates the amount of positional shift between the two first images, and the direction of the global vector indicates the direction of positional shift between the two first images. The magnitude of the global vector corresponds to each of the first motion amount, the second motion amount, and the third motion amount.

[0177] The motion detection unit 345 performs the motion detection process described above. The video processing unit 31 may also perform the motion detection process. In addition to calculating the global vector, the motion detection unit 345 may further calculate index values ​​for the forward / backward movement and twisting movement (roll rotation) of the subject or camera. The forward / backward movement is the movement of the camera moving approximately parallel to the optical axis of the camera's optical system relative to the subject, or the movement of the subject moving approximately parallel to the optical axis relative to the camera. This movement appears on the image as a change in the size of the subject image. The twisting movement is the movement of the camera rotating around the optical axis of the camera's optical system relative to the subject, or the movement of the subject rotating around an axis of rotation approximately parallel to the optical axis of the camera's optical system relative to the camera. This movement appears on the image as a rotation of the subject image. The motion detection unit 345 converts the direction of the movement vector determined to have high reliability into an angle, and calculates the variation (standard deviation) of this angle as index values ​​for the forward / backward movement and twisting movement.

[0178] 7 again, the operation of the endoscope device 1 in the image selection process will be described. After step S1300, the image selection unit 346 uses each image set acquired in the image acquisition process to determine whether measurement is possible (step S1301). In an example in which 11 image sets are acquired in the image acquisition process, the image selection unit 346 determines whether measurement is possible 11 times.

[0179] Step S1301 will now be described in detail. The image selection unit 346 uses the first motion information, second motion information, and third motion information generated in step S1300 to determine the amount of motion (speed), uniformity of motion, and linearity, as described below.

[0180] When the maximum of the three motion amounts is equal to or less than a predetermined first threshold, the image selection unit 346 determines that the motion amount of the image set is small. When the maximum of the three motion amounts is greater than the first threshold, the image selection unit 346 determines that the motion amount of the image set is large.

[0181] The image selection unit 346 quantifies the degree of change in the direction of the global vector. As a result, the image selection unit 346 calculates an index value Fs shown in equation (2). The index value Fs indicates straightness.

[0182]

[0183] In equation (2), Vt and Vt-1 represent two global vectors. The magnitude of the global vector indicates the amount of positional deviation. The image selection unit 346 calculates an index value Fs by using the first motion information and the second motion information, and calculates an index value Fs by using the second motion information and the third motion information. When the two index values ​​Fs are equal to or less than a preset second threshold, the image selection unit 346 determines that the movement of the tip of the insertion unit 20 is linear movement. When at least one of the two index values ​​Fs is greater than the second threshold, the image selection unit 346 determines that the movement of the tip of the insertion unit 20 is not linear movement.

[0184] The image selection unit 346 quantifies the degree of change in the magnitude of the positional deviation amount. As a result, the image selection unit 346 calculates an index value Fd shown in equation (3). The index value Fd indicates uniform velocity. Fd=|Vt|-|Vt-1| (3)

[0185] The image selection unit 346 calculates an index value Fd by using the first movement information and the second movement information, and calculates an index value Fd by using the second movement information and the third movement information. When the two index values ​​Fd are equal to or less than a preset third threshold, the image selection unit 346 determines that the movement of the tip of the insertion unit 20 is uniform movement. When at least one of the two index values ​​Fd is greater than the third threshold, the image selection unit 346 determines that the movement of the tip of the insertion unit 20 is not uniform movement.

[0186] When the amount of movement of the image set is small and the movement of the image set is uniform linear movement, the image selection unit 346 determines that measurement is possible using the image set. When the amount of movement of the image set is large, the movement of the image set is not linear movement, or the movement of the image set is not uniform motion, the image selection unit 346 determines that measurement is not possible using the image set. The image selection unit 346 may further determine that measurement is not possible when it detects forward / backward movement or twisting movement of the subject or camera. In other words, when the index values ​​of forward / backward movement and twisting movement calculated by the motion detection unit 345 are greater than predetermined thresholds, the image selection unit 346 may determine that the movement of the image set is not linear movement and therefore determine that measurement is not possible.

[0187] After step S1301, the image selection unit 346 selects an image set based on either or both of the amount of motion and the image generation timing (step S1302). The image generation timing is the timing at which the image sensor 22 generates an image. For example, the amount of motion used to select the image set is the average of the first amount of motion, the second amount of motion, and the third amount of motion. Note that the amount of motion may be a statistical value other than the average (for example, the median or maximum value). The image set selected by the image selection unit 346 is set as a group of images to be used in the measurement process. When step S1302 is executed, the image selection process ends.

[0188] Details of step S1302 will be described. The image selection unit 346 selects one or more image sets determined in step S1301 to be measurable as candidates for the image group. The image selection unit 346 calculates the time difference between the first timing and the second timing. The first timing is a reference timing, which is the timing at which the user inputs a measurement instruction by operating the operation unit 4. In other words, the first timing is the timing at which a measurement trigger occurs. The second timing is the timing at which the image sensor 22 generates an image included in the image set. In other words, the second timing is the timing at which an image included in the image set is captured.

[0189] For example, the second timing may be the imaging timing of the first image generated among the images included in the image set. The second timing may also be the imaging timing of the last image generated among the images included in the image set. The second timing may also be the average of the imaging timings of five images included in the image set.

[0190] The image selection unit 346 compares the time difference for each image set selected as a candidate for the image group with a fourth threshold value set in advance. If the time difference is smaller than the fourth threshold value, the image selection unit 346 selects the image set used to calculate the time difference as the image group.

[0191] When two or more time differences are equal to or less than the fourth threshold, the image selection unit 346 selects, as the image group, the image set used to calculate the smallest amount of motion. When all time differences are greater than the fourth threshold, the image selection unit 346 selects, as the image group, the image set used to calculate the smallest amount of motion. The image selection unit 346 may select, as the image group, the image set used to calculate the smallest time difference without comparing the time differences with the fourth threshold.

[0192] If the image selection unit 346 determines that measurement is not possible for all of the image sets, the main control unit 340 determines in step S135 that image selection has failed.

[0193] 12 shows the procedure of the operation of the endoscope device 1 in the measurement process (step S140). The operation of the endoscope device 1 in the measurement process will be described with reference to FIG.

[0194] The measurement processing unit 347 generates motion compensation information for compensating for motion in the image group set in the image selection process (step S1400).

[0195] Step S1400 will be described in detail. The image group set in the image selection process includes five images generated at different times. Specifically, the image group includes three first images and two second images. Alternatively, the image group includes three second images and two first images.

[0196] When the image group includes three first images and two second images, the measurement processing unit 347 uses three images excluding the first image generated first and the last first image generated first. For example, the measurement processing unit 347 uses the second image R(t+1), the first image L(t+2), and the second image R(t+3) included in the image set SET1 shown in FIG. 8 .

[0197] If the image group includes three second images and two first images, the measurement processing unit 347 uses three images excluding the first and last generated second images. For example, the measurement processing unit 347 uses the first image L(t+2), the second image R(t+3), and the first image L(t+4) included in the image set SET2 shown in FIG. 8 .

[0198] The measurement processing unit 347 may generate a first corrected image by correcting optical distortion of the first image. The measurement processing unit 347 may also generate a second corrected image by correcting optical distortion of the second image. The measurement processing unit 347 may generate motion compensation information by using the first corrected image and the second corrected image.

[0199] In the following example, the measurement processing unit 347 uses two second images and one first image. Fig. 13 shows an example of images used to generate motion compensation information. The measurement processing unit 347 uses the second image R1, the second image R2, and the first image L1 shown in Fig. 13.

[0200] The measurement processing unit 347 performs template matching processing using the first image L1 as a reference image to search for corresponding points PR1 in the second image R1 that correspond to each pixel PL1 in the first image L1. The measurement processing unit 347 performs this processing for all pixels in the first image L1 to generate a first disparity map as disparity information. The first disparity map indicates the distribution of the amount of disparity between each pixel PL1 in the first image L1 and corresponding points PR1 in the second image R1.

[0201] The measurement processing unit 347 performs template matching using the first image L1 as a reference image to search for corresponding points PR2 in the second image R2 that correspond to each pixel PL1 in the first image L1. The measurement processing unit 347 performs this process for all pixels in the first image L1 to generate a second disparity map as disparity information. The second disparity map indicates the distribution of the amount of disparity between each pixel PL1 in the first image L1 and its corresponding point PR2 in the second image R2. The measurement processing unit 347 calculates the average of the amount of disparity for each pixel in the first disparity map and the amount of disparity for each pixel in the second disparity map to calculate a corrected amount of disparity for each pixel.

[0202] In an example in which the measurement processing unit 347 uses two first images and one second image, the measurement processing unit 347 generates a first disparity map and a second disparity map by performing the following process. Hereinafter, the two first images will be referred to as a first image L1 and a first image L2.

[0203] The measurement processing unit 347 performs template matching processing using the second image as a reference image to search for corresponding points in the first image L1 that correspond to each pixel in the second image. The measurement processing unit 347 performs this processing for all pixels in the second image to generate a first disparity map as disparity information. The first disparity map indicates the distribution of the amount of disparity between each pixel in the second image R1 and the corresponding point in the first image L1.

[0204] The measurement processing unit 347 performs template matching processing using the second image as a reference image to search for corresponding points in the first image L2 that correspond to each pixel in the second image. The measurement processing unit 347 performs this processing for all pixels in the second image to generate a second disparity map as disparity information. The second disparity map indicates the distribution of the amount of disparity between each pixel in the second image R1 and the corresponding point in the first image L2. The measurement processing unit 347 calculates the average of the amount of disparity for each pixel in the first disparity map and the amount of disparity for each pixel in the second disparity map to calculate a corrected amount of disparity for each pixel.

[0205] In another example in which the measurement processing unit 347 uses two second images and one first image, the measurement processing unit 347 generates a first disparity map and a second disparity map by performing the following process. Hereinafter, the two second images will be referred to as second image R1 and second image R2.

[0206] The measurement processing unit 347 performs template matching processing using the second image R1 as a reference image to search for corresponding points in the first image L1 that correspond to each pixel in the second image R1. The measurement processing unit 347 performs this processing for all pixels in the second image R1 to generate a first disparity map as disparity information. The first disparity map indicates the distribution of the amount of disparity between each pixel in the second image R1 and corresponding points in the first image L1.

[0207] The measurement processing unit 347 performs template matching processing using the second image R1 as a reference image to search for corresponding points in the second image R2 that correspond to each pixel in the second image R1. The measurement processing unit 347 performs this processing for all pixels in the second image R1 to generate a second disparity map as disparity information. The second disparity map indicates the distribution of the amount of disparity between each pixel in the second image R1 and its corresponding point in the second image R2. The measurement processing unit 347 calculates a corrected amount of disparity for each pixel by adding half the amount of disparity for each pixel in the second disparity map to the amount of disparity for each pixel in the first disparity map.

[0208] The measurement processing unit 347 may perform a process similar to the above process by using two first images and one second image. For example, the measurement processing unit 347 may use the first image L1 as a reference image, generate a first disparity map based on the first image L1 and the second image R1, and generate a second disparity map based on the first image L1 and the first image L2.

[0209] Each pixel in the first disparity map and each pixel in the second disparity map are associated with the same pixel in the reference image. The measurement processing unit 347 generates a corrected disparity map including a corrected amount of disparity for each pixel. The corrected disparity map constitutes motion compensation information. Each amount of disparity included in the corrected disparity map is associated with a pixel in the first image or the second image used as the reference image. The measurement processing unit 347 compensates for motion in the first image or the second image used as the reference image by generating the corrected disparity map.

[0210] After step S1400, the measurement processing unit 347 executes 3D reconstruction processing based on the images included in the image group and the motion compensation information (step S1401).

[0211] Details of step S1401 will be described. The measurement processing unit 347 uses the corrected parallax map generated in step S1400 to calculate the 3D coordinates (3D point cloud) of two or more points on the surface of the subject by triangulation. The calculated 3D coordinates correspond to the 3D coordinates of pixel PL1 in the first image L1 shown in FIG. 13 and the 3D coordinates of corresponding point PR3 in the second image R3 shown in FIG. 13. The second image R3 is a virtual image generated at the same time as the first image L1 was generated. The measurement processing unit 347 associates the calculated 3D coordinates with pixel PL1 in the first image L1. The measurement processing unit 347 repeats the above process to calculate the 3D coordinates corresponding to each pixel in the first image L1.

[0212] In reality, the image sensor 22 does not generate the second image R3. The measurement processing unit 347 can perform stereo measurement similar to stereo measurement using the first image L1 and the second image R3 generated at the same time. Therefore, the measurement processing unit 347 can compensate for errors in the amount of parallax caused by movement in the images, and can suppress increases in measurement errors.

[0213] The measurement processing unit 347 generates the first and second disparity maps in step S1400 by using the same reference image. Therefore, the measurement processing unit 347 does not need to perform alignment between the first and second disparity maps to generate the corrected disparity map. This prevents an increase in the processing time required for alignment and prevents a decrease in the accuracy of the 3D shape data due to alignment errors.

[0214] After step S1401, the measurement processing unit 347 generates color 3D shape data by associating the color information of each pixel of the first image or the second image used as the reference image with the 3D coordinates of each point (step S1402).

[0215] The color 3D shape data includes color information for each pixel of the first image or the second image used as the reference image and 3D coordinates corresponding to each pixel. The color information includes a red pixel value, a green pixel value, and a blue pixel value. In the color 3D shape data, the color information and the 3D coordinates are associated with each pixel.

[0216] The measurement processing unit 347 may convert the 3D point cloud into two or more meshes. The measurement processing unit 347 may generate a texture image based on the first image or the second image used as the reference image. The measurement processing unit 347 may generate a mesh polygon by combining two or more meshes and texture images. The color 3D shape data may be composed of mesh polygons.

[0217] After step S1402, the visualization processing unit 343 executes the visualization processing. In the visualization processing, the visualization processing unit 343 generates a perspective projection image by arranging the color 3D shape data in a virtual space. The perspective projection image is equivalent to an image generated when a virtual camera placed in the virtual space captures an image of a subject in the virtual space. The CPU 34 outputs the generated perspective projection image to the video processing unit 31. The video processing unit 31 outputs the perspective projection image to the display unit 5 (step S1403).

[0218] After step S1403, the display processing unit 341 displays the perspective projection image generated in step S1403 on the display unit 5. The display unit 5 displays the perspective projection image. The display processing unit 341 may display the perspective projection image on the display unit 5 instead of the first image, or may display the first image and the perspective projection image simultaneously on the display unit 5 (step S1404).

[0219] Step S1404 may be omitted. The visualization processing unit 343 may be omitted. Therefore, step S1403 may be omitted.

[0220] After step S1404, the measurement processing unit 347 uses the 3D coordinates calculated in step S1401 to perform dimension measurement processing (step S1405). When step S1405 is executed, the measurement processing ends.

[0221] Step S1405 will now be described in detail. The user specifies two or more points on the surface of the subject by operating the operation unit 4. The measurement processing unit 347 measures the dimensions of the subject based on the 3D coordinates of the two or more points specified by the user. Instead of measuring the dimensions of the subject, the measurement processing unit 347 may measure the object distance at one point on the surface of the subject.

[0222] In the above example, the image selection unit 346 determines that measurement is possible when the movement of the subject or the image sensor 22 is a uniform linear movement. When the movement of the subject or the image sensor 22 is a uniform linear movement with a uniform acceleration, the image selection unit 346 may determine that measurement is possible, and the measurement processing unit 347 may generate a corrected parallax map taking into account weights according to the first amount of movement and the third amount of movement.

[0223] The endoscope device 1 (imaging system) according to each aspect of the present invention includes an imaging element 22 (image sensor) and a CPU 34 (controller). The imaging element 22 generates an image of a subject under one of two or more imaging conditions, including a first imaging condition and a second imaging condition different from the first imaging condition. The CPU 34 continuously switches between the two or more imaging conditions. The CPU 34 acquires four or more images from the imaging element 22, including two or more first images generated under the first imaging condition and two or more second images generated under the second imaging condition. The CPU 34 calculates the amount of motion between a reference image included in the four or more images and an image other than the reference image included in the four or more images. Based on the amount of motion, the CPU 34 sets an image group including three or more images from the four or more images. The image group includes at least one first image of the two or more first images and at least one second image of the two or more second images. The CPU 34 generates motion compensation information for compensating for motion in the image group based on the at least one first image and the at least one second image, and generates 3D information of the subject based on the images included in the image group and the motion compensation information.

[0224] The reference image is included in the four or more images. In the first embodiment, the reference image is the first image or the second image. When the image group includes only one first image, the image group includes at least two second images. When the image group includes only one second image, the image group includes at least two first images. As described above, the first image may be an image generated based on the second optical image, and the second image may be an image generated based on the first optical image.

[0225] The imaging method according to each aspect of the present invention includes first to sixth steps. In a first step (steps S1251 and S1255), the CPU 34 continuously switches between two or more imaging conditions to cause the image sensor 22 to generate images of a subject under one of two or more imaging conditions, including a first imaging condition and a second imaging condition different from the first imaging condition. In a second step (steps S1250 and S1253), the CPU 34 acquires four or more images from the image sensor 22, including two or more first images generated under the first imaging condition and two or more second images generated under the second imaging condition. In a third step (step S1300), the CPU 34 calculates the amount of motion between a reference image included in the four or more images and an image other than the reference image included in the four or more images. In a fourth step (step S1302), the CPU 34 sets an image group including three or more images from the four or more images based on the amount of motion. In a fifth step (step S1400), the CPU 34 generates motion compensation information based on at least one first image and at least one second image included in the image group. In a sixth step (step S1401), the CPU 34 generates 3D information of the subject based on the images included in the image group and the motion compensation information.

[0226] The program according to each aspect of the present invention causes a computer to execute the first to sixth steps described above.

[0227] Each aspect of the present invention may include the following modifications: The CPU 34 generates disparity information based on at least one first image and at least one second image included in the image group, and generates motion compensation information based on the disparity information.

[0228] Each aspect of the present invention may include the following modifications. The CPU 34 generates two or more sets of disparity information based on at least two first images and at least one second image. For example, the CPU 34 uses the second image as a reference image and generates first disparity information (first disparity map) based on the first image and the reference image. The CPU 34 also generates second disparity information (second disparity map) based on the first image and the reference image. The first image used to generate the second disparity information is different from the first image used to generate the first disparity information.

[0229] Each aspect of the present invention may include the following modifications: The CPU 34 generates two or more sets of disparity information by executing a first process and a second process; The CPU 34 executes the first process based on a second image included in at least one second image and a first image included in at least two first images; The CPU 34 executes the second process based on a second image included in at least one second image and a first image included in at least two first images and different from the first image used in the first process.

[0230] Each aspect of the present invention may include the following modifications: The CPU 34 generates two or more sets of disparity information by executing a first process and a second process; The CPU 34 executes the first process based on a second image included in at least one second image and a first image included in at least two first images; The CPU 34 executes the second process based on at least two first images.

[0231] Each aspect of the present invention may include the following modifications. The CPU 34 acquires an image from the image sensor 22 during a readout period shorter than one frame period. The CPU 34 turns off the light source unit 32, which generates illumination light to be irradiated onto the subject, during the readout period. The CPU 34 turns on the light source unit 32 during at least a portion of a period excluding the readout period. The CPU 34 switches between a first imaging condition and a second imaging condition during the readout period.

[0232] Each aspect of the present invention may include the following modifications. The CPU 34 calculates the time difference between a reference time and the time at which the image sensor 22 generates an image included in the image group after the reference time. The CPU 34 sets the image group so that the time difference is equal to or less than a preset threshold. For example, the reference time is the time at which a measurement trigger occurs.

[0233] Each aspect of the present invention may include the following modifications. The CPU 34 sets a first provisional image group and a second provisional image group based on the amount of motion. Each of the first provisional image group and the second provisional image group includes three or more images out of four or more images. At least one image included in the first provisional image group is different from at least one image included in the second provisional image group. The CPU 34 calculates a first time difference between a reference time and a timing at which the image sensor 22 generates an image included in the first provisional image group after the reference time. The CPU 34 calculates a second time difference between the reference time and a timing at which the image sensor 22 generates an image included in the second provisional image group after the reference time. When the first time difference is smaller than the second time difference, the CPU 34 sets the first provisional image group as the image group. When the second time difference is smaller than the first time difference, the CPU 34 sets the second provisional image group as the image group. For example, the first provisional image group and the second provisional image group are two image sets from among the image sets SET1 to SET11.

[0234] Each aspect of the present invention may include the following modifications. The endoscope device 1 has a first objective optical system including a concave lens 23 a, a convex lens 24 a, and an imaging optical system 26, and a second objective optical system including a concave lens 23 b, a convex lens 24 b, and the imaging optical system 26. The second objective optical system has parallax with respect to the first objective optical system. The first imaging condition indicates that the image sensor 22 generates an image at a first timing based on an optical image of a subject formed through the first objective optical system. The second imaging condition indicates that the image sensor 22 generates an image at a second timing different from the first timing based on the optical image of a subject formed through the second objective optical system.

[0235] Each aspect of the present invention may include the following modifications. The CPU 34 calculates a first amount of motion between a first reference image (e.g., first image L(t)) included in two or more first images and a first image (e.g., first image L(t+2)) that is different from the first reference image and is included in the two or more first images. The CPU 34 calculates a second amount of motion between a second reference image (e.g., second image R(t+1)) included in two or more second images and a second image (e.g., second image R(t+3)) that is different from the second reference image and is included in the two or more second images. The CPU 34 sets an image group based on the first amount of motion and the second amount of motion.

[0236] Each aspect of the present invention may include the following modifications: The CPU 34 determines the linearity of the movement based on the amount of movement, and sets an image group based on the linearity.

[0237] Each aspect of the present invention may include the following modifications. The CPU 34 sets a first provisional image group and a second provisional image group. Each of the first provisional image group and the second provisional image group includes three or more images among four or more images. At least one image included in the first provisional image group and at least one image included in the second provisional image group are different from each other. The CPU 34 determines the linearity of the motion of each of the first provisional image group and the second provisional image group based on the amount of motion. If the linearity of the first provisional image group is higher than that of the second provisional image group, the CPU 34 sets the first provisional image group as an image group. If the linearity of the second provisional image group is higher than that of the first provisional image group, the CPU 34 sets the second provisional image group as an image group for motion compensation. For example, the first provisional image group and the second provisional image group are two image sets of image sets SET1 to SET11.

[0238] Each aspect of the present invention may include the following variations: The image group includes at least two first images of two or more first images and at least one second image of two or more second images.

[0239] Each aspect of the present invention may include the following modifications: The CPU 34 generates disparity information based on a first image included in an image group and a second image included in the image group. The CPU 34 generates motion compensation information by correcting the disparity information. The CPU 34 uses the first image or the second image used to generate the disparity information and the motion compensation information to generate, as 3D information, a 3D point cloud including two or more points on the subject.

[0240] Each aspect of the present invention may include the following modifications: The endoscope device 1 has an imaging device, which has an imaging element 22 and a CPU 34 .

[0241] In the first embodiment, the CPU 34 sets an image group based on the amount of motion in candidate image groups, and generates 3D information of the subject based on the images included in the image group and the motion compensation information. Since the probability of determining that measurement is not possible for all of the predetermined number of images is reduced, the endoscope device 1 does not need to repeatedly acquire the predetermined number of images. The user does not need to repeatedly input measurement instructions, and the examination time is shortened. Furthermore, since the 3D information is generated based on the motion compensation information, the accuracy of the 3D information is improved. Therefore, the endoscope device 1 can improve the accuracy of the 3D information of the subject without sacrificing usability.

[0242] The CPU 34 determines the linearity of the motion between images based on the amount of motion and selects a set of images with high linearity as candidate images for the image group. In situations where the motion in the images is not uniform within the camera's field of view, the CPU 34 can select an image group suitable for motion compensation. The CPU 34 also calculates a corrected parallax amount for each pixel by calculating the average of the parallax amount for each pixel in the first parallax map and the parallax amount for each pixel in the second parallax map. The CPU 34 generates a corrected parallax map including the corrected parallax amount. Even when the motion in the images is not uniform within the camera's field of view and the parallax amount for the same region differs significantly between the first and second parallax maps, the CPU 34 can generate motion compensation information that is less affected by differences in the amount of motion between regions in the images. The endoscope device 1 performs 3D reconstruction processing by using an image group suitable for motion compensation. This effectively suppresses increases in measurement errors and improves the accuracy of inspections.

[0243] The endoscope device 1 performs global exposure and switches the imaging conditions during the off period of the white light source 37. This shortens the time required to acquire a predetermined number of images. This reduces the effect of image motion on measurement errors, and the use of motion compensation information increases the effectiveness of suppressing increases in measurement errors.

[0244] The endoscope device 1 considers the time difference between the first timing at which a measurement instruction is input and the second timing at which candidate images for the image group are acquired, and selects an image set that minimizes this time difference. This reduces the likelihood of the composition intended by the user deviating from the composition of the images generated by the image sensor 22. This reduces the number of situations in which images are acquired with a composition unintended by the user, requiring the image to be acquired again, thereby suppressing a decrease in examination efficiency.

[0245] Second Embodiment A second embodiment of the present invention will be described. Fig. 14 shows the configuration of an endoscope device 1a according to the second embodiment. Explanation of the same parts as those shown in Fig. 2 will be omitted.

[0246] The main body 3 shown in Fig. 2 is changed to a main body 3a. In the main body 3a, the light source 32 shown in Fig. 2 is changed to a light source 32a. The light source 32a has a white light source 37 and a laser diode (LD) 38. In the main body 3a, the illumination control unit 33 shown in Fig. 2 is changed to an illumination control unit 33a. In the main body 3a, the switching control unit 36 ​​shown in Fig. 2 is changed to a switching control unit 36a.

[0247] The optical adapter 21 shown in Fig. 2 is changed to an optical adapter 21a. In the optical adapter 21a, the observation optical system 70 shown in Fig. 2 is changed to an observation optical system 70a. The observation optical system 70a has a concave lens 23 and a convex lens 24.

[0248] The illumination optical system 80 shown in FIG. 2 is changed to an illumination optical system 80a. The optical adapter 21a has a part of the illumination optical system 80a. The illumination optical system 80a has a condenser lens 81, a light guide 82, a rod lens 83, a diffusing lens 84, a fiber 85, and a stripe generating unit 89 (stripe pattern generator). The condenser lens 81 is disposed in the main body 3a. The light guide 82 and the fiber 85 are disposed in the main body 3a and the insertion unit 20. The rod lens 83, the diffusing lens 84, and the stripe generating unit 89 are disposed in the optical adapter 21a.

[0249] Fig. 15 shows the configuration of the stripe generating unit 89. The stripe generating unit 89 shown in Fig. 15 has a first optical path 890, a second optical path 891, a third optical path 892, and a phase shift unit 893 (phase shifter).

[0250] The following is an overview of the differences from Fig. 2. The endoscope device 1a has a white light source 37 (first light source) and an LD 38 (second light source) as light sources. The illumination light includes the first illumination light and the second illumination light. The white light source 37 generates white light as the first illumination light. The LD 38 generates the second illumination light. The stripe generating unit 89 imparts a spatial pattern including bright and dark areas to the second illumination light.

[0251] The image sensor 22 generates an image of the subject under one of mutually different first, second, third, and fourth imaging conditions. The image sensor 22 generates a first image of the subject by imaging the subject under the first imaging condition. The image sensor 22 generates a second image of the subject by imaging the subject under the second imaging condition. The image sensor 22 generates a third image of the subject by imaging the subject under the third imaging condition. The image sensor 22 generates a fourth image of the subject by imaging the subject under the fourth imaging condition.

[0252] The first imaging condition indicates a first stripe pattern of the second illumination light. The second imaging condition indicates a second stripe pattern of the second illumination light. The third imaging condition indicates a third stripe pattern of the second illumination light. The first stripe pattern, the second stripe pattern, and the third stripe pattern are different from each other. The fourth imaging condition indicates the first illumination light.

[0253] The phase shifter 893 functions as a switching unit. The phase shifter 893 switches between the first, second, and third imaging conditions by shifting the phase of the illumination light pattern. The phase of the pattern under the first imaging condition, the phase of the pattern under the second imaging condition, and the phase of the pattern under the third imaging condition are different from one another. The measurement processing unit 347 of the CPU 34 calculates the 3D coordinates of two or more points on the surface of the subject by using the phase shift method based on at least the first, second, and third images.

[0254] In the first, second, and third imaging conditions, the second illumination light is irradiated onto the subject and the irradiation of the first illumination light onto the subject is stopped, and in the fourth imaging condition, the first illumination light is irradiated onto the subject and the irradiation of the second illumination light onto the subject is stopped.

[0255] The imaging element 22 has red, green, and blue color filters. The imaging element 22 generates a color image as a fourth image. The color image includes two or more pixels. Each pixel has information on red brightness, green brightness, and blue brightness as a pixel value. The measurement processing unit 347 generates 3D shape data in which the 3D coordinates of two or more points are associated with the pixel values ​​of the two or more points. The visualization processing unit 343 generates graphic data that visualizes the 3D shape data.

[0256] The switching control unit 36a causes the phase shift unit 893 to set a first imaging condition in two or more first frame periods. The switching control unit 36a causes the phase shift unit 893 to set a second imaging condition in two or more second frame periods. Each of the two or more second frame periods is different from each of the two or more first frame periods. The switching control unit 36a causes the phase shift unit 893 to set a third imaging condition in two or more third frame periods. Each of the two or more third frame periods is different from each of the two or more first frame periods and different from each of the two or more second frame periods. The operations of the illumination control unit 33a and the switching control unit 36a are controlled by the main control unit 340 of the CPU 34.

[0257] Details of the differences from FIG. 2 will be described below. The LD 38 generates blue laser light as the second illumination light. The LD 38 may be disposed in the optical adapter 21a. The fiber 85 is a single-mode fiber. The fiber 85 is connected to the LD 38. The laser light emitted from the LD 38 is transmitted to the tip of the insertion section 20 via the fiber 85. The fiber 85 is connected to a stripe generating unit 89. The laser light (second illumination light) emitted from the fiber 85 is incident on the stripe generating unit 89.

[0258] The laser light is incident on a first end In of the stripe generating unit 89. A first optical path 890 is connected to the first end In. The laser light incident on the first end In is transmitted through the first optical path 890. The first optical path 890 branches into a second optical path 891 and a third optical path 892. The second optical path 891 and the third optical path 892 are connected to the first optical path 890. The laser light transmitted through the first optical path 890 is split into a first laser light and a second laser light. The first laser light is incident on the second optical path 891. The second laser light is incident on the third optical path 892. The first laser light incident on the second optical path 891 is transmitted through the second optical path 891. The second laser light incident on the third optical path 892 is transmitted through the third optical path 892.

[0259] The second optical path 891 is connected to the second end Out1. The third optical path 892 is connected to the third end Out2. The length of the second optical path 891 is equal to the length of the third optical path 892. The first laser light transmitted through the second optical path 891 is emitted from the second end Out1 and is irradiated onto the subject. The second laser light transmitted through the third optical path 892 is emitted from the third end Out2 and is irradiated onto the subject.

[0260] The second end Out1 and the third end Out2 are separated from each other by a predetermined distance d. The first laser beam emitted from the second end Out1 and the second laser beam emitted from the third end Out2 interfere with each other. A stripe pattern is formed by the interference of the laser beams. This pattern can be approximately considered to be a stripe pattern emitted from a single light source. In the stripe pattern, elongated bright portions and elongated dark portions are arranged alternately.

[0261] The period of the stripe pattern is based on the principle of Young's interference fringes. This period is based on the wavelength of the laser light guided to the first end In and the width d. For example, when the width d is fixed and the wavelength of the laser light guided to the first end In becomes shorter, the period of the stripe pattern becomes shorter. When the width d is fixed and the wavelength of the laser light guided to the first end In becomes longer, the period of the stripe pattern becomes longer. When the wavelength band of the laser light guided to the first end In is fixed and the width d becomes wider, i.e., the second end Out1 and the third end Out2 become farther apart, the period of the stripe pattern becomes shorter. When the wavelength band of the laser light guided to the first end In is fixed and the width d becomes narrower, i.e., the second end Out1 and the third end Out2 become closer to each other, the period of the stripe pattern becomes longer.

[0262] A portion of the third optical path 892 is disposed inside the phase shift unit 893. The switching control unit 36a disposed in the main body unit 3a supplies a current to the phase shift unit 893. The phase shift unit 893 generates heat based on the current supplied from the switching control unit 36a. The generated heat changes the refractive index of the third optical path 892 and also changes the optical path length of the third optical path 892. The phase shift unit 893 shifts the phase of the second laser light guided to the third optical path 892 based on a change in temperature.

[0263] The phases of the first laser light and the second laser light are different from each other. The first laser light passes through a second optical path 891 and is output from a second end Out1. The second laser light passes through a third optical path 892, which has a temperature different from that of the second optical path 891, and is output from a third end Out2. The phases of the stripes in the stripe pattern are changed by interference between the first laser light and the second laser light in a phase-shifted state. In other words, the phases of the stripes are shifted. The method by which the phase shift unit 893 shifts the phase of the laser light is not limited to a method using a change in temperature.

[0264] The observation optical system 70a is a monocular optical system. The observation optical system 70a captures light reflected from the surface of an object illuminated with white light or patterned light. The patterned light has a spatial pattern. The concave lens 23 and the convex lens 24 form an optical image based on the light from the object in the imaging area S1 of the image sensor 22.

[0265] The image sensor 22 sequentially reads out pixel signals row by row from at least some of two or more cells during a first period. The illumination control unit 33a controls the white light source 37 and the LD 38. The illumination control unit 33a causes the white light source 37 and the LD 38 to generate illumination light during a second period. The illumination control unit 33a causes the white light source 37 and the LD 38 to stop generating illumination light during a third period. The illumination control unit 33a functions as a switching unit and a switching control unit. The illumination control unit 33a starts switching the imaging conditions during the third period and completes switching the imaging conditions during the third period. The switching control unit 36a causes the phase shift unit 893 to start switching the imaging conditions during the third period and completes switching the imaging conditions during the third period.

[0266] The illumination control unit 33a turns on the LD 38 and turns off the white light source 37 under the first imaging condition. Therefore, the illumination optical system 80a irradiates the subject with pattern light under the first imaging condition. The illumination optical system 80a does not irradiate the subject with white light under the first imaging condition. The switching control unit 36a causes the phase shift unit 893 to set the phase of the pattern light to the first phase. Under the first imaging condition, the observation optical system 70a captures light reflected from the surface of the subject illuminated by the pattern light.

[0267] The illumination control unit 33a turns on the LD 38 and turns off the white light source 37 under the second imaging condition. Therefore, the illumination optical system 80a irradiates the subject with pattern light under the second imaging condition. The illumination optical system 80a does not irradiate the subject with white light under the second imaging condition. The switching control unit 36a causes the phase shift unit 893 to set the phase of the pattern light to a second phase. The second phase is different from the first phase. Under the second imaging condition, the observation optical system 70a captures light reflected from the surface of the subject illuminated by the pattern light.

[0268] The illumination control unit 33a turns on the LD 38 and turns off the white light source 37 under the third imaging condition. Therefore, the illumination optical system 80a irradiates the subject with patterned light under the third imaging condition. The illumination optical system 80a does not irradiate the subject with white light under the third imaging condition. The switching control unit 36a causes the phase shift unit 893 to set the phase of the patterned light to a third phase. The third phase is different from the first phase and different from the second phase. Under the third imaging condition, the observation optical system 70a captures light reflected from the surface of the subject illuminated by the patterned light.

[0269] Under the fourth imaging condition, the illumination control unit 33a turns on the white light source 37 and turns off the LD 38. Therefore, under the fourth imaging condition, the illumination optical system 80a irradiates the subject with white light. Under the fourth imaging condition, the illumination optical system 80a does not irradiate the subject with patterned light. Under the fourth imaging condition, the observation optical system 70a captures light reflected from the surface of the subject illuminated with white light.

[0270] The illumination control unit 33a causes the light source unit 32a to set the fourth imaging condition. As described above, the switching control unit 36a causes the phase shift unit 893 to set the first imaging condition, the second imaging condition, and the third imaging condition. The illumination control unit 33a and the switching control unit 36a successively switch among the first imaging condition, the second imaging condition, the third imaging condition, and the fourth imaging condition.

[0271] 16 shows the relationship between the operation of the image sensor 22 in measurement mode, the illumination state, and the image capturing condition state. The operation of the image sensor 22 will be described with reference to Fig. 16. Explanation of the same parts as those shown in Fig. 4 will be omitted.

[0272] Timing chart TC11 shows the operation of the image sensor 22. The cells in the first to eighth rows are reset in sequence, and the exposure periods for the cells in the first to eighth rows in frame period i are started in sequence.

[0273] When the exposure period of the cells in the first row in the frame period i starts, the white light source 37 and the LD 38 are turned off. When the exposure period of the cells in the first row in the frame period i starts, the imaging conditions are not set.

[0274] When the exposure period of the cells in the eighth row in frame period i starts, the white light source 37 starts to light up. The illumination control unit 33a outputs a control signal for lighting to the white light source 37, thereby lighting up the white light source 37. As a result, the illumination control unit 33a sets the imaging condition to the fourth imaging condition. The white light source 37 starts to light up based on the control signal from the illumination control unit 33a.

[0275] When the readout period for the cells in the first row in frame period i starts, the white light source 37 stops being lit. The illumination control unit 33a outputs a control signal for turning off the white light source 37 to the white light source 37, thereby turning off the white light source 37. The white light source 37 is turned off based on the control signal from the illumination control unit 33a. As a result, the illumination control unit 33a starts switching from the fourth imaging condition to the first imaging condition.

[0276] When the readout period for the cells in the eighth row in frame period i ends, the LD 38 starts to light up. The illumination control unit 33a outputs a control signal for lighting to the LD 38, thereby lighting up the LD 38. The LD 38 starts to light up based on the control signal from the illumination control unit 33a. The switching control unit 36a outputs a current to the phase shift unit 893, thereby setting the phase of the pattern light to the first phase. The phase shift unit 893 sets the phase of the pattern light to the first phase based on the current from the switching control unit 36a. This allows the illumination control unit 33a and the switching control unit 36a to complete switching from the fourth imaging condition to the first imaging condition.

[0277] When the readout period for the cells in the first row in frame period (i+1) starts, the LD 38 stops lighting. The illumination control unit 33a outputs a control signal for turning off the LD 38 to turn off the LD 38. The LD 38 turns off based on the control signal from the illumination control unit 33a. This causes the illumination control unit 33a to start switching from the first imaging condition to the second imaging condition.

[0278] When the readout period for the cells in the 8th row in frame period (i+1) ends, the LD 38 starts to light up. The illumination control unit 33a outputs a control signal for lighting to the LD 38, thereby lighting up the LD 38. The LD 38 starts to light up based on the control signal from the illumination control unit 33a. The switching control unit 36a outputs a current to the phase shift unit 893, thereby setting the phase of the pattern light to the second phase. The phase shift unit 893 sets the phase of the pattern light to the second phase based on the current from the switching control unit 36a. This allows the illumination control unit 33a and the switching control unit 36a to complete switching from the first imaging condition to the second imaging condition.

[0279] When the readout period for the cells in the first row in frame period (i+2) starts, the LD 38 stops lighting. The illumination control unit 33a outputs a control signal for turning off the LD 38 to turn off the LD 38. The LD 38 turns off based on the control signal from the illumination control unit 33a. This causes the illumination control unit 33a to start switching from the second imaging condition to the third imaging condition.

[0280] When the readout period for the cells in the 8th row in the frame period (i+2) ends, the LD 38 starts to light up. The illumination control unit 33a outputs a control signal for lighting to the LD 38, thereby lighting up the LD 38. The LD 38 starts to light up based on the control signal from the illumination control unit 33a. The switching control unit 36a outputs a current to the phase shift unit 893, thereby setting the phase of the pattern light to the third phase. The phase shift unit 893 sets the phase of the pattern light to the third phase based on the current from the switching control unit 36a. This allows the illumination control unit 33a and the switching control unit 36a to complete switching from the second imaging condition to the third imaging condition.

[0281] When the readout period for the cells in the first row in frame period (i+3) starts, the LD 38 stops lighting. The illumination control unit 33a outputs a control signal for turning off the LD 38 to turn off the LD 38. The LD 38 turns off based on the control signal from the illumination control unit 33a. This causes the illumination control unit 33a to start switching from the third imaging condition to the fourth imaging condition.

[0282] When the readout period for the cells in the eighth row in the frame period (i+3) ends, the white light source 37 starts to light up. The illumination control unit 33a outputs a control signal for lighting to the white light source 37, thereby lighting up the white light source 37. The white light source 37 starts to light up based on the control signal from the illumination control unit 33a. This completes the switching by the illumination control unit 33a from the third imaging condition to the fourth imaging condition.

[0283] The white light source 37 is turned on during the simultaneous exposure period in frame period i. The LD 38 is turned on during the simultaneous exposure periods in each of frame periods (i+1), (i+2), and (i+3). The white light source 37 and the LD 38 are repeatedly turned on and off.

[0284] The illumination control unit 33a and the switching control unit 36a repeatedly switch from the fourth imaging condition to the first imaging condition, from the first imaging condition to the second imaging condition, from the second imaging condition to the third imaging condition, and from the third imaging condition to the fourth imaging condition. The illumination control unit 33a and the switching control unit 36a switch the imaging conditions for each frame period.

[0285] The image sensor 22 sequentially reads out pixel signals from at least some of two or more cells, row by row, during a first period T1. In the timing chart TC11 shown in Fig. 16, the first period T1 is the period from the start of readout of pixel signals from the cells in the first row to the completion of readout of pixel signals from the cells in the eighth row.

[0286] The white light source 37 or LD 38 is turned on simultaneously with the completion of the readout of pixel signals in one frame period. That is, the white light source 37 or LD 38 is turned on simultaneously with the completion of the first period T1. The white light source 37 or LD 38 is turned off simultaneously with the start of the readout of pixel signals in one frame period. That is, the white light source 37 or LD 38 is turned off simultaneously with the start of the first period T1.

[0287] The illumination control unit 33a causes the white light source 37 or the LD 38 to generate illumination light during the second period T2. The white light source 37 or the LD 38 continues to be lit during the second period T2. The second period T2 is the entire period excluding the first period T1. In the timing chart TC11 shown in FIG. 16 , the sum of the lengths of the first period T1 and the second period T2 is the same as the length of the frame period.

[0288] The illumination control unit 33a causes the white light source 37 or the LD 38 to stop generating illumination light during the third period T3. The white light source 37 or the LD 38 remains extinguished during the third period T3. The third period T3 is the entire period excluding the second period T2. In the timing chart TC11 shown in FIG. 16, the third period T3 is the same as the first period T1.

[0289] The illumination control unit 33a and the phase shift unit 893 start switching the imaging conditions in the third period T3 and complete the switching of the imaging conditions in the third period T3. The illumination control unit 33a and the phase shift unit 893 complete the switching of the imaging conditions before the start of the next second period. In other words, the illumination control unit 33a and the phase shift unit 893 complete the switching of the imaging conditions in the same third period as the third period in which the switching of the imaging conditions started.

[0290] When the third period T3 starts, the illumination control unit 33a starts switching the imaging conditions by turning off the white light source 37 or the LD 38. When the third period T3 ends, the illumination control unit 33a completes switching the imaging conditions by turning on the white light source 37 or the LD 38. When the third period T3 ends, the phase shift unit 893 completes switching the imaging conditions by changing the phase of the pattern light. In the timing chart TC11 shown in FIG. 16 , the third period T3 is the same as the first period T1, so the illumination control unit 33a and the phase shift unit 893 start switching the imaging conditions in the first period T1 and complete switching the imaging conditions in the first period T1.

[0291] After the readout period for the cells in the first row in each frame period has started and before the readout period for the cells in the eighth row in each frame period has completed, the phase shift section 893 may switch the phase of the pattern light.

[0292] Fig. 17 shows the procedure of the operation of the endoscope device 1a. The operation of the endoscope device 1a will be described with reference to Fig. 17. Description of the same processes as those shown in Fig. 5 will be omitted.

[0293] After step S100, the illumination control unit 33a turns on the white light source 37 to set the imaging condition to observation illumination (fourth imaging condition) (step S105a).

[0294] After step S105a, the image sensor 22 generates one frame of an image and outputs the generated image. Because the imaging condition is observation illumination, the image sensor 22 outputs a fourth image. When the third period starts, the illumination control unit 33a turns off the white light source 37 (step S110a).

[0295] After step S110a, the display processing unit 341 displays the fourth image generated in step S110a on the display unit 5. The display unit 5 displays the fourth image (step S115a). After step S115a, step S120 is executed.

[0296] When the main control unit 340 determines in step S120 that measurement is to be performed, the CPU 34 executes an image acquisition process to acquire candidate images to be used in the measurement process (step S125a). After step S125a, step S130 is executed.

[0297] 18 shows the procedure of the operation of the endoscope device 1a in the image acquisition process (step S125a). The operation of the endoscope device 1a in the image acquisition process will be described with reference to Fig. 18. Description of the same processes as those shown in Fig. 6 will be omitted.

[0298] When the main control unit 340 determines in step S120 that measurement is to be performed, the image sensor 22 generates one frame of image and outputs the generated image. Since the imaging condition is observation illumination, the image sensor 22 outputs a fourth image. When the third period starts, the illumination control unit 33a turns off the white light source 37 (step S1260).

[0299] After step S1260, the display processing unit 341 displays the fourth image generated in step S1260 on the display unit 5. The display unit 5 displays the fourth image (step S1261).

[0300] After step S1261, the illumination control unit 33a turns on the LD 38, and the phase shift unit 893 sets the first phase, which causes the illumination control unit 33a and the phase shift unit 893 to set the imaging condition to stripes having the first phase (first imaging condition) (step S1262).

[0301] After step S1262, the image sensor 22 generates one frame of image and outputs the generated image. Since the imaging condition is stripes having the first phase, the image sensor 22 outputs the first image. When the third period starts, the illumination control unit 33a turns off the LD 38 (step S1263).

[0302] After step S1263, the illumination control unit 33a turns on the LD 38, and the phase shift unit 893 sets a second phase. The second phase is a phase obtained by shifting the first phase by 2π / 3. As a result, the illumination control unit 33a and the phase shift unit 893 set the imaging condition to stripes having the second phase (second imaging condition) (step S1264).

[0303] After step S1264, the image sensor 22 generates one frame of image and outputs the generated image. Since the imaging condition is stripes having the second phase, the image sensor 22 outputs the second image. When the third period starts, the illumination control unit 33a turns off the LD 38 (step S1265).

[0304] After step S1265, the illumination control unit 33a turns on the LD 38, and the phase shift unit 893 sets a third phase. The third phase is a phase obtained by shifting the second phase by 2π / 3. As a result, the illumination control unit 33a and the phase shift unit 893 set the imaging condition to stripes having the third phase (third imaging condition) (step S1266).

[0305] After step S1266, the image sensor 22 generates one frame of image and outputs the generated image. Since the imaging condition is stripes having a third phase, the image sensor 22 outputs a third image. When the third period starts, the illumination control unit 33a turns off the LD 38 (step S1267). After step S1267, step S1254 is executed.

[0306] In step S1254, the main control unit 340 determines whether or not the acquisition of a predetermined number of images has been completed. In the example described below, the motion detection unit 345 uses six or more images including two or more first images, two or more second images, and two or more third images, and also uses two or more fourth images to determine whether or not measurement is possible. For example, the predetermined number is 20 images.

[0307] If the main control unit 340 determines in step S1254 that the acquisition of the predetermined number of images has not been completed, step S1268 is executed. In step S1268, the same processing as in step S105a is executed. After step S1268, step S1260 is executed.

[0308] In the image selection process (step S130), the following process is performed. In the image acquisition process, 20 images are acquired. The 20 images include five first images, five second images, five third images, and five fourth images. In step S1300, the motion detection unit 345 performs a differentiation process on each of the first, second, third, and fourth images. This removes the influence of the presence or absence of a stripe pattern and the influence of the phase difference of the stripe pattern from each image. In the process of calculating the amount of motion, each image that has undergone the differentiation process is used.

[0309] In step S1300 (FIG. 7), the motion detection unit 345 selects a reference image and calculates the amount of motion between each image and the reference image. For example, the reference image is the fourth image that was generated first. The reference image may also be the fourth image that was generated second or third, etc. The motion detection unit 345 calculates the amount of motion between the reference image and each of the five first images, five second images, five third images, and four fourth images.

[0310] Figure 19 shows an example of images acquired in the image acquisition process. Time progresses from left to right in Figure 19. The main control unit 340 acquires first images A1 to A3, second images B1 to B3, third images C1 to C3, and fourth images W1 to W3. The motion detection unit 345 uses the first acquired fourth image W1 as a reference image and calculates the amount of motion between each image and the fourth image W1.

[0311] For example, the motion detection unit 345 calculates a first motion amount indicating the amount of motion of the first image A1 relative to the fourth image W1. Similarly, the motion detection unit 345 calculates a first motion amount indicating the amount of motion of each of the first images A2 and A3 relative to the fourth image W1. The motion detection unit 345 calculates a second motion amount indicating the amount of motion of the second image B1 relative to the fourth image W1. Similarly, the motion detection unit 345 calculates a second motion amount indicating the amount of motion of each of the second images B2 and B3 relative to the fourth image W1. The motion detection unit 345 calculates a third motion amount indicating the amount of motion of the third image C1 relative to the fourth image W1. Similarly, the motion detection unit 345 calculates a third motion amount indicating the amount of motion of the third image C2 and C3 relative to the fourth image W1.

[0312] The motion detection unit 345 calculates a fourth motion amount that indicates the amount of motion of the fourth image W2 relative to the fourth image W1. Similarly, the motion detection unit 345 calculates a fourth motion amount that indicates the amount of motion of the fourth image W3 relative to the fourth image W1.

[0313] In step S1301, the image selection unit 346 compares the first motion amount of each first image, the second motion amount of each second image, and the third motion amount of each third image. In step S1301, the image selection unit 346 determines whether measurement is possible based on the comparison results of the motion amounts. When the first motion amount, the second motion amount, and the third motion amount are close to each other, the image selection unit 346 extracts the first image used to calculate the first motion amount, the second image used to calculate the second motion amount, and the third image used to calculate the third motion amount. The image selection unit 346 determines that measurement is possible using the extracted three images.

[0314] For example, the image selection unit 346 calculates the difference between the first motion amount and the second motion amount, the difference between the second motion amount and the third motion amount, and the difference between the third motion amount and the first motion amount. When the absolute values ​​of all of the three differences are equal to or less than a preset threshold, the image selection unit 346 determines that measurement using the above three images is possible. When at least one of the three differences is greater than the threshold, the image selection unit 346 determines that measurement using the above three images is not possible.

[0315] The three images determined to be measurable are not necessarily images generated consecutively by the image sensor 22. For example, the three images may include a first image and a second image generated in a frame period that is five or more frame periods after the frame period in which the first image was generated.

[0316] 20 shows the distribution of the motion amounts calculated in step S1300. The distribution includes the motion amount in the X direction for each image and the motion amount in the Y direction for each image. The first motion amount, second motion amount, and third motion amount enclosed by the circle CR1 are close to each other. Therefore, the image selection unit 346 determines that measurement is possible using the images used to calculate each of these motion amounts. For example, the image selection unit 346 determines that measurement is possible using the first image A3, second image B2, and third image C3 shown in FIG. 19.

[0317] The image selection unit 346 selects a fourth image used to calculate a fourth amount of motion that is close to the amounts of motion of each of the three images determined to be measurable. For example, the fourth amount of motion MA4 shown in FIG. 20 is close to the first amount of motion, the second amount of motion, and the third amount of motion enclosed by the circle CR1. Therefore, the image selection unit 346 selects the fourth image used to calculate the fourth amount of motion MA4. For example, the image selection unit 346 selects the fourth image W3 shown in FIG. 19. The fourth image selected by the image selection unit 346 is used as a texture image for generating mesh polygons.

[0318] In step S1302, the image selection unit 346 selects one or more image sets including three images determined to be measurable as candidates for the image group. The image selection unit 346 calculates the time difference between the first timing and the second timing. The first timing is a reference timing, which is the timing at which the user inputs a measurement instruction by operating the operation unit 4. The second timing is the average of the imaging timings of the three images included in the image set. The second timing may be the imaging timing of the first image generated among the three images. The second timing may be the imaging timing of the last image generated among the three images.

[0319] The image selection unit 346 compares the time difference for each image set selected as a candidate for the image group with a preset threshold value. If the time difference is smaller than the threshold value, the image selection unit 346 selects the image set used to calculate the time difference as the image group.

[0320] When two or more time differences are equal to or less than the threshold, the image selection unit 346 selects, as the image group, the image set used to calculate the smallest amount of motion. When all time differences are greater than the threshold, the image selection unit 346 selects, as the image group, the image set used to calculate the smallest amount of motion. The image selection unit 346 may select, as the image group, the image set used to calculate the smallest time difference without comparing the time differences with the threshold.

[0321] The measurement processing unit 347 performs measurement processing (step S140) using an image group including three or more images selected by the image selection unit 346. In the measurement processing, the following processing is performed.

[0322] In step S1400, the measurement processing unit 347 corrects optical distortion of each image included in the image group. Specifically, the measurement processing unit 347 generates a first corrected image by correcting optical distortion of a first image. The measurement processing unit 347 generates a second corrected image by correcting optical distortion of a second image. The measurement processing unit 347 generates a third corrected image by correcting optical distortion of a third image. The measurement processing unit 347 generates a fourth corrected image by correcting optical distortion of a fourth image. In the measurement process, the measurement processing unit 347 calculates the 3D coordinates of each point on the surface of the subject by a phase shift method using the first corrected image, the second corrected image, and the third corrected image.

[0323] At this time, in step S1400, the measurement processing unit 347 calculates a phase value of each pixel using the luminance values ​​of the first corrected image, the second corrected image, and the third corrected image. In step S1400, the measurement processing unit 347 uses the phase values ​​to generate a phase map on which phase unwrapping processing has been performed. In step S1401, the measurement processing unit 347 performs 3D reconstruction processing based on the images included in the image group and the phase map.

[0324] In step S1402, the measurement processing unit 347 performs the following processing. Using a fourth differential image generated by differentiating the fourth corrected image and a first differential image generated by differentiating the first corrected image, the measurement processing unit 347 performs template matching processing using the fourth differential image as a reference image. As a result, the measurement processing unit 347 searches for corresponding points in the first corrected image that correspond to pixels in the fourth corrected image. The measurement processing unit 347 performs this processing for all pixels in the fourth differential image and generates a disparity map as motion compensation information. The disparity map indicates the distribution of the amount of disparity between each pixel in the fourth corrected image and its corresponding point in the first corrected image. The measurement processing unit 347 shifts each pixel in the fourth corrected image within the coordinate system of the fourth corrected image by the amount of disparity indicated by the motion compensation information. For example, the measurement processing unit 347 generates motion compensation information by calculating a disparity map between a fourth corrected image, which is an image generated by correcting the optical distortion of the fourth image W3 shown in FIG. 19 , and a first corrected image, which is an image generated by correcting the optical distortion of the first image A3. The measurement processing unit 347 shifts each pixel of the fourth corrected image by the amount of disparity indicated by the motion compensation information. The measurement processing unit 347 generates color 3D shape data, which is 3D information, by correlating color information of each pixel of the fourth corrected image having the shifted pixels with the 3D coordinates of each point.

[0325] A pixel common to the fourth corrected image having the shifted pixel, the first corrected image, the second corrected image, and the third corrected image is associated with a 3D coordinate, and color information of the common pixel and the 3D coordinate are associated with each other in the color 3D shape data.

[0326] The measurement processing unit 347 converts the 3D point cloud calculated in step S1401 into two or more meshes. The measurement processing unit 347 generates a texture image based on the fourth image having the pixels shifted as described above. The measurement processing unit 347 generates mesh polygons, which are 3D information, by combining the two or more meshes and the texture images. The color 3D shape data is composed of mesh polygons.

[0327] In the above example, the phase shifter 893 sets three phases. The amount of phase change when switching imaging conditions is 2π / 3. The phase shifter 893 may set four or more phases. When four or more phases are set, the amount of phase change when switching imaging conditions is π / 2 or less. When four or more phases are set, one or more images are acquired in addition to the three images (first image, second image, and third image) corresponding to the three phases.

[0328] The white light source 37, the condenser lens 81, the light guide 82, the rod lens 83, and the diffusion lens 84 are not essential. The illumination control unit 33 a and the phase shift unit 893 may switch the imaging conditions among only the first imaging condition, the second imaging condition, and the third imaging condition.

[0329] The means for generating a stripe pattern based on the second illumination light and the means for shifting the phase of the stripe pattern may be based on methods other than the interference of laser light. For example, the fourth embodiment may be applied to a method in which light emitted by an LED array is projected and the lighting pattern of the LED array is switched to shift the phase of the stripe pattern.

[0330] The second illumination light generated by the LD 38 may be laser light having a wavelength other than blue. For example, the wavelength of the second illumination light may be red, green, or infrared. Furthermore, irradiation of the first illumination light does not need to be stopped under at least one of the first imaging condition, the second imaging condition, and the third imaging condition.

[0331] Each aspect of the present invention may include the following modifications. The image sensor 22 further generates images of the subject under a third imaging condition different from both the first imaging condition and the second imaging condition. The CPU 34 continuously switches between the first imaging condition, the second imaging condition, and the third imaging condition. The CPU 34 acquires six or more images from the image sensor 22, including two or more first images generated under the first imaging condition, two or more second images generated under the second imaging condition, and two or more third images generated under the third imaging condition. The CPU 34 calculates the amount of motion between an image included in the six or more images and the reference image. The CPU 34 sets an image group including three or more images of the six or more images based on the amount of motion. The image group includes at least one first image of the two or more first images, at least one second image of the two or more second images, and at least one third image of the two or more third images. The CPU 34 generates motion compensation information based on at least two images generated under different imaging conditions from the at least one first image, the at least one second image, and the at least one third image.

[0332] Each aspect of the present invention may include the following modifications: The image sensor 22 generates a reference image by generating an image of the subject under an imaging condition different from both the first imaging condition and the second imaging condition, and the reference image is not included in the six or more images.

[0333] Each aspect of the present invention may include the following modifications. The endoscope device 1a has a stripe generating unit 89 (stripe pattern generator). The stripe generating unit 89 generates light having a stripe pattern to be projected onto a subject. The first imaging condition indicates a first stripe pattern that the light has. The second imaging condition indicates a second stripe pattern that the light has. The second stripe pattern is different from the first stripe pattern. The second stripe pattern has a phase that is the inverse of the phase of the first stripe pattern.

[0334] Each aspect of the present invention may include the following modifications. The image sensor 22 further generates an image of the subject under a third imaging condition different from both the first and second imaging conditions. The third imaging condition indicates a third stripe pattern of light projected onto the subject. The third stripe pattern is different from both the first and second stripe patterns. The image sensor 22 further generates an image of the subject under a fourth imaging condition different from both the first, second, and third imaging conditions. The CPU 34 successively switches between the first, second, third, and fourth imaging conditions. The CPU 34 acquires six or more images from the image sensor 22, including two or more first images, two or more second images, and two or more third images generated under the third imaging condition. The CPU 34 calculates the amount of motion between an image included in the six or more images and the reference image. The CPU 34 sets an image group including three or more images of the six or more images based on the amount of motion. The image group includes at least one first image of the two or more first images, at least one second image of the two or more second images, and at least one third image of the two or more third images. The CPU 34 generates motion compensation information based on at least one image of the at least one first image, the at least one second image, and the at least one third image, and a fourth image generated under a fourth imaging condition.

[0335] Each aspect of the present invention may include the following modifications: The CPU 34 calculates a first amount of motion between a first image included in two or more first images and a reference image; The CPU 34 calculates a second amount of motion between a second image included in two or more second images and the reference image; The CPU 34 calculates a third amount of motion between a third image included in two or more third images and the reference image; The CPU 34 sets an image group based on the first amount of motion, the second amount of motion, and the third amount of motion.

[0336] Each aspect of the present invention may include the following variations: the CPU 34 generates a 3D point cloud including two or more points on the object, the CPU 34 shifts pixels of the fourth image based on the motion compensation information, and the CPU 34 generates 3D information by correlating the colors of the shifted pixels with points included in the two or more points on the object.

[0337] Each aspect of the present invention may include the following modifications: The CPU 34 generates a texture image based on the fourth image having the pixels shifted as described above; The CPU 34 converts the 3D point cloud into two or more meshes; The CPU 34 generates a mesh polygon as 3D information by combining the two or more meshes and the texture image.

[0338] In the second embodiment, the CPU 34 sets an image group based on the amount of motion in the candidate image groups, and generates 3D information of the subject based on the images included in the image group and the motion compensation information. Therefore, the endoscope device 1 a can improve the accuracy of the 3D information of the subject without sacrificing usability, as in the first embodiment.

[0339] When 3D shape data composed of mesh polygons is displayed and dimension measurement processing is performed based on the 3D coordinates of a point specified on the 3D shape data, users often use the texture pattern as a guide to adjust the position of the point. In the above example, the texture image is generated based on the fourth image with shifted pixels, which reduces the misalignment of the texture relative to the 3D point cloud. This reduces the increase in measurement error.

[0340] (Modification of the Second Embodiment) A modification of the second embodiment of the present invention will be described below. In the modification of the second embodiment, an endoscope device 1a shown in FIG.

[0341] The image acquisition process shown in Fig. 18 is changed to the image acquisition process shown in Fig. 21. Fig. 21 shows the procedure of the operation of the endoscope device 1a in the image acquisition process. The operation of the endoscope device 1a in the image acquisition process will be described with reference to Fig. 21. Description of the same process as that shown in Fig. 18 will be omitted.

[0342] 18 , when the main control unit 340 determines in step S1254 that acquisition of a predetermined number of images has not been completed, step S1268 is executed. After the imaging conditions are set to the observation illumination in step S1268, the image sensor 22 outputs a fourth image in step S1260. Steps S1268 and S1260 are repeatedly executed until acquisition of a predetermined number of images is completed. Therefore, the image sensor 22 generates the fourth image two or more times.

[0343] On the other hand, in the image acquisition process shown in Fig. 21, when the main control unit 340 determines in step S1254 that acquisition of the predetermined number of images has not been completed, step S1268 is not executed and step S1263 is executed. In the image acquisition process shown in Fig. 21, step S1260 is executed only once, and the image sensor 22 generates the fourth image only once.

[0344] The image selection process shown in Figure 17 is modified as follows: 16 images are acquired in the image acquisition process, including five first images, five second images, five third images, and one fourth image.

[0345] In the image selection process shown in Fig. 17, the motion detection unit 345 selects a fourth image as a reference image and calculates a fourth amount of motion between the reference image and another fourth image different from the reference image. However, in the image acquisition process shown in Fig. 21, the motion detection unit 345 does not calculate the fourth amount of motion. Except for this, the image selection process in the modified example of the second embodiment is the same as the image selection process shown in Fig. 17.

[0346] The measurement process (step S140) is modified as follows. In step S1402, the measurement processing unit 347 performs the following process. The measurement processing unit 347 calculates the average of the first motion amount, the second motion amount, and the third motion amount. The measurement processing unit 347 shifts each pixel of the fourth image selected as the reference image by the average of the first motion amount, the second motion amount, and the third motion amount within the coordinate system of the fourth image. The measurement processing unit 347 uses the fourth image with the shifted pixels to generate color 3D shape data and a texture image, similar to the measurement process in the second embodiment. Except for this, the measurement process in the modified second embodiment is the same as the measurement process in the second embodiment.

[0347] In the modified example of the second embodiment, the number of fourth images acquired in the image acquisition process is reduced. Since the processing time of the image acquisition process is reduced, the influence of movement in the image on measurement errors is reduced.

[0348] Third Embodiment A third embodiment of the present invention will be described below. In the third embodiment, an endoscope device 1 shown in FIG.

[0349] In the third embodiment, the first optical system (concave lens 23 a, convex lens 24 a, and imaging optical system 26) and the second optical system (concave lens 23 b, convex lens 24 b, and imaging optical system 26) are decentered optical systems. The asymmetry of the distortion distribution is approximately symmetric between the first optical system and the second optical system. For example, in the first optical system, the distortion on the right side of the field of view is stronger than the distortion on the left side of the field of view, and in the second optical system, the distortion on the left side of the field of view is stronger than the distortion on the right side of the field of view.

[0350] The processing shown in Fig. 5 is changed to the processing shown in Fig. 22. Fig. 22 shows the procedure of the operation of the endoscope device 1. The operation of the endoscope device 1 will be described with reference to Fig. 22. Description of the same processing as the processing shown in Fig. 5 will be omitted.

[0351] When the main control unit 340 determines in step S135 that the image selection has been successful, the CPU 34 executes a switching determination process to determine whether or not the switching of the imaging optical path has been successful (step S145). After step S145, step S140 is executed.

[0352] The CPU 34 executes the following process in the switching determination process (step S145). The motion detection unit 345 uses the image group set in the image selection process to calculate a first motion amount, a second motion amount, and a third motion amount. If the image group includes three first images and two second images, the motion detection unit 345 uses the three images excluding the first image generated first and the last first image generated to calculate each motion amount. If the image group includes three second images and two first images, the motion detection unit 345 uses the three images excluding the second image generated first and the last second image generated to calculate each motion amount.

[0353] For example, the motion detection unit 345 uses two second images and one first image to calculate a first motion amount MA1, a second motion amount MA2, and a third motion amount MA3 shown in Fig. 23. Fig. 23 shows an example of a combination of images used to calculate the motion amount of an image. The image sensor 22 generates a second image R(t-1), a first image L(t), and a second image R(t+1) in this order.

[0354] Specifically, the motion detection unit 345 calculates a first motion amount MA1 that indicates the amount of motion between the second image R(t-1) and the first image L(t). The motion detection unit 345 calculates a second motion amount MA2 that indicates the amount of motion between the second image R(t-1) and the second image R(t+1). The motion detection unit 345 calculates a third motion amount MA3 that indicates the amount of motion between the first image L(t) and the second image R(t+1).

[0355] When the motion detection unit 345 uses two first images and one second image, the motion detection unit 345 calculates a first motion amount indicating the amount of motion between the second image and one of the two first images. The motion detection unit 345 further calculates a second motion amount indicating the amount of motion between the two first images. The motion detection unit 345 further calculates a third motion amount indicating the amount of motion between the second image and the other of the two first images.

[0356] The motion detection unit 345 compares the first motion amount, the second motion amount, and the third motion amount with each other. At this time, each motion amount is the average of the magnitudes of the motion vectors of two or more regions. The motion detection unit 345 may recalculate the first motion amount, the second motion amount, and the third motion amount by performing a process similar to the motion detection process (step S1300) again.

[0357] The motion detection unit 345 compares the first motion amount, the second motion amount, and the third motion amount and determines whether the result satisfies the following switching condition: Switching condition: Ma1≈Ma3>Ma2×CoffMa

[0358] Ma1 in the switching conditions indicates a first amount of motion, Ma2 in the switching conditions indicates a second amount of motion, and Ma3 in the switching conditions indicates a third amount of motion. The switching conditions indicate that the first amount of motion is approximately the same as the third amount of motion, and that the third amount of motion is greater than the second amount of motion. The motion detection unit 345 may calculate only one of the first amount of motion and the third amount of motion. CoffMa in the switching conditions is a coefficient greater than 1.0, and is set to 1.5, for example. Setting this coefficient has the effect of improving the stability of the above-mentioned determination.

[0359] When the comparison result of the first motion amount, the second motion amount, and the third motion amount satisfies the switching condition, the motion detection unit 345 determines that the imaging optical path switching has been successful. At this time, the measurement process (step S140) is executed.

[0360] Optical images acquired by an endoscope have strong geometric distortion. The motion detection unit 345 calculates each amount of motion using images without correcting this distortion. If switching of the imaging optical path is successful, the first and third amounts of motion calculated using the first and second images are equal to or greater than a predetermined amount. Furthermore, in the image selection process, a group of images is set so that the second amount of motion is smaller than a threshold. Therefore, if switching of the imaging optical path is successful, the second amount of motion is not large. Therefore, if the switching condition is satisfied, the motion detection unit 345 can determine that switching of the imaging optical path is successful. In consideration of the above, the motion detection unit 345 may determine that switching of the imaging optical path is possible if the values ​​of the first amount of motion (Ma1) and the third amount of motion (Ma3) are equal to or greater than a threshold. Alternatively, the motion detection unit 345 may determine that switching of the imaging optical path is possible if the values ​​of the first amount of motion (Ma1) and the third amount of motion (Ma3) are equal to or greater than a threshold and the second amount of motion (Ma2) is smaller than a threshold.

[0361] When the comparison result of the first amount of movement, the second amount of movement, and the third amount of movement does not satisfy the switching condition, the movement detection unit 345 determines that the switching of the imaging optical path has failed. At this time, the display processing unit 341 causes the display unit 5 to display information notifying the user that the switching of the imaging optical path has failed. The display unit 5 displays the information. Furthermore, the measurement process (step S140) is not executed, and step S105 is executed.

[0362] If the motion detection unit 345 determines that the switching of the imaging optical path has failed a predetermined number of times in succession, the motion detection unit 345 determines that a malfunction has occurred in the mechanism required for switching the imaging optical path. In this case, the display unit 5 displays information notifying the user that the switching of the imaging optical path has failed, and the measurement process (step S140) is stopped.

[0363] Condensation or the like may occur in a low-temperature environment, which may cause temporary failure in switching the imaging optical path. Alternatively, a malfunction may occur in the actuator that moves the shutter of the switching unit 25, which may cause failure in switching the imaging optical path. In the third embodiment, the endoscope device 1 can avoid performing inaccurate measurements using images acquired in such a situation.

[0364] After the 3D shape data is displayed and a dimension measurement process is performed based on the 3D coordinates of a point specified on the 3D shape data, the user may notice an inaccurate 3D shape or an unnatural measurement result and may have to redo the composition adjustment for imaging. In the third embodiment, such a large backtracking operation is avoided, and the decrease in inspection efficiency is minimized.

[0365] (Variation of the Third Embodiment) A variation of the third embodiment of the present invention will be described. When the motion detection unit 345 determines that switching of the imaging optical path has failed, the image selection unit 346 reselects an image set to be used as an image group from two or more image sets that have already been acquired. At this time, the image selection unit 346 excludes from the selection targets images included in the image set for which it has been determined that switching of the imaging optical path has failed.

[0366] After the image set is set again, the CPU 34 executes the switching determination process (step S145) again. When the motion detection unit 345 determines that the switching of the imaging optical path was successful, the measurement process (step S140) is executed. When the motion detection unit 345 again determines that the switching of the imaging optical path was unsuccessful, the display unit 5 displays information notifying the user that the switching of the imaging optical path was unsuccessful. Furthermore, the measurement process (step S140) is not executed, and step S105 is executed.

[0367] In the modification of the third embodiment, if a failure to switch the imaging optical path occurs by chance, the measurement process is unlikely to be stopped, and therefore, a decrease in inspection efficiency due to a failure to switch the imaging optical path is minimized.

[0368] (Fourth Embodiment) A fourth embodiment of the present invention will be described. In the fourth embodiment, a device that acquires an image of a subject is different from a device that performs measurement processing. Fig. 24 shows the configuration of an endoscope system 100 (imaging system) of the fourth embodiment. The endoscope system 100 shown in Fig. 24 includes an endoscope device 1 and an external device 6. The endoscope device 1 performs image acquisition processing and image selection processing, and transmits a group of images selected in the image selection processing to the external device 6. The external device 6 receives the group of images from the endoscope device 1 and performs measurement processing using the group of images.

[0369] The configuration of the endoscope device 1 is the same as the configuration shown in Fig. 2. The endoscope device 1 communicates with an external device 6 and transmits a group of images to the external device 6.

[0370] For example, the endoscope device 1 may have a wireless module and perform wireless communication with the external device 6. The endoscope device 1 and the external device 6 may be connected by a cable such as a LAN (Local Area Network) cable, and the endoscope device 1 may perform communication with the external device 6 via the cable.

[0371] The external device 6 may be any of a desktop PC, a laptop PC, a smartphone, and a tablet terminal, or may be a computer system that operates on the cloud.

[0372] 25 shows the functional configuration of the external device 6. The external device 6 shown in FIG.

[0373] The data communication unit 60 receives the image group from the endoscope device 1. For example, the data communication unit 60 is a wireless module and performs wireless communication with the endoscope device 1. The data communication unit 60 may also perform communication with the endoscope device 1 via a cable.

[0374] 3. The CPU 61 also controls communication that the data communication unit 60 executes with the endoscope device 1. That is, the CPU 61 causes the data communication unit 60 to receive a group of images from the endoscope device 1.

[0375] The CPU 61 may load a program including instructions that define the operation of the CPU 61 and execute the loaded program. That is, the functions of the CPU 61 may be realized by software. The method of implementing this program is the same as the method of implementing a program that realizes the functions of the endoscope device 1.

[0376] The display unit 62 has a display screen and displays images, operation menus, etc. on the display screen. The display unit 62 is a monitor (display) such as an LCD. The display unit 62 may be a touch panel display.

[0377] The RAM 63 temporarily stores information that the CPU 61 uses to control the external device 6 .

[0378] The following describes the operations of the endoscope device 1 and the external device 6. The endoscope device 1 executes steps S100 to S135 shown in Fig. 5. When the main control unit 340 determines in step S135 that image selection has been successful, the CPU 34 transmits the group of images selected in the image selection process to the external device 6.

[0379] The CPU 61 of the external device 6 causes the data communication unit 60 to receive the image group from the endoscope device 1. As a result, the data communication unit 60 receives the image group from the endoscope device 1. The CPU 61 uses the image group to execute measurement processing similar to the measurement processing shown in FIG. 12 .

[0380] The CPU 61 of the external device 6 serves as a control unit and can replace or share some or all of the operations performed by the CPU 34 of the endoscope device 1. All of the embodiments and modified examples of the present invention can be realized by using the CPU 61 of the external device 6 and the CPU 34 of the endoscope device 1 in combination, with one or more control units executing processing. Which CPU executes each operation can be changed as appropriate. The CPU 61 may be composed of at least one of a processor other than a CPU and a logic circuit. For example, the CPU 61 is at least one of a DSP and a GPU. Alternatively, the CPU 61 may be an SoC incorporating a DSP, a GPU, or both. For example, the logic circuit is at least one of an ASIC and an FPGA.

[0381] Instead of the endoscope device 1, an endoscope device 1a shown in FIG. 14 may be used.

[0382] Each aspect of the present invention may include the following modifications: An endoscope system 100 (imaging system) has an endoscope 2 which is an imaging device and a main body 3. A CPU 61 is included in an external device 6 which is different from the imaging device.

[0383] In the fourth embodiment, the CPU 34 of the endoscope device 1 sets an image group based on the amount of motion in the candidate image groups. The CPU 61 of the external device 6 generates 3D information of the subject based on the images included in the image group and the motion compensation information. Therefore, the endoscope system 100 can improve the accuracy of the 3D information of the subject without sacrificing usability, similar to the first embodiment.

[0384] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and their modifications. Addition, omission, substitution, and other modifications of the configuration are possible within the scope of the spirit of the present invention. Furthermore, the present invention is not limited by the above description, but is limited only by the scope of the appended claims.

[0385] According to each embodiment of the present invention, the imaging system, imaging method, and program can improve the accuracy of three-dimensional information of a subject without sacrificing ease of use.

[0386] DESCRIPTION OF SYMBOLS 1, 1a Endoscope device 2 Endoscope 3, 3a Main body 4 Operation unit 5, 62 Display unit 6 External device 20 Insertion unit 21, 21a Optical adapter 22 Imaging element 23, 23a, 23b Concave lens 24, 24a, 24b Convex lens 25 Switching unit 26 Imaging optical system 30 Imaging control unit 31 Video processing unit 32, 32a Light source unit 33, 33a Illumination control unit 34, 61 CPU 35 Memory 36, 36a Switching control unit 37 White light source 38 LD 60 Data communication unit 63 RAM 70, 70a Observation optical system 80, 80a Illumination optical system 81 Condenser lens 82 Light guide 83 Rod lens 84 Diffusion lens 85 Fiber 89 Stripe generating unit 100 Endoscope system 340 Main control unit 341 Display processing unit 342 Image processing unit 343 Visualization processing unit 344 Recording unit 345 Movement detection unit 346 Image selection unit 347 Measurement processing unit 890 First optical path 891 Second optical path 892 Third optical path 893 Phase shift unit

Claims

1. An imaging system comprising: an image sensor that generates images of a subject under one of two or more imaging conditions, including a first imaging condition and a second imaging condition different from the first imaging condition; and a control unit that continuously switches between the two or more imaging conditions, acquires four or more images from the image sensor, including two or more first images generated under the first imaging condition and two or more second images generated under the second imaging condition, calculates an amount of motion between a reference image included in the four or more images and an image included in the four or more images other than the reference image, sets an image group including three or more images of the four or more images based on the amount of motion, the image group including at least one first image of the two or more first images and at least one second image of the two or more second images, generates motion compensation information for compensating for motion in the image group based on the at least one first image and the at least one second image, and generates three-dimensional information of the subject based on the images included in the image group and the motion compensation information.

2. The imaging system according to claim 1, wherein the control unit generates disparity information based on the at least one first image and the at least one second image, and generates the motion compensation information based on the disparity information.

3. The imaging system according to claim 2, wherein the control unit generates two or more sets of disparity information based on at least two first images including the at least one first image and the at least one second image.

4. The imaging system described in claim 3, wherein the control unit generates the two or more sets of disparity information by executing a first process and a second process, executes the first process based on a second image included in the at least one second image and a first image included in the at least two first images, and executes the second process based on a second image included in the at least one second image and a first image included in the at least two first images and different from the first image used in the first process.

5. The imaging system described in claim 3, wherein the control unit generates the two or more sets of disparity information by executing a first process and a second process, executes the first process based on a second image included in the at least one second image and a first image included in the at least two first images, and executes the second process based on the at least two first images.

6. The image sensor further generates images of the subject under a third imaging condition different from both the first imaging condition and the second imaging condition, and the control unit: successively switches between the first imaging condition, the second imaging condition, and the third imaging condition; acquires six or more images from the image sensor, including the two or more first images, the two or more second images, and two or more third images generated under the third imaging condition; calculates an amount of motion between an image included in the six or more images and the reference image; sets an image group including three or more images of the six or more images based on the amount of motion, the image group including at least one first image of the two or more first images, at least one second image of the two or more second images, and at least one third image of the two or more third images; and generates the motion compensation information based on at least two images generated under different imaging conditions from the at least one first image, the at least one second image, and the at least one third image. The imaging system according to claim 2 .

7. The imaging system described in claim 1, wherein the control unit acquires an image from the image sensor during a readout period shorter than one frame period, turns off a light source that generates illumination light to be irradiated onto the subject during the readout period, turns on the light source during at least a part of a period excluding the readout period, and switches between the first imaging condition and the second imaging condition during the readout period.

8. The imaging system of claim 1, wherein the control unit calculates the time difference between a reference time and the time at which the image sensor generates an image included in the image group after the reference time, and sets the image group so that the time difference is equal to or less than a predetermined threshold value.

9. The imaging system of claim 1, wherein the control unit: sets a first provisional image group and a second provisional image group based on the amount of motion, each of the first provisional image group and the second provisional image group including three or more images out of the four or more images, and at least one image included in the first provisional image group is different from at least one image included in the second provisional image group; calculates a first time difference between a reference timing and a timing at which the image sensor generates an image included in the first provisional image group after the reference timing; calculates a second time difference between the reference timing and a timing at which the image sensor generates an image included in the second provisional image group after the reference timing; sets the first provisional image group as the image group when the first time difference is smaller than the second time difference; and sets the second provisional image group as the image group when the second time difference is smaller than the first time difference.

10. The imaging system of claim 1, comprising: a first objective optical system; and a second objective optical system having parallax relative to the first objective optical system, wherein the first imaging condition indicates that the image sensor generates an image at a first timing based on an optical image of the subject formed through the first objective optical system; and the second imaging condition indicates that the image sensor generates an image at a second timing different from the first timing based on the optical image of the subject formed through the second objective optical system.

11. The imaging system of claim 1, wherein the control unit: calculates a first amount of motion between a first reference image included in the two or more first images and a first image that is different from the first reference image and is included in the two or more first images; calculates a second amount of motion between a second reference image included in the two or more second images and a second image that is different from the second reference image and is included in the two or more second images; and sets the group of images based on the first amount of motion and the second amount of motion.

12. The imaging system according to claim 1, wherein the control unit determines the linearity of the movement based on the amount of movement, and sets the image group based on the linearity.

13. The imaging system described in claim 12, wherein the control unit: sets a first provisional image group and a second provisional image group, each of which includes three or more images out of the four or more images, and at least one image included in the first provisional image group is different from at least one image included in the second provisional image group; determines the linearity of the movement of each of the first provisional image group and the second provisional image group based on the amount of motion; sets the first provisional image group as the image group when the linearity of the first provisional image group is higher than the linearity of the second provisional image group; and sets the second provisional image group as the image group when the linearity of the second provisional image group is higher than the linearity of the first provisional image group.

14. The imaging system of claim 1, wherein the group of images includes at least two first images of the two or more first images and at least one second image of the two or more second images.

15. The imaging system of claim 1, wherein the control unit generates disparity information based on a first image included in the image group and a second image included in the image group, generates the motion compensation information by correcting the disparity information, and generates a three-dimensional point cloud including two or more points on the subject as the three-dimensional information by using the first image or the second image used to generate the disparity information and the motion compensation information.

16. The imaging system according to claim 1, wherein the image sensor generates the reference image by generating an image of the subject under a third imaging condition different from both the first imaging condition and the second imaging condition.

17. The imaging system of claim 1, further comprising a stripe pattern generator that generates light having a stripe pattern and that is projected onto the subject, wherein the first imaging condition indicates a first stripe pattern that the light has, and the second imaging condition indicates a second stripe pattern that the light has, and the second stripe pattern is different from the first stripe pattern.

18. The image sensor further generates an image of the subject under a third imaging condition different from both the first imaging condition and the second imaging condition, the third imaging condition indicating a third stripe pattern of the light, the third stripe pattern being different from both the first stripe pattern and the second stripe pattern, the image sensor further generates an image of the subject under a fourth imaging condition different from both the first imaging condition, the second imaging condition, and the third imaging condition, the control unit: successively switches between the first imaging condition, the second imaging condition, the third imaging condition, and the fourth imaging condition, acquires six or more images from the image sensor, including the two or more first images, the two or more second images, and two or more third images generated under the third imaging condition, calculates the amount of motion between an image included in the six or more images and the reference image, the imaging system of claim 17 further comprising: an image group including three or more images of the six or more images based on the amount of motion; the image group including at least one first image of the two or more first images, at least one second image of the two or more second images, and at least one third image of the two or more third images; and the imaging system of claim 17 further comprising: a fourth image generated under the fourth imaging condition; 19. The imaging system of claim 18, wherein the control unit: calculates a first amount of motion between a first image included in the two or more first images and the reference image; calculates a second amount of motion between a second image included in the two or more second images and the reference image; calculates a third amount of motion between a third image included in the two or more third images and the reference image; and sets the group of images based on the first amount of motion, the second amount of motion, and the third amount of motion.

20. The imaging system of claim 18, wherein the control unit generates a three-dimensional point cloud including two or more points on the subject, shifts pixels of the fourth image based on the motion compensation information, and generates the three-dimensional information by correlating the colors of the shifted pixels with points included in the two or more points.

21. The imaging system of claim 18, wherein the control unit generates a three-dimensional point cloud including two or more points on the subject, shifts pixels of the fourth image based on the motion compensation information, generates a texture image based on the fourth image having the shifted pixels, converts the three-dimensional point cloud into two or more meshes, and generates a mesh polygon as the three-dimensional information by combining the two or more meshes and the texture image.

22. The imaging system according to claim 1, further comprising an imaging device having the image sensor and the control unit.

23. The imaging system according to claim 1, further comprising an imaging device having the image sensor, wherein the control unit is included in a device different from the imaging device.

24. A control unit successively switches between two or more imaging conditions to cause an image sensor to generate an image of a subject under any one of the two or more imaging conditions, the imaging conditions including a first imaging condition and a second imaging condition different from the first imaging condition; the control unit acquires four or more images from the image sensor, the four or more images including two or more first images generated under the first imaging condition and two or more second images generated under the second imaging condition; the control unit calculates the amount of motion between a reference image included in the four or more images and an image other than the reference image included in the four or more images, the control unit setting an image group including three or more images of the four or more images based on the amount of motion, the image group including at least one first image of the two or more first images and at least one second image of the two or more second images; the control unit generating motion compensation information for compensating for motion in the image group based on the at least one first image and the at least one second image; generating, by the control unit, three-dimensional information of the subject based on the images included in the image group and the motion compensation information.

25. A method for detecting an object using an image sensor, comprising: continuously switching between two or more imaging conditions, the imaging conditions including a first imaging condition and a second imaging condition different from the first imaging condition, to generate an image of the object under one of the two or more imaging conditions; acquiring four or more images from the image sensor, the four or more images including two or more first images generated under the first imaging condition and two or more second images generated under the second imaging condition; calculating an amount of motion between a reference image included in the four or more images and an image other than the reference image included in the four or more images, based on the amount of motion; setting an image group including three or more images of the four or more images based on the amount of motion, the image group including at least one first image of the two or more first images and at least one second image of the two or more second images; generating motion compensation information for compensating for motion in the image group based on the at least one first image and the at least one second image; generating three-dimensional information of the object based on the images included in the image group and the motion compensation information. A program that causes a computer to execute the following.

Images