Control device, learned model generation device, endoscope system, medical support method, learned model generation method, and program

Abstract

To provide a control device, a learned model generation device, an endoscope system, a medical support method, a learned model generation method, and a program that are capable of further improving detection accuracy of a distance from an optical system to a measurement target object such as a subject.SOLUTION: A control device according to the present invention is configured to input a first image, the second image, and a third image as input parameters to a learned model that outputs, as an output parameter, a depth map indicating a distance from an optical system to a subject, so as to infer the depth map of the subject; and output the depth map of the subject.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] The present disclosure relates to a control device, a trained model generation device, an endoscope system, a medical support method, a trained model generation method, and a program. [Background technology]

[0002] Conventionally, in imaging devices such as digital cameras, a technique is known for estimating the distance between a subject and the imaging device from the degree of blur in a captured image. For example, in Patent Document 1, a second image affected by the aberration of the optical system is input to a statistical model generated by learning the blur that changes nonlinearly according to the distance to the subject in a first image affected by the aberration of the optical system, and distance information indicating the distance to the subject in the second image is obtained.

[0003] In addition, in Patent Document 2, the distance between the imaging device and the subject in each image region in the sweep image is calculated based on the amount of blur in each image region relative to an all-focus image in a sweep image captured by exposing an image sensor while continuously moving the focal length of the lens from the near end to the far end, and the optical coefficient values ​​of the imaging device including the focal length of the lens, and a depth map (distance map) is generated that indicates this distance with pixel values ​​in each image region. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-148483 A [Patent Document 2] Patent No. 5868183 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the above-mentioned cited documents 1 and 2, the data captured during focus drive is not effectively utilized, so there is room for further improvement in the detection accuracy of the distance from the optical system to a measurement target such as a subject.

[0006] The present disclosure has been made in consideration of the above, and aims to provide a control device, a trained model generation device, an endoscopic system, a medical support method, a trained model generation method, and a program that can further improve the detection accuracy of the distance from an optical system to a measurement object such as a subject. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the object, a control device according to the present disclosure is a control device including a processor, wherein the processor acquires from an imaging device including an optical system capable of changing a focal length, an imaging element that exposes an image formed by the optical system to generate a still image, and a focus adjustment mechanism that moves the optical system along an optical axis direction of the optical system to change the focal length: a first image generated by the imaging element imaging a subject on the near point side of the optical system; a second image generated by the imaging element imaging the subject during a period in which the optical system is moved between the near point side and the far point side by the focus adjustment mechanism while changing the focal length; and a third image generated by the imaging element imaging the subject on the far point side of the optical system; and infers a depth map of the subject by inputting the first image, the second image, and the third image as input parameters into a trained model that outputs a depth map indicating a distance from the optical system to the subject as an output parameter, and outputs the depth map of the subject.

[0008] In addition, in the control device of the present disclosure, in the above disclosure, the second image is at least one frame image in a video composed of a group of temporally consecutive images taken by the imaging element during a period in which the focus adjustment mechanism changes the focal length between the near point side and the far point side.

[0009] In addition, in the control device according to the present disclosure, in the above disclosure, the processor acquires camera parameters of the optical system, and estimates a shape of the subject based on the camera parameters and a depth map of the subject.

[0010] In the control device according to the present disclosure, the processor controls the focus adjustment mechanism to stop the optical system at each of the near point end and the far point end.

[0011] In the control device according to the present disclosure, the processor controls the focus adjustment mechanism to move the optical system linearly from a near point end to a far point end.

[0012] In the control device according to the present disclosure, the processor controls the focus adjustment mechanism so as to stop the optical system at each of the near point end and the far point end within a certain period of time.

[0013] Further, a trained model generation device according to the present disclosure is a trained model generation device including a processor, wherein the processor acquires learning data from an imaging device including an optical system capable of changing a focal length, an imaging element that generates a still image by exposing an image formed by the optical system, and a focus adjustment mechanism that changes the focal length by moving the optical system along an optical axis direction of the optical system, in which the processor acquires learning data combining, for each of the multiple objects, a plurality of first images generated by the imaging element by imaging each of the multiple objects on the near point side of the optical system, a plurality of second images generated by the imaging element by imaging each of the multiple objects during a period in which the optical system is moved between the near point side and the far point side by the focus adjustment mechanism while changing the focal length, a plurality of third images generated by the imaging element by imaging each of the multiple objects on the far point side of the optical system, and a ground truth value of a depth map relating to the distance from the optical system to each of the multiple objects, and performs learning using the learning data to generate a trained model that uses the first image, the second image, and the third image as input parameters and outputs a depth map indicating the distance from the optical system to a subject as an output parameter.

[0014] Moreover, an endoscopic system according to the present disclosure includes an endoscope capable of imaging the inside of a subject's body, and a control device including a processor and to which the endoscope can be connected, the endoscope including an optical system capable of changing a focal length, an imaging element that exposes an image formed by the optical system to generate a still image, and a focus adjustment mechanism that moves the optical system along an optical axis direction of the optical system to change the focal length, the processor acquires from the endoscope a first image generated by the imaging element imaging the subject on a near point side of the optical system, a second image generated by the imaging element imaging the subject during a period in which the optical system is moved between the near point side and the far point side by the focus adjustment mechanism while changing the focal length, and a third image generated by the imaging element imaging the subject on the far point side of the optical system, and infers a depth map of the subject by inputting the first image, the second image, and the third image as input parameters into a trained model that outputs a depth map indicating a distance from the optical system to the subject as an output parameter, and outputs the depth map of the subject.

[0015] Furthermore, a medical support method according to the present disclosure is a medical support method executed by a control device having a processor, the processor acquiring from an imaging device including an optical system capable of changing a focal length, an imaging element that exposes an image formed by the optical system to generate a still image, and a focus adjustment mechanism that moves the optical system along an optical axis direction of the optical system to change the focal length: a first image generated by the imaging element imaging a subject on a near point side of the optical system; a second image generated by the imaging element imaging the subject during a period in which the optical system is moved between a near point side and a far point side by the focus adjustment mechanism while changing the focal length; and a third image generated by the imaging element imaging the subject on the far point side of the optical system; and inferring a depth map of the subject by inputting the first image, the second image, and the third image as input parameters into a trained model that outputs a depth map indicating a distance from the optical system to the subject as an output parameter, and outputting the depth map of the subject.

[0016] Further, a learning method according to the present disclosure is a learning method executed by a trained model generation device including a processor, in which the processor acquires from an imaging device including an optical system capable of changing a focal length, an image sensor that exposes an image formed by the optical system to generate a still image, and a focus adjustment mechanism that changes the focal length by moving the optical system along an optical axis of the optical system, a plurality of first images generated by the image sensor capturing images of a plurality of objects on a near point side of the optical system, and a focus adjustment mechanism that changes the focal length by moving the optical system along an optical axis of the optical system, the plurality of first images being acquired by the image sensor on the near point side of the optical system, and a plurality of still images being acquired by the image sensor during a period in which the optical system is moved between a near point side and a far point side by the focus adjustment mechanism. the acquisition of learning data combining, for each of the plurality of objects, a plurality of second images generated by imaging each of the plurality of objects, a plurality of third images generated by the imaging element imaging each of the plurality of objects at a far point side of the optical system, and a ground truth value of a depth map relating to a distance from the optical system to each of the plurality of objects, and a trained model that uses the first image, the second image, and the third image as input parameters and outputs, as an output parameter, a depth map indicating a distance from the optical system to a subject, by training using the training data.

[0017] In addition, a program according to the present disclosure is a program executed by a control device having a processor, the program acquiring from an imaging device including an optical system capable of changing a focal length, an imaging element that generates a still image by exposing an image formed by the optical system, and a focus adjustment mechanism that changes the focal length by moving the optical system along an optical axis direction of the optical system: a first image generated by the imaging element imaging a subject on the near point side of the optical system; a second image generated by the imaging element imaging the subject during a period in which the optical system is moved between the near point side and the far point side by the focus adjustment mechanism while changing the focal length; and a third image generated by the imaging element imaging the subject on the far point side of the optical system; and inferring a depth map of the subject by inputting the first image, the second image, and the third image as input parameters into a trained model that outputs a depth map indicating a distance from the optical system to the subject as an output parameter, and outputting the depth map of the subject.

[0018] Further, a program according to the present disclosure is a program executed by a trained model generation device having a processor, the program acquiring learning data from an imaging device including an optical system capable of changing a focal length, an imaging element that generates a still image by exposing an image formed by the optical system, and a focus adjustment mechanism that changes the focal length by moving the optical system along an optical axis direction of the optical system, in which learning data is combined with each of the multiple objects, the multiple first images generated by the imaging element by imaging each of the multiple objects on the near point side of the optical system, the multiple second images generated by the imaging element by imaging each of the multiple objects during a period in which the optical system is moved between the near point side and the far point side by the focus adjustment mechanism while changing the focal length, the multiple third images generated by the imaging element by imaging each of the multiple objects on the far point side of the optical system, and a ground truth value of a depth map relating to the distance from the optical system to each of the multiple objects, and the program acquires learning data by combining the first image, the second image, and the third image with each of the multiple objects, the multiple first images being input parameters, and a depth map indicating the distance from the optical system to a subject being output as an output parameter. Effect of the Invention

[0019] According to the present disclosure, an effect is achieved in that the estimation accuracy of the three-dimensional shape of a measurement object can be further improved. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a block diagram illustrating a functional configuration of a trained model generation device according to an embodiment. [Diagram 2] FIG. 2 is a diagram illustrating the movement of a focus lens in an optical system of a trained model generation device according to one embodiment. [Diagram 3] FIG. 3 is a diagram showing an overall configuration of an endoscope system according to an embodiment. [Figure 4] FIG. 4 is a block diagram showing the functional configuration of the main parts of an endoscope and a control device according to an embodiment. [Diagram 5] FIG. 5 is a flowchart showing an outline of processing executed by a trained model generation device according to one embodiment. [Figure 6] FIG. 6 is a diagram illustrating a learning data set that is input to a learning model by a learning unit according to an embodiment. [Figure 7] FIG. 7 is a diagram for explaining a schematic link between a depth map and an image. [Figure 8] FIG. 8 is a flowchart illustrating an outline of processing executed by the endoscope system according to one embodiment. [Figure 9] FIG. 9 is a flowchart showing an outline of the measurement mode process in FIG. [Figure 10] FIG. 10 is a timing chart showing the position on the optical axis of a focus lens provided in an endoscope and exposure timing of an image sensor during measurement mode processing executed by an endoscope system according to one embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a subject imaged by an endoscope. [Figure 12] FIG. 12 is a diagram showing an example of image data captured by an endoscope. [Figure 13] FIG. 13 is a diagram showing another example of image data captured by the endoscope. [Figure 14] FIG. 14 is a diagram showing another example of image data captured by the endoscope. [Figure 15] FIG. 15 is a diagram for explaining a schematic method for the calculation unit to estimate the shape of the subject. [Figure 16] FIG. 16 is a diagram showing an example of an image corresponding to the shape of a subject. [Figure 17] FIG. 17 is a timing chart showing the position on the optical axis of a focus lens provided in an endoscope and exposure timing of an image sensor during measurement mode processing executed by an endoscope system according to a first modified example of an embodiment. [Figure 18]FIG. 18 is a timing chart showing the position on the optical axis of the focus lens provided in the endoscope and the exposure timing of the image sensor during measurement mode processing executed by the endoscope system according to the first modification of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] The trained model generating device and the endoscope system according to the present disclosure will be described in detail below with reference to the drawings. Note that the present disclosure is not limited to the following embodiments. In addition, each figure referred to in the following description merely shows the shape, size, and positional relationship in a schematic manner to the extent that the contents of the present disclosure can be understood. That is, the present disclosure is not limited to only the shape, size, and positional relationship exemplified in each figure. Furthermore, in the description of the drawings, the same parts are denoted by the same reference numerals. In addition, the configuration of the trained model generating device according to the present disclosure will be described below, followed by the configuration of the endoscope system.

[0022] [Configuration of trained model generation device] Fig. 1 is a block diagram showing a functional configuration of a trained model generating device according to an embodiment. The trained model generating device 1 shown in Fig. 1 is a device that generates a trained model by inputting training data of a plurality of combinations in which a plurality of images having different focal positions (focal points) are associated with a depth map (correct value) indicating the distance between the subject and the optical system for each pixel, into a predetermined training model.

[0023] The trained model generation device 1 shown in FIG. 1 includes an imaging device 11, a display unit 12, an input unit 13, a communication unit 14, a recording unit 15, and a first control unit 16.

[0024] Under the control of the first control unit 16, the imaging device 11 generates image data (RAW data) of a subject image by collecting and receiving light from a predetermined field of view area, and outputs the image data to the first control unit 16. The imaging device 11 has an optical system 101, an imaging element 102, and a focus adjustment mechanism 103.

[0025] The optical system 101 is composed of lenses L1 to L10 and an aperture Di. The optical system 101 forms an image of a subject on a light receiving surface of an image sensor 102. The optical system 101 also has a focus lens L5, among the lenses L1 to L10, that is movable along the optical axis O1. The focus lens L5 is moved back and forth along the optical axis O1 by a focus adjustment mechanism 103, which will be described later.

[0026] The image sensor 102 generates image data by receiving the subject image formed by the optical system 101. The image sensor 102 is configured using an image sensor such as a Complementary Metal Oxide Semiconductor (CMOS) sensor or a Charge Coupled Device (CCD) sensor.

[0027] The focus adjustment mechanism 103 changes the focal position (focal point) of the optical system 101 by moving the focus lens L5 of the optical system 101 along the optical axis O1 under the control of the first control unit 16. The focus adjustment mechanism 103 has a focus drive unit 103a and a detection unit 103b.

[0028] The focus driver 103a is configured using an actuator including, for example, a stepping motor, a voice coil motor, or the like, and is driven under the control of the first controller 16 to move the focus lens L5 to a predetermined position on the optical axis O1.

[0029] The detector 103b detects focal length information related to the position or focal length of the focus lens L5 in the optical system 101, and outputs the detection result to the first controller 16. The detector 103b is configured using, for example, an encoder or a photocoupler. The detector 103b may detect the position of the focus lens L5 on the optical axis O1 based on the drive amount of the focus driver 103a.

[0030] 2 is a diagram for explaining the movement of the focus lens L5 in the optical system 101. As shown in (a) and (b) of Fig. 2, the focus driver 103a, under the control of the first controller 16, drives the focus lens L5 to move it to a predetermined position on the optical axis O1, thereby changing the focal position P1 (focal point) of the optical system 101 ((a) of Fig. 2→(b) of Fig. 2). This allows the optical system 101 to change the focal position P1 between the far point end and the near point end.

[0031] Returning to FIG. 1, the description of the configuration of the trained model generation device 1 will continue. The display unit 12 displays an image corresponding to various information and image data input from the first control unit 16. The display unit 12 is configured using an organic electroluminescent display (OLED) or a liquid crystal display.

[0032] The input unit 13 receives input of various operations by the user, and outputs information corresponding to the received various operations to the first control unit 16. The input unit 13 is configured using an input interface such as a keyboard, a mouse, a touch panel, and the like.

[0033] The communication unit 14 transmits various information input from the first control unit 16 to the outside, and outputs various information received from the outside to the first control unit 16. The communication unit 14 is configured using, for example, a communication module capable of Wi-Fi (registered trademark) or Bluetooth (registered trademark).

[0034] The recording unit 15 records various information related to the trained model generation device 1. The recording unit 15 is configured using a hard disk drive (HDD), a solid state drive (SSD), a flash memory, a volatile memory, a non-volatile memory, etc. The recording unit 15 has a program recording unit 151, a trained model recording unit 152, and a trained model recording unit 153.

[0035] The program recording unit 151 records various programs executed by the trained model generation device 1, for example, a program for generating a trained model.

[0036] The learning model recording unit 152 records a learning model for the first control unit 16, which will be described later, to input input parameters.

[0037] The trained model recording unit 153 records a plurality of trained models trained using training data.

[0038] The first control unit 16 is realized by using a processor having hardware such as a field-programmable gate array (FPGA), a graphics processing unit (GPU), or a central processing unit (CPU), and a memory that is a temporary storage area used by the processor. The first control unit 16 has a drive control unit 161, an imaging control unit 162, an acquisition unit 163, a determination unit 164, a learning unit 165, and an output control unit 166.

[0039] The drive control unit 161 drives the focus adjustment mechanism 103 to move the focus lens L5 to the near point end of the optical system 101.

[0040] The imaging control unit 162 causes the imaging element 102 to capture images at the near point end of the focus lens L5, while the lens is moving from the near point end to the far point end, and at the far point end.

[0041] The acquisition unit 163 acquires image data generated by the imaging element 102. Specifically, the acquisition unit 163 acquires image data (hereinafter simply referred to as "first image") generated by the imaging element 102 capturing an image at the near end of the focus lens L5. The acquisition unit 163 also acquires a plurality of image data (hereinafter simply referred to as "second images") generated by the imaging element 102 capturing a plurality of images during a moving period in which the focus lens L5 moves from the near end to the far end. The acquisition unit 163 acquires image data (hereinafter simply referred to as "third image") generated by the imaging element 102 capturing an image at the far end of the focus lens L5.

[0042] The determination unit 164 determines whether or not a predetermined number of images required for learning have been acquired.

[0043] The learning unit 165 learns using a predetermined learning model and learning data that combines a plurality of images, each of which is a plurality of images for each subject acquired by the acquisition unit 163 and has different focus positions, and a depth map. Specifically, the learning unit 165 learns using a learning model generated by machine learning using artificial intelligence (AI (Artificial Intelligence)). The method for constructing the learning model used by the learning unit 165 is not particularly limited, and various machine learning methods such as deep learning using a neural network, a support vector machine, a decision tree, a simple Bayes method, and a k-nearest neighbor method can be used. In addition, the depth map is information in which the subject distance from the position of the image sensor 203 (the position of the tip 24) to a corresponding position on the observation target corresponding to the pixel position in the captured image is detected for each pixel position.

[0044] The output control unit 166 outputs the trained model generated by the learning unit 165 to the trained model recording unit 153.

[0045] [Overall configuration of endoscope system] Next, a description will be given of the configuration of the endoscope system 100. Fig. 3 is a diagram showing the overall configuration of the endoscope system 100 according to one embodiment.

[0046] The endoscope system 100 shown in Fig. 3 continuously captures images of the inside of a subject, such as a human or animal, by inserting the insertion section 21 of the endoscope 2 from the mouth to the esophagus or from the anus to the large intestine of the subject, and displays the captured image data group on the display device 3 in chronological order. Furthermore, the endoscope system 100 detects (recognizes or estimates) characteristic regions including organs, parts, polyps, cancers, etc. of multiple observation sites shown in the image data and treatments on the living body using treatment tools such as forceps and snares, based on the image data group input from the control device 4 to the image diagnostic device 5 (CAD: Computer Aided Detection) and an inference model (trained model) trained using training data previously obtained by imaging the organs, parts, lesions, etc. of the subject, and outputs the detection result information to the control device 4. An operator, such as a doctor, observes and treats the subject while checking the display image displayed on the display device 3 and the detection result (estimation result) of the image diagnostic device 5.

[0047] As shown in FIG. 3, the endoscope system 100 includes an endoscope 2, a display device 3, a control device 4, and an image diagnostic device 5.

[0048] The endoscope 2 continuously generates image data (RAW data) by capturing images of the inside of the subject, and sequentially outputs the image data to the control device 4. As shown in FIG. 1, the endoscope 2 includes an insertion section 21, an operation section 22, and a universal cord 23.

[0049] The insertion section 21 has at least a portion that is flexible and is inserted into a subject. As shown in Fig. 1, the insertion section 21 includes a tip section 24 provided at the tip of the insertion section 21, a bending section 25 that is connected to a base end side (operation section 22 side) of the tip section 24 and configured to be bendable, and a long flexible tube section 26 that is connected to the base end side of the bending section 25 and has flexibility.

[0050] The operation unit 22 is connected to a base end portion of the insertion unit 21. The operation unit 22 receives various operations on the endoscope 2. As shown in FIG. 1, the operation unit 22 is provided with a bending knob 221, an insertion port 222, and a plurality of operation members 223.

[0051] The bending knob 221 is configured to be rotatable in response to a user operation by a user such as an operator. The bending knob 221 rotates to operate a bending mechanism (not shown) such as a metal or resin wire disposed in the insertion section 21. As a result, the bending section 25 is bent.

[0052] The insertion port 222 communicates with a treatment instrument channel (not shown), which is a duct extending from the tip of the insertion portion 21, and is an insertion port for inserting a treatment instrument or the like from the outside of the endoscope 2 into the treatment instrument channel.

[0053] The multiple operation members 223 are composed of buttons and the like that accept various operations by a user such as an operator, and output operation signals corresponding to the various operations to the control device 4 via the universal cord 23. Examples of the various operations include a release operation that instructs the endoscope 2 to capture a still image, and an operation that switches the observation mode of the endoscope 2 to a normal light observation mode or a special observation mode.

[0054] The universal cord 23 extends from the operation section 22 in a direction different from the extending direction of the insertion section 21, and is a cord on which a light guide 231 (see FIG. 2) made of an optical fiber or the like, a first signal line 232 (see FIG. 2) for transmitting the above-mentioned image data, a second signal line 233 (see FIG. 2) for transmitting the above-mentioned operation signal, and the like are arranged. A first connector section 27 is provided at the base end of the universal cord 23, as shown in FIG. 1. The first connector section 27 is detachably connected to the control device 4.

[0055] The display device 3 is composed of a display monitor such as a liquid crystal or organic EL display monitor, and under the control of the control device 4, displays an image based on image data that has been image-processed in the control device 4, as well as various information related to the endoscope 2.

[0056] The control device 4 is realized by using a processor, which is a processing device having hardware such as a GPU, FPGA, or CPU, and a memory, which is a temporary storage area used by the processor. The control device 4 comprehensively controls the operation of each part of the endoscope 2 according to a program recorded in the memory.

[0057] The image diagnostic device 5 (CAD) detects (estimates) organs, parts, the insertion speed of the insertion part 21, characteristic regions (abnormal regions or lesion candidate regions) including polyps or cancer, and treatments on a living body by treatment tools such as forceps and snares, which are captured in the image data, using a group of image data input from the control device 4 and a trained model trained in advance with training data, and outputs detection result information that associates the detection results with the detection times to the control device 4. The image diagnostic device 5 is realized using a processor, which is a processing device having hardware such as a GPU, FPGA, or CPU, and a memory, which is a temporary storage area used by the processor.

[0058] [Functional configuration of the main parts of the medical support system] Next, the functional configuration of the main parts of the above-mentioned endoscope system 100 will be described. 4 is a block diagram showing the functional configuration of the main parts of the endoscope 2 and the control device 4. The endoscope 2 and the control device 4 will be described below in that order.

[0059] [Functional configuration of the endoscope] First, the configuration of the endoscope 2 will be described. 4, the endoscope 2 includes an illumination optical system 201, an optical system 202, an image sensor 203, an A / D converter 204, a P / S converter 205, a focus adjustment mechanism 206, and an image capture control unit 207. Each of the illumination optical system 201, the optical system 202, the image sensor 203, the A / D converter 204, the P / S converter 205, the focus adjustment mechanism 206, and the image capture control unit 207 is disposed within the tip portion 24.

[0060] The illumination optical system 201 is composed of one or more lenses and irradiates the subject with illumination light supplied from a light guide 231 .

[0061] The optical system 202 is configured using lenses L1 to L10 and an aperture Di. The optical system 202 forms an image of a subject on a light receiving surface of an image sensor 203. The optical system 202 also includes a focus lens L5 that is movable along an optical axis O1 direction among the lenses L1 to L10. The focus lens L5 is moved back and forth along the optical axis O1 direction by a focus adjustment mechanism 206. In one embodiment, the optical system 202 can switch between a focus position suitable for observing a middle or distant view and a focus position suitable for observing a near point. Of course, the optical system 202 does not need to switch the focus position discretely with two values, and may be continuously changeable to a plurality of desired positions.

[0062] The image sensor 203 is configured using a CCD or CMOS (image sensor) in which one of color filters constituting a Bayer array (RGGB) is arranged on each of a plurality of pixels arranged in a two-dimensional matrix. The image sensor 203 receives a subject image formed by the optical system 202 under the control of an image capture control unit 207, and performs photoelectric conversion to generate a captured image (analog signal). Note that in one embodiment, the image sensor 203 may be configured integrally with an image sensor and a TOF (Time Of Flight) sensor that acquires a depth map (subject distance information) by a TOF method. Note that the configuration for generating the depth map is not limited to the above-mentioned TOF sensor, and an image sensor including a phase difference sensor may be adopted. Hereinafter, the depth map and the captured image will be collectively referred to as image data. The image sensor 203 outputs the image data to the A / D conversion unit 204.

[0063] The A / D conversion unit 204 is configured using an A / D conversion circuit etc. Under the control of the imaging control unit 207, the A / D conversion unit 204 performs A / D conversion processing on the analog image data input from the imaging element 203 and outputs the result to the P / S conversion unit 205.

[0064] The P / S conversion unit 205 is configured using a P / S conversion circuit, etc., and under the control of the imaging control unit 207, performs parallel / serial conversion on the digital image data input from the A / D conversion unit 204, and outputs it to the control device 4 via the first signal line 232. Note that, instead of the P / S conversion unit 205, an E / O conversion unit that converts image data into an optical signal may be provided, and the image data may be output to the control device 4 by the optical signal. Also, the image data may be transmitted to the control device 4 by wireless communication such as Wi-Fi (Wireless Fidelity) (registered trademark).

[0065] The focus adjustment mechanism 206 changes the focal point (focus position) of the optical system 101 by moving the focus lens L5 of the optical system 101 along the optical axis O1 under the control of the second control unit 410 or the imaging control unit 207. The focus adjustment mechanism 206 has a focus drive unit 206a and a detection unit 206b.

[0066] The focus driver 206a is configured using an actuator including, for example, a stepping motor or a voice coil motor, and is driven under the control of the second controller 410 or the imaging controller 207 to move the focus lens L5 to a predetermined position on the optical axis O1.

[0067] The detector 206b detects focal length information related to the position or focal length of the focus lens L5 in the optical system 202, and outputs the detection result to the second controller 410. The detector 206b is configured using, for example, an encoder or a photocoupler. The detector 206b may detect the position of the focus lens L5 on the optical axis O1 based on the drive amount of the focus driver 206a.

[0068] The imaging control unit 207 is realized by using a TG (Timing Generator), a processor which is a processing device having hardware such as a CPU, and a memory which is a temporary storage area used by the processor. The imaging control unit 207 controls the operations of the imaging element 203, the A / D conversion unit 204, and the P / S conversion unit 205 based on setting data received from the control device 4 via a second signal line 233.

[0069] [Configuration of the control device] Next, the configuration of the control device 4 will be described. As shown in FIG. 2, the control device 4 includes a focusing lens 401, a first light source unit 402, a second light source unit 403, a light source control unit 404, an S / P conversion unit 405, an image processing unit 406, an input unit 407, a recording unit 408, a communication unit 409, and a second control unit 410.

[0070] The condenser lens 401 condenses the light emitted by each of the first light source unit 402 and the second light source unit 403 , and emits the light to the light guide 231 .

[0071] The first light source unit 402 emits visible white light (normal light) under the control of the light source control unit 404, thereby supplying the white light as illumination light to the light guide 231. The first light source unit 402 is configured using a collimator lens, a white LED (Light Emitting Diode) lamp, a driving driver, and the like. The first light source unit 402 may be configured using a red LED lamp, a green LED lamp, and a blue LED lamp that simultaneously emit light to supply visible white light. The first light source unit 402 may also be configured using a halogen lamp, a xenon lamp, and the like.

[0072] The second light source unit 403 emits special light having a predetermined wavelength band under the control of the light source control unit 404, thereby supplying the special light as illumination light to the light guide 231. Here, the special light is light used for narrow band imaging (NBI) using narrow band light including 390 to 445 nm and 530 to 550 nm. Of course, the special light may be amber light (600 nm and 630 nm) used for red dichromatic imaging (RDI) other than narrow band light.

[0073] The light source control unit 404 is realized by using a processor having hardware such as an FPGA or a CPU, and a memory that is a temporary storage area used by the processor. The light source control unit 404 controls the light emission timing and light emission time of each of the first light source unit 402 and the second light source unit 403 based on control data input from the second control unit 410.

[0074] The S / P conversion unit 405, under the control of the second control unit 410, performs serial / parallel conversion on the image data received from the endoscope 2 via the first signal line 232, and outputs the data to the image processing unit 406. When the endoscope 2 outputs image data as an optical signal, an O / E conversion unit that converts an optical signal into an electrical signal may be provided instead of the S / P conversion unit 405. When the endoscope 2 transmits image data via wireless communication, a communication module capable of receiving wireless signals may be provided instead of the S / P conversion unit 405.

[0075] The image processing unit 406 is realized by using a processor having hardware such as a GPU or FPGA, and a memory that is a temporary storage area used by the processor. Under the control of the second control unit 410, the image processing unit 406 performs predetermined image processing on the image data of parallel data input from the S / P conversion unit 405, and outputs the image data to the display device 3. Here, examples of the predetermined image processing include demosaic processing, white balance processing, gain adjustment processing, gamma correction processing, and format conversion processing.

[0076] The input unit 407 is composed of a mouse, a foot switch, a keyboard, buttons, switches, a touch panel, etc., and accepts user operations by a user such as a surgeon, and outputs an operation signal corresponding to the user operation to the second control unit 410.

[0077] The recording unit 408 is configured using a recording medium such as a volatile memory, a non-volatile memory, an SSD (Solid State Drive), an HDD (Hard Disk Drive), a memory card, etc. The recording unit 408 records data including various parameters and the like required for the operation of the control device 4 and the endoscope 2. The recording unit 408 also has a program recording unit 408a that records various programs for operating the endoscope 2 and the control device 4, an image data recording unit 408b that records an image file that stores an image corresponding to the image data, a trained model recording unit 408c that records a trained model, and a camera parameter recording unit 408d.

[0078] The image data recording unit 408b, under the control of the second control unit 410, records an image (endoscopic image) corresponding to the image data group and still image data generated by the endoscope 2 successively capturing images of a plurality of observation sites on the subject, in association with patient information, etc. Here, the still image data is image data captured by the image sensor 203 when imaging is instructed by a release signal input from the operation unit 22, and is image data with a higher resolution than the live view images sequentially generated by the image sensor 203.

[0079] The trained model recording unit 408c records a trained model that uses multiple images with different focus positions as input parameters and outputs a depth map indicating the distance from the optical system 202 to the subject as output parameters.

[0080] The camera parameter recording unit 408d records the camera parameters of the optical system 202. Specifically, the camera parameters are conversion coefficients that link an image plane (position of pixels of the image sensor 203) with an object (position of a subject). For example, the camera parameters are conversion matrices that can be acquired by performing camera calibration using a checkered flat plate. The camera parameter recording unit 408d may also record the value of a point (x, y) on an image corresponding to a depth z for each pixel of the image sensor 203 as a lookup table. In this case, the second control unit 410 described later may derive a point (x, y) on an image corresponding to an estimated depth z. The camera parameter recording unit 408d may also record an approximation of a point (x, y) on an image corresponding to a depth z for each pixel of the image sensor 203. In this case, the second control unit 410 described later may derive a point (x, y) on an image corresponding to an estimated depth z.

[0081] The communication unit 409 transmits various information to the image diagnostic apparatus 5 or the trained model generating apparatus 1 under the control of the second control unit 410, and receives various information from the image diagnostic apparatus 5 or the trained model generating apparatus 1 and outputs it to the second control unit 410. Specifically, under the control of the second control unit 410, the communication unit 409 transmits a group of temporally continuous image data subjected to image processing by the image processing unit 406 to the image diagnostic apparatus 5, receives detection results including organs, parts, characteristic regions, etc. detected in the image diagnostic apparatus 5, and outputs them to the second control unit 410. In addition, under the control of the second control unit 410, the communication unit 409 receives a trained model received from the trained model generating apparatus 1 and outputs it to the trained model recording unit 408c. The communication unit 409 is configured using a communication module, etc.

[0082] The second control unit 410 corresponds to the second processor according to the present disclosure. The second control unit 410 is realized using a second processor, which is a processing device having hardware such as an FPGA or a CPU, and a memory, which is a temporary storage area used by the second processor. The second control unit 410 comprehensively controls each unit constituting the endoscope 2 and the control device 4. The second control unit 410 has a drive control unit 410a, an imaging control unit 410b, an acquisition unit 410c, an inference unit 410d, a calculation unit 410e, and an output control unit 410f.

[0083] The drive control unit 410a drives the focus adjustment mechanism 206 to move the focus lens L5 to the near point side or the far point end.

[0084] The imaging control unit 410b exposes the image sensor 203 to the subject image focused by the focus lens L5 at the near point end or the far point end, and captures the image. Furthermore, the imaging control unit 410b exposes the image sensor 203 to the subject image focused by the focus lens L5 during a moving period in which the focus lens L5 moves from the near point end to the far point end, and captures the image.

[0085] The acquisition unit 410c acquires a first image generated by the image sensor 203 capturing an image of a subject on the near point side of the focus lens L5. The acquisition unit 410c also acquires one or more of a plurality of second images generated by the image sensor 203 capturing an image of a subject during a period in which the focus lens L5 is moved between the near point side and the far point side by the focus adjustment mechanism 206 while changing the focal length. The acquisition unit 410c also acquires a third image generated by the image sensor 203 capturing an image of a subject on the far point side of the focus lens L5.

[0086] The inference unit 410d estimates a depth map of the subject using the first image, the second image, and the third image acquired by the acquisition unit 410c and the trained model recorded by the trained model recording unit 408c.

[0087] The calculation unit 410e estimates the shape of the subject based on the depth map of the subject estimated by the inference unit 410d and the camera parameters of the optical system 202 recorded by the camera parameter recording unit 408d.

[0088] The output control unit 410f outputs to the display device 3 the shape of the subject estimated by the calculation unit 410e.

[0089] [Processing by trained model generation device] Next, the process executed by the trained model generating device 1 will be described. FIG. 5 is a flowchart showing an outline of the process executed by the trained model generating device 1. Here, the subject distance is the distance from the tip of the optical system 101 to the subject K1. In the following, the subject will be described as a planar subject without any irregularities facing the optical system 101. The distance range of the planar subject to be trained is wider than the near end and far end at the focal position. For example, when the near end at the focal position is 4 mm and the far end is 20 mm, the distance of the subject to be trained will be described as 2 mm to 40 mm. This makes it possible to estimate the depth of a subject located at a near point or far point from the focal position.

[0090] 5, first, the trained model generation device 1 sets a subject (step S100). Specifically, the trained model generation device 1 sets a subject based on operation information input by the user via the input unit 13. In this case, in the case of an LCD monitor, the user inputs the type and content of the picture and the distance between the LCD monitor and the trained model generation device 1 using the input unit 13.

[0091] After that, the drive control unit 161 drives the focus adjustment mechanism 103 to move the focus lens L5 to the near point end of the optical system 101 (step S101).

[0092] Next, the imaging control unit 162 causes the imaging element 102 to capture an image at the near end of the focus lens L5 (step S102).

[0093] After that, the acquisition unit 163 acquires the first image generated by the image sensor 102 by capturing an image at the near point end of the focus lens L5 (step S103).

[0094] Next, the drive control unit 161 drives the focus adjustment mechanism 103 to move the focus lens L5 from the near point end to the far point end (step S104).

[0095] After that, the imaging control unit 162 causes the imaging element 102 to capture a plurality of images during the movement period in which the focus lens L5 moves from the near point end to the far point end (step S105).

[0096] Next, the acquisition unit 163 acquires a plurality of second images generated by the imaging element 102 capturing a plurality of images during the moving period in which the focus lens L5 moves from the near point end to the far point end (step S106). Note that the acquisition unit 163 acquires all of the plurality of second images generated by the imaging element 102 capturing a plurality of images during the moving period in which the focus lens L5 moves from the near point end to the far point end, but is not limited thereto, and it is sufficient to acquire one or more images during the moving period in which the focus lens L5 moves from the near point end to the far point end.

[0097] After that, the imaging control unit 162 causes the imaging element 102 to capture an image at the far end of the focus lens L5 (step S107).

[0098] Next, the acquisition unit 163 acquires a third image generated by the imaging element 102 capturing an image at the far end of the focus lens L5 (step S108).

[0099] Thereafter, the determination unit 164 determines whether or not a predetermined number of images required for learning have been acquired (step S109). Specifically, as shown in FIG. 6 to FIG. 9, the determination unit 164 determines whether or not the number of images acquired by capturing images at each of a plurality of focus positions for each of a plurality of subjects exceeds a predetermined number (for example, 10,000). If the determination unit 164 determines that the predetermined number of images required for learning have been acquired (step S109: Yes), the trained model generation device 1 proceeds to step S110, which will be described later. On the other hand, if the determination unit 164 determines that the predetermined number of images required for learning have not been acquired (step S109: No), the trained model generation device 1 returns to step S101.

[0100] In step S110, the acquisition unit 163 acquires a depth map for each subject from an external device or the recording unit 15 via the communication unit 14. A depth map (depth distribution) is obtained by adding a correct answer value of the distance from the optical system 101 to the subject to each pixel of each subject image. Of course, the acquisition unit 163 may acquire a depth map input by a user via the input unit 13.

[0101] Next, the learning unit 165 learns using a predetermined learning model and learning data that combines a plurality of images for each subject acquired by the acquisition unit 163, the plurality of images having different focus positions, and a depth map (step S111). Specifically, the learning unit 165 learns using a learning model generated by machine learning using artificial intelligence (AI (Artificial Intelligence)). The method of constructing the learning model used by the learning unit 165 is not particularly limited, and various machine learning methods such as deep learning using a neural network, a support vector machine, a decision tree, a simple Bayes method, and a k-nearest neighbor method can be used.

[0102] Fig. 6 is a diagram showing a model of a learning data set input to a learning model by the learning unit 165. As shown in Fig. 6, the learning unit 165 learns using a predetermined learning model and learning data that combines a depth map with a plurality of images FP1 to FP4, which are images FP1 to FP4 for each subject acquired by the acquisition unit 163 and have different focus positions, to generate a trained model. That is, the learning unit 165 generates a trained model that uses the first image, the second image, and the third image as input parameters and outputs distance information between the subject and the optical system 101 as output parameters.

[0103] In one embodiment, the link between the depth map and the image may be at least one of the first image at the near end and the second image at the far end among the multiple images. FIG. 7 is a diagram for explaining the link between the depth map and the image. In FIG. 7, the first image P100, the third image P101, and the second image P102, which have different focal positions (focal points) of the optical system 101, are explained. Furthermore, in FIG. 7, the frame W1 of the first image P100 is the angle of view corresponding to the third image P101, and the frame W2 is the angle of view corresponding to the second image P102. Furthermore, the area D1 is the area where the depth map can be estimated.

[0104] 7, in one embodiment, since the first image P100 and the third image P101 each contain a predetermined feature, for example, a characteristic pattern, it is possible to associate the first image P100 and the third image P101 with each other, and therefore it is possible to link the depth map to another image, for example, the second image P102. For this reason, in one embodiment, when linking the depth map, it is desirable to link it to an image with a wide angle of view, for example, the first image P100.

[0105] Returning to FIG. 5, the description of step S112 and subsequent steps will be continued. In step S102, the output control unit 166 outputs the trained model generated by the learning unit 165 to the trained model recording unit 153 (step S112). In this case, the output control unit 166 may output the trained model to an external device, for example, the endoscope system 100 described later, via the communication unit 14. After step S111, the trained model generating device 1 ends this process.

[0106] [Processing Executed by the Medical Support System] Next, a description will be given of the processing executed by the endoscope system 100. Fig. 8 is a flowchart showing an outline of the processing executed by the endoscope system 100.

[0107] 8, the second control unit 410 judges whether or not an instruction signal has been input from the operation unit 22 of the endoscope 2 to instruct the observation mode of the endoscope 2 to a measurement mode for measuring the shape of the subject (step S201). If the second control unit 410 judges that an instruction signal has been input from the operation unit 22 of the endoscope 2 to instruct the observation mode of the endoscope 2 to a measurement mode for measuring the shape of the subject (step S201: Yes), the endoscope system 100 proceeds to step S202, which will be described later. On the other hand, if the second control unit 410 judges that an instruction signal has not been input from the operation unit 22 of the endoscope 2 to instruct the observation mode of the endoscope 2 to a measurement mode for measuring the shape of the subject (step S201: No), the endoscope system 100 proceeds to step S204, which will be described later.

[0108] In step S202, the endoscope system 100 executes a measurement mode process for measuring the shape of a predetermined object in the subject. The measurement mode process will be described in detail later.

[0109] Next, the second control unit 410 determines whether or not an instruction signal to end observation by the endoscope 2 has been input from the operation unit 22 of the endoscope 2 (step S203). If the second control unit 410 determines that an instruction signal to end observation by the endoscope 2 has been input from the operation unit 22 of the endoscope 2 (step S203: Yes), the endoscope system 100 ends this process. On the other hand, if the second control unit 410 determines that an instruction signal to end observation by the endoscope 2 has not been input from the operation unit 22 of the endoscope 2 (step S203: No), the endoscope system 100 returns to the above-mentioned step S201.

[0110] In step S204, the endoscope system 100 performs normal observation of the subject. Specifically, the second control unit 410 controls the light source control unit 404 to cause the first light source unit 402 or the second light source unit 403 to emit light, and causes the image sensor 203 to capture reflected light from the subject and generate image data. Then, the second control unit 410 causes the image processing unit 406 to perform image processing on the image data generated by the image sensor 203 and display the image on the display device 3, thereby performing normal observation of the subject. After step S204, the endoscope system 100 proceeds to step S203.

[0111] [Overview of measurement mode processing] Next, the measurement mode process described in step S202 of Fig. 8 will be described in detail. Fig. 9 is a flowchart showing an outline of the measurement mode process. Fig. 10 is a timing chart showing the position of the focus lens L5 on the optical axis O1 of the endoscope 2 and the exposure timing of the image sensor 203 during the measurement mode process. In Fig. 10, (a) of Fig. 10 shows the position of the focus lens L5, and (b) of Fig. 10 shows the exposure timing of the image sensor 203.

[0112] As shown in FIG. 10, first, the drive control unit 410a controls the focus adjustment mechanism 206 to stop the AF drive by the focus adjustment mechanism 206 (step S301).

[0113] Next, the drive control unit 410a drives the focus adjustment mechanism 206 to move the focus lens L5 toward the near point side (step S302), and stops the focus lens L5 at the near point end (step S303). Specifically, as shown in (a) of FIG. 10, the drive control unit 410a drives the focus adjustment mechanism 206 to move the focus lens L5 toward the near point side, and stops the focus lens L5 at the near point end.

[0114] After that, the imaging control unit 410b exposes the image sensor 203 to the subject image focused by the focus lens L5 at the near point end and captures the image (step S304). Specifically, as shown in Fig. 10 and Fig. 11, the imaging control unit 410b exposes the image sensor 203 to the subject image of the subject K1 focused by the focus lens L5 at the near point end and captures the image. In this case, as shown in Fig. 12, the acquisition unit 410c acquires the image data generated by the image sensor 203 as a first image P200.

[0115] Next, the drive control unit 410a drives the focus adjustment mechanism 206 to move the focus lens L5 to the far point side (step S302).

[0116] After that, the imaging control unit 410b causes the imaging element 203 to expose and capture a subject image according to a predetermined frame rate during the movement period in which the focus lens L5 moves from the near point end to the far point end (step S306). Specifically, as shown in (b) of Fig. 10, the imaging control unit 410b causes the imaging element 203 to expose and capture a subject image according to a predetermined frame rate, for example, 60 FPS, during the movement period in which the focus lens L5 moves from the near point end to the far point end, thereby generating multiple image data. In this case, as shown in Fig. 13, the acquisition unit 410c acquires one or more of the multiple image data generated by the imaging element 203 as the second image P201.

[0117] Next, the drive control unit 410a drives the focus adjustment mechanism 206 to stop the focus lens L5 at the far point end (step S307). Specifically, as shown in (a) of FIG. 10, the drive control unit 410a drives the focus adjustment mechanism 206 to stop the focus lens L5 at the far point end.

[0118] Thereafter, the imaging control unit 410b causes the imaging element 203 to expose and capture the subject image focused by the focus lens L5 at the far point end (step S308). Specifically, as shown in Fig. 10(b), the imaging control unit 410b causes the imaging element 203 to expose and capture the subject image focused by the focus lens L5 at the far point end. In this case, as shown in Fig. 14, the acquisition unit 410c acquires the image data generated by the imaging element 203 as a third image P202.

[0119] Next, the drive control unit 410a controls the focus adjustment mechanism 206 to move the focus lens L5 to the AF drive stop position (step S309), and starts the AF drive by the focus adjustment mechanism 206 (step S310).

[0120] Thereafter, the inference unit 410d estimates a depth map of the subject using the first image, the second image, and the third image acquired by the acquisition unit 410c and the trained model recorded by the trained model recording unit 408c (step S311). Specifically, the inference unit 410d inputs the first image, the second image, and the third image acquired by the acquisition unit 410c to the trained model as input parameters, and estimates a depth map of the subject that is output by the trained model as an output parameter.

[0121] Next, the calculation unit 410e estimates the shape of the subject based on the depth map of the subject estimated by the inference unit 410d and the camera parameters of the optical system 202 recorded by the camera parameter recording unit 408d (step S312).

[0122] Fig. 15 is a diagram for explaining a model of an estimation method for estimating the shape of a subject by the calculation unit 410e. As shown in Fig. 15, the calculation unit 410e determines that the following formula (1) holds when the position of a predetermined pixel in the image sensor 203 is x_imager, the angle at which the optical axis O1 of the optical system 202 intersects with a line connecting the position Q1 of the predetermined subject and the position x of the image sensor 203 is θ, and the distance from the optical system 202 to the position Q1 of the predetermined subject based on the depth map is z. x = z × tan θ (1) The calculation unit 410e estimates the shape of the subject using the above formula (1) for each pixel of the image sensor 203.

[0123] After that, the output control unit 410f outputs the shape of the subject estimated by the calculation unit 410e to the display device 3 (step S313). Specifically, as shown in Fig. 16, the output control unit 410f outputs an image P203 corresponding to the shape of the subject estimated by the calculation unit 410e. After step S313, the endoscope system 100 returns to the main routine of Fig. 8.

[0124] According to the embodiment described above, the inference unit 410d estimates a depth map of the subject using the first image, the second image, and the third image acquired by the acquisition unit 410c and the learned model recorded by the learned model recording unit 408c, thereby improving the detection accuracy of the distance from the optical system to the subject, which is the object to be measured.

[0125] Furthermore, according to one embodiment, the calculation unit 410e estimates the shape of the subject based on the depth map of the subject estimated by the inference unit 410d and the camera parameters of the optical system 202 recorded by the camera parameter recording unit 408d, thereby further improving the estimation accuracy of the three-dimensional shape of the subject, which is the measurement object.

[0126] In one embodiment, during the period in which the focus lens L5 moves from the near end to the far end, the acquisition unit 163 acquires a plurality of second images that constitute the moving image generated by the image sensor 102. However, this is not limited to this, and for example, focal sweep images may be used instead of frames that constitute the moving image.

[0127] In addition, in one embodiment, the control device 4 is provided with the acquisition unit 410c, the inference unit 410d, the calculation unit 410e, and the output control unit 410f, but the present invention is not limited to this, and the functions of the inference unit 410d, the calculation unit 410e, and the output control unit 410f may be provided in the image diagnostic device 5. In one embodiment, the functions of the inference unit 410d, the calculation unit 410e, and the output control unit 410f may be provided in an external server or the like, other than the image diagnostic device 5.

[0128] (Variation 1) Next, a first modified example of the embodiment will be described. The first modified example has the same configuration as the endoscope system 100 according to the embodiment described above, but executes different processes. Specifically, in the embodiment described above, the image sensor 203 is exposed to light while the focus lens L5 is stopped at the near point end and the far point end to generate the first and third images, but in the first modified example, the image sensor 203 is exposed to light at the timing when the focus lens L5 reaches the near point end and the far point end to generate the first and third images. Therefore, the process executed by the endoscope system 100 according to the first modified example of the embodiment will be described below.

[0129] Fig. 17 is a timing chart showing the position of the focus lens L5 on the optical axis O1 of the endoscope 2 during measurement mode processing and the exposure timing of the image sensor 203. In Fig. 17, (a) of Fig. 17 shows the position of the focus lens L5, and (b) of Fig. 17 shows the exposure timing of the image sensor 203.

[0130] As shown in Fig. 17, the drive control unit 410a drives the focus adjustment mechanism 206 to move the focus lens L5 from the current position toward the near point side, and after reaching the near point end, moves the focus lens L5 linearly toward the far point side. In this case, the imaging control unit 410b exposes the image sensor 203 to an object image focused by the focus lens L5 at the near point end, and captures the image. At this time, the acquisition unit 410c acquires the image data generated by the image sensor 203 as a first image.

[0131] Next, the drive control unit 410a moves the focus lens L5 to the AF stop position after the focus lens L5 arrives at the far end. In this case, the imaging control unit 410b causes the imaging element 203 to capture an image of a subject condensed during the movement of the focus lens L5 from the near end to the far end by exposing the image to the imaging element 203. At this time, the acquisition unit 410c acquires the image data generated by the imaging element 203 as a second image. Furthermore, the imaging control unit 410b causes the imaging element 203 to capture an image of a subject condensed at the far end by exposing the image to the imaging element 203. At this time, the acquisition unit 410c acquires the image data generated by the imaging element 203 as a third image.

[0132] In this way, the drive control unit 410a linearly moves the focus lens L5 from the near point end to the far point end, thereby shortening the time required to return from the measurement mode to the normal mode.

[0133] According to the above-described first modification, since a depth map of the subject is estimated, it is possible to improve the detection accuracy of the distance from the optical system to the subject, which is the measurement target.

[0134] (Variation 2) Next, a second modification of the embodiment will be described. The second modification has the same configuration as the endoscope system 100 according to the embodiment described above, but executes different processes. Therefore, the process executed by the endoscope system 100 according to the first modification of the embodiment will be described below.

[0135] Fig. 18 is a timing chart showing the position on the optical axis O1 of the focus lens L5 of the endoscope 2 during measurement mode processing and the exposure timing of the image sensor 203. Note that in Fig. 18, (a) of Fig. 18 shows the position of the focus lens L5, and (b) of Fig. 18 shows the exposure timing of the image sensor 203.

[0136] As shown in Fig. 22, the drive control unit 410a drives the focus adjustment mechanism 206 to move the focus lens L5 from the current position toward the near point side, and stops at the near point end within a certain time. After reaching the near point end, the drive control unit 410a moves the focus lens L5 toward the far point side. In this case, the imaging control unit 410b exposes the image sensor 203 to an object image focused by the focus lens L5 at the near point end, and captures the image. At this time, the acquisition unit 410c acquires the image data generated by the image sensor 203 as a first image.

[0137] Next, the drive control unit 410a moves the focus lens L5 to the far end so that it stops within a certain time, and after arriving at the far end, moves the focus lens L5 to the AF stop position. In this case, the imaging control unit 410b causes the imaging element 203 to capture an image of a subject condensed by the focus lens L5 while it is moving from the near end to the far end. At this time, the acquisition unit 410c acquires the image data generated by the imaging element 203 as a second image. Furthermore, the imaging control unit 410b causes the imaging element 203 to capture an image of a subject condensed by the focus lens L5 at the far end by exposing it to the image. At this time, the acquisition unit 410c acquires the image data generated by the imaging element 203 as a third image.

[0138] In this way, the drive control unit 410a smoothly moves the focus lens L5 from the near point end to the far point end in a sinusoidal manner, thereby making it possible to obtain the first image and the third image in which blurring caused by slight movement of the focus lens L5 is suppressed.

[0139] According to the above-described second modification, a depth map of the subject is estimated, so that it is possible to improve the detection accuracy of the distance from the optical system to the subject, which is the measurement target.

[0140] (Other embodiments) Various inventions can be formed by appropriately combining a plurality of components disclosed in the endoscope system according to the embodiment of the present disclosure described above. For example, some components may be deleted from all the components described in the endoscope system according to the embodiment of the present disclosure described above. Furthermore, the components described in the endoscope system according to the embodiment of the present disclosure described above may be appropriately combined.

[0141] Furthermore, in the endoscope system according to an embodiment of the present disclosure, the above-described "unit" can be read as "means" or "circuit," etc. For example, a control unit can be read as control means or a control circuit.

[0142] In addition, a program to be executed by an endoscopic system according to one embodiment of the present disclosure is provided as file data in an installable or executable format recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, a digital versatile disk (DVD), a USB medium, or a flash memory.

[0143] In addition, the program to be executed by the endoscopic system according to one embodiment of the present disclosure may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network.

[0144] In the description of the flowcharts in this specification, the order of processing between steps is clearly indicated using expressions such as "first," "then," and "continue," but the order of processing required to implement the present invention is not uniquely determined by these expressions. In other words, the order of processing in the flowcharts described in this specification can be changed as long as there is no contradiction.

[0145] Although some of the embodiments of the present application have been described in detail above with reference to the drawings, these are merely examples, and the present invention can be embodied in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the forms described in the disclosure of the present invention. [Explanation of symbols]

[0146] 1. Trained model generation device 2 Endoscope 3 Display device 4. Control device 5. Diagnostic imaging equipment 11 Imaging device 12 Display section 13,407 Input section 14,409 Communications Department 15,408 Recording section 16 First control section 100 Endoscope system 101,202 Optical system 102,203 Image sensor 103,206 Focus adjustment mechanism 103a, 206a Focus drive unit 103b, 206b Detection section 151,408a Program Recording Section 152 Learning Model Recording Section 153,408b Trained model record section 161,410a Drive control unit 162,207,410b Imaging control unit 163,410c acquisition department 164 Judgment section 165 Learning Department 166,410f Output control section 201 Illumination optical system 408b Image data recording unit 408d Camera parameter recording section 410 Second control section 410d Reasoning Department 410e Calculation section 410f Output control unit L1~L4, L6~L10 lenses L5 Focus Lens

Claims

1. A control device including a processor, The processor, An imaging device comprising: an optical system capable of changing a focal length; an image sensor that generates a still image by exposing an image formed by the optical system; and a focus adjustment mechanism that changes the focal length by moving the optical system along an optical axis direction of the optical system, a first image generated by the imaging element capturing an image of a subject on the near point side of the optical system, and a second image generated by the imaging element capturing an image of the subject during a period in which the optical system is moved between the near point side and the far point side by the focus adjustment mechanism while changing a focal length; a third image generated by the imaging element capturing an image of the subject on the far point side of the optical system; and Get Inferring a depth map of the subject by inputting the first image, the second image, and the third image as input parameters into a trained model that outputs a depth map indicating a distance from the optical system to a subject as an output parameter; outputting a depth map of the object; Control device.

2. The control device according to claim 1 , The second image is At least one frame image in a moving image formed by a group of images successive in time, captured by the image sensor during a period in which the focus adjustment mechanism changes the focal length between the near point side and the far point side. Control device.

3. The control device according to claim 2, The processor, Obtaining camera parameters of the optical system; estimating a shape of the object based on the camera parameters and a depth map of the object; Control device.

4. The control device according to claim 3, The processor, controlling the focus adjustment mechanism so as to stop the optical system at each of the near point end and the far point end; Control device.

5. The control device according to claim 3, The processor, Controlling the focus adjustment mechanism so as to move the optical system linearly from a near point end to a far point end. Control device.

6. The control device according to claim 3, The processor, controlling the focus adjustment mechanism so as to stop the optical system at each of the near point end and the far point end within a certain period of time; Control device.

7. A trained model generation device including a processor, The processor, An imaging device comprising: an optical system capable of changing a focal length; an image sensor that generates a still image by exposing an image formed by the optical system; and a focus adjustment mechanism that changes the focal length by moving the optical system along an optical axis direction of the optical system, a plurality of first images generated by the imaging element capturing images of a plurality of objects on the near point side of the optical system, and a plurality of second images generated by the imaging element capturing images of the plurality of objects during a period in which the optical system is moved between the near point side and the far point side by the focus adjustment mechanism while changing a focal length; a plurality of third images generated by the imaging element capturing images of the plurality of objects on the far point side of the optical system; and A depth map correct value relating to a distance from the optical system to each of the plurality of objects; to each of the plurality of objects, By learning using the learning data, a trained model is generated that uses the first image, the second image, and the third image as input parameters and outputs a depth map indicating a distance from the optical system to a subject as an output parameter. Trained model generator.

8. an endoscope capable of imaging the inside of a subject's body; a control device including a processor and to which the endoscope can be connected; Equipped with The endoscope includes: An optical system capable of changing a focal length; an image sensor that exposes the image formed by the optical system to generate a still image; a focus adjustment mechanism that changes the focal length by moving the optical system along an optical axis of the optical system; Equipped with The processor, From the endoscope, a first image generated by the imaging element capturing an image of a subject on the near point side of the optical system, and a second image generated by the imaging element capturing an image of the subject during a period in which the optical system is moved between the near point side and the far point side by the focus adjustment mechanism while changing a focal length; a third image generated by the imaging element capturing an image of the subject on the far point side of the optical system; and Get Inferring a depth map of the subject by inputting the first image, the second image, and the third image as input parameters into a trained model that outputs a depth map indicating a distance from the optical system to a subject as an output parameter; outputting a depth map of the object; Endoscopy system.

9. A medical support method executed by a control device having a processor, comprising: The processor, An imaging device comprising: an optical system capable of changing a focal length; an image sensor that generates a still image by exposing an image formed by the optical system; and a focus adjustment mechanism that changes the focal length by moving the optical system along an optical axis direction of the optical system, a first image generated by the imaging element capturing an image of a subject on the near point side of the optical system, and a second image generated by the imaging element capturing an image of the subject during a period in which the optical system is moved between the near point side and the far point side by the focus adjustment mechanism while changing a focal length; a third image generated by the imaging element capturing an image of the subject on the far point side of the optical system; and Get Inferring a depth map of the subject by inputting the first image, the second image, and the third image as input parameters into a trained model that outputs a depth map indicating a distance from the optical system to a subject as an output parameter; outputting a depth map of the object; Including, Medical assistance methods.

10. A learning method executed by a trained model generation device having a processor, The processor, An imaging device comprising: an optical system capable of changing a focal length; an image sensor that generates a still image by exposing an image formed by the optical system; and a focus adjustment mechanism that changes the focal length by moving the optical system along an optical axis direction of the optical system, a plurality of first images generated by the imaging element capturing images of a plurality of objects on the near point side of the optical system, and a plurality of second images generated by the imaging element capturing images of the plurality of objects during a period in which the optical system is moved between the near point side and the far point side by the focus adjustment mechanism while changing a focal length; a plurality of third images generated by the imaging element capturing images of the plurality of objects on the far point side of the optical system; and A depth map correct value relating to a distance from the optical system to each of the plurality of objects; to each of the plurality of objects, By learning using the learning data, a trained model is generated that uses the first image, the second image, and the third image as input parameters and outputs a depth map indicating a distance from the optical system to a subject as an output parameter. Including, How to learn.

11. A program executed by a control device having a processor, The processor, An imaging device comprising: an optical system capable of changing a focal length; an image sensor that generates a still image by exposing an image formed by the optical system; and a focus adjustment mechanism that changes the focal length by moving the optical system along an optical axis direction of the optical system, a first image generated by the imaging element capturing an image of a subject on the near point side of the optical system, and a second image generated by the imaging element capturing an image of the subject during a period in which the optical system is moved between the near point side and the far point side by the focus adjustment mechanism while changing a focal length; a third image generated by the imaging element capturing an image of the subject on the far point side of the optical system; and Get Inferring a depth map of the subject by inputting the first image, the second image, and the third image as input parameters into a trained model that outputs a depth map indicating a distance from the optical system to a subject as an output parameter; outputting a depth map of the object; program.

12. A program executed by a trained model generation device having a processor, The processor, An imaging device comprising: an optical system capable of changing a focal length; an image sensor that generates a still image by exposing an image formed by the optical system; and a focus adjustment mechanism that changes the focal length by moving the optical system along an optical axis direction of the optical system, a plurality of first images generated by the imaging element capturing images of a plurality of objects on the near point side of the optical system, and a plurality of second images generated by the imaging element capturing images of the plurality of objects during a period in which the optical system is moved between the near point side and the far point side by the focus adjustment mechanism while changing a focal length; a plurality of third images generated by the imaging element capturing images of the plurality of objects on the far point side of the optical system; and A depth map correct value relating to a distance from the optical system to each of the plurality of objects; to each of the plurality of objects, By learning using the learning data, a trained model is generated that uses the first image, the second image, and the third image as input parameters and outputs a depth map indicating a distance from the optical system to a subject as an output parameter. program.

Images