Control device, learned model generation device, learned model generation method, and medical assistance method

Abstract

The invention provides a control device, a learned model generation device, a learned model generation method, and a medical assistance method, which can further improve the detection accuracy of the distance from an optical system to a measurement object such as a subject. A depth map of a subject is derived by inputting a first image, a second image, and a third image as input parameters to a learned model that outputs a depth map representing a distance from an optical system to the subject as output parameters, and the depth map of the subject is output.

Description

Technical Field

[0001] The present invention relates to a control device, a learned model generation device, a medical assistance method, a learned model generation method and a recording medium. Background Art

[0002] Conventionally, in an imaging device such as a digital camera, there is a known technique for estimating the distance between a subject and the imaging device based on the blur degree of a captured image. For example, in Patent Document 1, a second image affected by the aberration of an optical system is input into a statistical model, and distance information indicating the distance to the subject in the second image is obtained. The statistical model is generated by learning blur generated in a first image affected by the aberration of the optical system, and the blur of the first image changes nonlinearly according to the distance to the subject.

[0003] In addition, in Patent Document 2, based on the optical coefficient values ​​of the camera device including the focal length of the lens and the blur amount of each image area in the scanned image captured by exposing the image sensor while continuously moving the focal length of the lens from the near end to the far end, relative to the all-focus image, the distance between the camera device and the subject in each image area in the scanned image is calculated, and a depth map (distance map) is generated in which the distance is represented by the pixel value in each image area.

[0004] [Prior art literature]

[0005] [Patent Document]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2020-148483

[0007] [Patent Document 2] Japanese Patent No. 5868183 Summary of the invention

[0008] [Problems to be Solved by the Invention]

[0009] However, in the above-mentioned References 1 and 2, since the data captured during focus driving is not effectively used, there is room for further improvement in the accuracy of detecting the distance from the optical system to the measurement object such as the subject.

[0010] The present invention is completed in view of the above situation, and its purpose is to provide a control device, a learned model generation device, a medical assistance method, a learned model generation method and a recording medium that can further improve the detection accuracy of the distance from an optical system to a measurement object such as a subject.

[0011] [Methods for solving the problem]

[0012] In order to solve the above-mentioned problems and achieve the purpose, a control device of the present invention is a control device having a processor, the processor obtaining a first image, a second image and a third image from a camera device, the processor deriving a depth map of a subject by inputting the first image, the second image and the third image as input parameters into a learned model, the processor outputting the depth map of the subject, the depth map indicating the distance from an optical system to the subject, the learned model being a model that outputs the depth map as an output parameter, and the camera device having: the optical system capable of changing the focal length; An imaging element that generates a static image by exposing an image formed by the optical system; and a focus adjustment mechanism that causes the optical system to move along the optical axis direction of the optical system to change the focal length, wherein the first image is an image generated by the imaging element photographing the subject at the near side of the optical system, the second image is an image generated by the imaging element photographing the subject while the optical system is moved between the near side and the far side by changing the focal length through the focus adjustment mechanism, and the third image is an image generated by the imaging element photographing the subject at the far side of the optical system.

[0013] Furthermore, in the control device of the present invention in the above invention, the second image is at least one frame image in a dynamic image consisting of a group of images captured continuously in time by the camera element during the period when the focus adjustment mechanism changes the focal length between the near side and the far side.

[0014] In the control device of the present invention disclosed above, the processor acquires camera parameters of the optical system, and estimates the shape of the object based on the camera parameters and a depth map of the object.

[0015] Furthermore, in the control device of the present invention in the above invention, the processor controls the focus adjustment mechanism so that the optical system stops at the near end and the far end, respectively.

[0016] Furthermore, in the control device of the present invention, in the above invention, the processor controls the focus adjustment mechanism so that the optical system moves linearly from the near end to the far end.

[0017] Furthermore, in the control device of the present invention, in the above invention, the processor controls the focus adjustment mechanism so that the optical system stops at the near end and the far end for a certain period of time respectively.

[0018] In addition, the learned model generation device according to the present invention is a learned model generation device including a processor, wherein the processor acquires learning data from an imaging device, generates a learned model by learning using the learning data,

[0019] The imaging device includes: an optical system capable of changing focal length; an imaging element that generates a static image by exposing an image formed by the optical system; and a focus adjustment mechanism that changes the focal length by moving the optical system along the optical axis direction of the optical system, wherein the learning data is data in which a plurality of first images, a plurality of second images, a plurality of third images, and correct solution values ​​of depth maps related to respective distances from the optical system to a plurality of objects are combined with the plurality of objects, the plurality of first images are images generated by the imaging element capturing the plurality of objects respectively on the near side of the optical system, the plurality of second images are images generated by the imaging element capturing the plurality of objects respectively while the optical system is moved between the near side and the far side by the focus adjustment mechanism, and the plurality of third images are images generated by the imaging element capturing the plurality of objects respectively on the far side of the optical system, and the learned model is a model that uses the first image, the second image, and the third image as input parameters and outputs a depth map representing the distance from the optical system to the subject as an output parameter.

[0020] Furthermore, the endoscope system involved in the present invention comprises: an endoscope capable of photographing the inside of a subject's body; and a control device, the control device comprising a processor, which can be connected to the endoscope, the endoscope comprising: an optical system capable of changing the focal length; a camera element, which exposes an image formed by the optical system to generate a static image; and a focus adjustment mechanism, which moves the optical system along the optical axis direction of the optical system to change the focal length, the processor obtains a first image, a second image, and a third image from the endoscope, and inputs the first image, the second image, and the third image as input parameters into a learned model model to derive a depth map of the subject and output the depth map of the subject, wherein the depth map represents the distance from the optical system to the subject, the learned model is a model that outputs the depth map as an output parameter, the first image is an image generated by the imaging element on the near side of the optical system photographing the subject, the second image is an image generated by the imaging element photographing the subject while the optical system is moved between the near side and the far side by changing the focal length through the focus adjustment mechanism, and the third image is an image generated by the imaging element on the far side of the optical system photographing the subject.

[0021] In addition, the medical assistance method involved in the present invention is a medical assistance method executed by a control device having a processor, and the medical assistance method includes the following steps: the processor obtains a first image, a second image, and a third image from a camera device; the processor derives a depth map of the subject by inputting the first image, the second image, and the third image as input parameters into a learned model; and the processor outputs the depth map of the subject, the depth map representing the distance from the optical system to the subject, the learned model is a model that outputs the depth map as an output parameter, and the camera device includes: the optical system , which can change the focal length; a camera element, which generates a static image by exposing the image formed by the optical system; and a focus adjustment mechanism, which changes the focal length by moving the optical system along the optical axis direction of the optical system, the first image is an image generated by the camera element photographing the subject on the near side of the optical system, the second image is an image generated by the camera element photographing the subject during the period when the optical system is moved between the near side and the far side by changing the focal length through the focus adjustment mechanism, and the third image is an image generated by the camera element photographing the subject on the far side of the optical system.

[0022] In addition, the learning method involved in the present invention is a learning method performed by a learned model generating device having a processor, the learning method comprising the following steps: the processor obtains learning data from a camera device; and the processor generates a learned model by learning using the learning data, the camera device comprising: an optical system capable of changing the focal length; an imaging element that exposes an image formed by the optical system to generate a static image; and a focus adjustment mechanism that moves the optical system along the optical axis direction of the optical system to change the focal length, the learning data being a positive image of a plurality of first images, a plurality of second images, a plurality of third images, and a depth map related to the distance from the optical system to each of a plurality of objects. The plurality of first images are images generated by respectively photographing the plurality of objects by the imaging element at the near side of the optical system, the plurality of second images are images generated by respectively photographing the plurality of objects by the imaging element during a period in which the optical system is moved between the near side and the far side by changing the focal length through the focus adjustment mechanism, the plurality of third images are images generated by respectively photographing the plurality of objects by the imaging element at the far side of the optical system, and the learned model is a model that takes the first images, the second images, and the third images as input parameters, and outputs a depth map representing the distance from the optical system to the subject as an output parameter.

[0023] Furthermore, the recording medium according to the present invention is a non-transitory computer-readable recording medium having recorded thereon an executable program, the program executing the following processing: acquiring a first image, a second image, and a third image from an imaging device;

[0024] A depth map of a subject is derived by inputting the first image, the second image, and the third image as input parameters into a learned model; and the depth map of the subject is output, wherein the depth map represents the distance from the optical system to the subject. The learned model is a model that outputs the depth map as an output parameter. The imaging device comprises: an optical system capable of changing the focal length; an imaging element that generates a static image by exposing an image formed by the optical system; and a focus adjustment mechanism that changes the focal length by moving the optical system along the optical axis of the optical system. The first image is an image generated by the imaging element photographing the subject on the near side of the optical system. The second image is an image generated by the imaging element photographing the subject while the optical system is moving between the near side and the far side by changing the focal length through the focus adjustment mechanism. The third image is an image generated by the imaging element photographing the subject on the far side of the optical system.

[0025] In addition, the recording medium involved in the present invention is a non-temporary computer-readable recording medium that records an executable program, and the program performs the following processing: obtaining learning data from a camera device; and generating a learned model by learning using the learning data, the camera device comprising: an optical system that can change the focal length; a camera element that exposes the image formed by the optical system to generate a static image; and a focus adjustment mechanism that moves the optical system along the optical axis direction of the optical system to change the focal length, the learning data is a combination of a plurality of first images, a plurality of second images, a plurality of third images, and a correct solution value of a depth map related to the distance from the optical system to each of the plurality of objects and the The plurality of first images are images generated by respectively photographing the plurality of objects by the imaging element on the near side of the optical system, the plurality of second images are images generated by respectively photographing the plurality of objects by the imaging element during a period in which the optical system is moved between the near side and the far side by changing the focal length through the focus adjustment mechanism, the plurality of third images are images generated by respectively photographing the plurality of objects by the imaging element on the far side of the optical system, and the learned model is a model that takes the first images, the second images, and the third images as input parameters, and outputs a depth map representing the distance from the optical system to the subject as an output parameter.

[0026] [Effects of the invention]

[0027] According to the present invention, there is an effect that the estimation accuracy of the three-dimensional shape of the measurement object can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 This is a block diagram showing the functional structure of a learned model generation device according to one embodiment.

[0029] Figure 2 This is a diagram schematically illustrating the movement of a focus lens in an optical system of a learned model generation device according to an embodiment.

[0030] Figure 3 It is a diagram showing the overall configuration of an endoscope system according to one embodiment.

[0031] Figure 4 This is a block diagram showing the functional configuration of main parts of an endoscope and a control device according to one embodiment.

[0032] Figure 5 This is a flowchart showing an overview of processing executed by a learned model generation device according to one embodiment.

[0033] Figure 6 This is a diagram schematically showing a learning data set input to a learning model by a learning unit according to one embodiment.

[0034] Figure 7 A diagram schematically illustrating the relationship between a depth map and an image.

[0035] Figure 8 This is a flowchart showing an overview of processing executed by the endoscope system according to one embodiment.

[0036] Fig. 9 Yes means Figure 8 Flowchart showing an overview of the measurement mode processing in .

[0037] Fig.10 This is a timing chart showing the position of a focus lens included in an endoscope on the optical axis and the exposure timing of an imaging element during measurement mode processing executed by an endoscope system according to one embodiment.

[0038] Fig.11 This is a diagram showing an example of a subject photographed by an endoscope.

[0039] Fig.12 FIG. 1 is a diagram showing an example of image data captured by an endoscope.

[0040] Fig.13 This is a diagram showing another example of image data captured by an endoscope.

[0041] Fig.14This is a diagram showing another example of image data captured by an endoscope.

[0042] Fig.15 FIG. 1 is a diagram schematically illustrating a method for estimating the shape of a subject by a calculation unit.

[0043] Fig.16 This is a diagram showing an example of an image corresponding to the shape of a subject.

[0044] Fig.17 This is a timing chart showing the position of a focus lens included in an endoscope on the optical axis and the exposure timing of an imaging element during measurement mode processing executed by an endoscope system according to Modification 1 of one embodiment.

[0045] Fig.18 This is a timing chart showing the position of a focus lens included in an endoscope on the optical axis and the exposure timing of an imaging element during measurement mode processing executed by an endoscope system according to Modification 1 of one embodiment.

[0046] Description of symbols

[0047] 1Learned model generation device

[0048] 2 Endoscope

[0049] 3 Display device

[0050] 4 Control device

[0051] 5. Image diagnostic device

[0052] 11 Camera Device

[0053] 12 Display unit

[0054] 13.407 Input section

[0055] 14. 409 Department of Communications

[0056] 15. 408 Records Department

[0057] 16First control unit

[0058] 100 Endoscope System

[0059] 101, 202 Optical System

[0060] 102, 203 camera element

[0061] 103, 206 focus adjustment mechanism

[0062] 103a, 206a Focus driving unit

[0063] 103b, 206b detection unit

[0064] 151, 408a Program Recording Department

[0065] 152 Learning Model Recording Department

[0066] 153, 408b Learned model record section

[0067] 161, 410a drive control unit

[0068] 162, 207, 410b Camera control unit

[0069] 163, 410c Acquisition Department

[0070] 164 Judgment Department

[0071] 165 Learning Department

[0072] 166, 410f output control unit

[0073] 201 Illumination Optical System

[0074] 408b Image data recording unit

[0075] 408d Camera parameter recording unit

[0076] 410 Second Control Unit

[0077] 410d Derivation

[0078] 410e Computing Department

[0079] 410f Output control unit

[0080] L1~L4、L6~L10 lens

[0081] L5 Focus Lens DETAILED DESCRIPTION

[0082] Hereinafter, the learned model generation device and the endoscope system involved in the present invention will be described in detail with reference to the accompanying drawings. In addition, the present invention is not limited to the following embodiments. In addition, the various figures referred to in the following description are merely schematic illustrations of shapes, sizes, and positional relationships to the extent that the contents of the present invention can be understood. That is, the present invention is not limited only to the shapes, sizes, and positional relationships illustrated in the various figures. Moreover, in the description of the accompanying drawings, the same parts are marked with the same reference numerals for description. In addition, hereinafter, after describing the structure of the learned model generation device of the present invention, the structure of the endoscope system will be described.

[0083] [Structure of the learned model generation device]

[0084] Figure 1 This is a block diagram showing the functional structure of a learned model generation device according to one embodiment. Figure 1 The learned model generating device 1 shown is a device that generates a learned model by inputting into a prescribed learning model learning data of multiple combinations of images that correspond to each other in focus positions (focal points) for each subject and a depth map (correct answer value) representing the distance between the subject and the optical system for each pixel.

[0085] Figure 1 The learned model generating device 1 shown includes an imaging device 11 , a display unit 12 , an input unit 13 , a communication unit 14 , a recording unit 15 , and a first control unit 16 .

[0086] The imaging device 11 generates image data (RAW data) of a subject image by collecting and receiving light from a predetermined field of view under the control of the first control unit 16, and outputs the image data to the first control unit 16. The imaging device 11 includes an optical system 101, an imaging element 102, and a focus adjustment mechanism 103.

[0087] The optical system 101 is composed of lenses L1 to L10 and an aperture Di. The optical system 101 forms a subject image on the light receiving surface of the imaging element 102. In addition, the optical system 101 includes a focus lens L5 that can move along the optical axis O1 direction among the lenses L1 to L10. The focus lens L5 moves forward and backward along the optical axis O1 direction by a focus adjustment mechanism 103 described later.

[0088] The imaging element 102 generates image data by receiving the subject image formed by the optical system 101. The imaging element 102 is configured using an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor.

[0089] The focus adjustment mechanism 103 changes the focusing position (focus) of the optical system 101 by moving the focus lens L5 of the optical system 101 along the optical axis O1 under the control of the first control unit 16. The focus adjustment mechanism 103 includes a focus driving unit 103a and a detection unit 103b.

[0090] The focus driving unit 103 a is configured using an actuator including a stepping motor or a voice coil motor, for example, and is driven under the control of the first control unit 16 to move the focus lens L5 to a predetermined position on the optical axis O1 .

[0091] The detection unit 103b detects focal length information related to the position or focal length of the focus lens L5 in the optical system 101, and outputs the detection result to the first control unit 16. The detection unit 103b is configured using, for example, an encoder or an optical coupler. In addition, the detection unit 103b may also detect the position of the focus lens L5 on the optical axis O1 based on the driving amount of the focus driving unit 103a.

[0092] Figure 2 1 is a diagram schematically illustrating the movement of the focus lens L5 in the optical system 101. Figure 2 As shown in (a) and (b) of FIG. 1 , the focus driving unit 103a, under the control of the first control unit 16, drives the focus lens L5 to move to a predetermined position on the optical axis O1, thereby changing the focal position P1 (focus) of the optical system 101 ( Figure 2 (a) → Figure 2 (b)). Thus, the optical system 101 can change the focusing position P1 from the far end to the near end.

[0093] return Figure 1 , continue to explain the structure of the learned model generating device 1.

[0094] The display unit 12 displays a display image corresponding to various information and image data input from the first control unit 16. The display unit 12 is configured using an organic EL display (Organic Electroluminescent Display) or a liquid crystal display or the like.

[0095] The input unit 13 receives input of various operations by the user, and outputs information corresponding to the received various operations to the first control unit 16. The input unit 13 is configured using an input interface such as a keyboard, a mouse, or a touch panel.

[0096] The communication unit 14 transmits various information input from the first control unit 16 to the outside, and outputs various information received from the outside to the first control unit 16. The communication unit 14 is configured using a communication module capable of Wi-Fi (registered trademark) or Bluetooth (registered trademark), for example.

[0097] The recording unit 15 records various information related to the learned model generation device 1. The recording unit 15 is configured using a HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, a volatile memory, a non-volatile memory, etc. The recording unit 15 includes a program recording unit 151, a learning model recording unit 152, and a learned model recording unit 153.

[0098] The program recording unit 151 records various programs executed by the learned model generating device 1 , for example, a program for generating a learned model.

[0099] The learning model recording unit 152 records a learning model for inputting an input parameter to the first control unit 16 described later.

[0100] The learned model recording unit 153 records a plurality of learned models learned using the learning data.

[0101] The first control unit 16 is implemented using a processor having hardware such as an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), or a CPU (Central Processing Unit), and a memory as a temporary storage area used by the processor. The first control unit 16 includes a drive control unit 161, an imaging control unit 162, an acquisition unit 163, a determination unit 164, a learning unit 165, and an output control unit 166.

[0102] The drive control section 161 drives the focus adjustment mechanism 103 to move the focus lens L5 to the near end of the optical system 101 .

[0103] The imaging control unit 162 causes the imaging element 102 to capture images when the focus lens L5 is located at the near end, in the process of moving from the near end to the far end, and at the far end.

[0104] The acquisition unit 163 acquires the image data generated by the imaging element 102. Specifically, the acquisition unit 163 acquires the image data generated by the imaging element 102 shooting at the near end of the focus lens L5 (hereinafter referred to as the "first image"). In addition, the acquisition unit 163 acquires a plurality of image data generated by the imaging element 102 shooting multiple times during the movement period of the focus lens L5 from the near end to the far end (hereinafter referred to as the "multiple second images"). The acquisition unit 163 acquires the image data generated by the imaging element 102 shooting at the far end of the focus lens L5 (hereinafter referred to as the "third image").

[0105] The determination unit 164 determines whether a predetermined number of images required for learning have been acquired.

[0106] The learning unit 165 performs learning using a prescribed learning model and learning data, the learning data being obtained by combining a plurality of images and a depth map of each subject obtained by the acquisition unit 163, each of which has different focus positions. Specifically, the learning unit 165 performs learning using a learning model generated by machine learning using artificial intelligence (AI (Artificial Intelligence)). The method for constructing the learning model used in the learning unit 165 is not particularly limited, and for example, various machine learning methods such as deep learning using a neural network, a support vector machine, a decision tree, naive Bayes, and a k-nearest neighbor method can be used. In addition, the depth map is information obtained by detecting the subject distance at each pixel position, and the subject distance is the distance from the position of the imaging element 203 (the position of the front end 24) to the corresponding position on the observed object corresponding to the pixel position in the captured image.

[0107] The output control unit 166 outputs the learned model generated by the learning unit 165 to the learned model recording unit 153 .

[0108] [Schematic structure of endoscope system]

[0109] Next, the configuration of the endoscope system 100 will be described. Figure 3 FIG. 1 is a diagram showing the overall configuration of an endoscope system 100 according to an embodiment.

[0110] Figure 3 The endoscope system 100 shown in the figure continuously images the inside of the subject by inserting the insertion portion 21 of the endoscope 2 from the mouth of the subject such as a human or an animal through the esophagus or from the anus of the subject into the large intestine, and displays the image data group captured along the time series on the display device 3. In the endoscope system 100, the image diagnosis device 5 (CAD: Computer Aided Detection) detects (recognizes or estimates) organs, parts, characteristic areas including polyps or cancers, etc. of a plurality of observation parts captured in the image data, and treatment of the living body by treatment instruments such as forceps and snares, based on the image data group input from the control device 4 and the derivation model (learned model) learned in advance using learning data obtained by capturing organs, parts, and lesions of the subject, and outputs the detection result information to the control device 4. The operator such as a doctor observes and treats the subject while confirming the display image displayed on the display device 3 and the detection result (estimated result) of the image diagnosis device 5.

[0111] like Figure 3 As shown, the endoscope system 100 includes an endoscope 2 , a display device 3 , a control device 4 , and an image diagnosis device 5 .

[0112] The endoscope 2 continuously generates image data (RAW data) by photographing the inside of the subject, and sequentially outputs the image data to the control device 4. Figure 1 As shown, the endoscope 2 includes an insertion portion 21 , an operation portion 22 , and a universal cable 23 .

[0113] At least a portion of the insertion portion 21 is flexible and is inserted into the subject. Figure 1 As shown, the insertion portion 21 includes: a front end portion 24, which is arranged at the front end of the insertion portion 21; a bending portion 25, which is connected to the base end side (operation portion 22 side) of the front end portion 24 and is configured to be bendable; and a long flexible tube portion 26, which is connected to the base end side of the bending portion 25 and has flexibility.

[0114] The operation unit 22 is connected to the base end portion of the insertion unit 21. The operation unit 22 accepts various operations on the endoscope 2. Figure 1 As shown, the operating portion 22 is provided with a bending knob 221 , an insertion port 222 , and a plurality of operating members 223 .

[0115] The bending knob 221 is configured to be rotatable in response to a user operation by a user such as an operator. The bending knob 221 is rotated to operate a bending mechanism (not shown) such as a metal or resin wire disposed in the insertion portion 21. As a result, the bending portion 25 bends.

[0116] The insertion port 222 communicates with a treatment instrument channel (not shown) which is a pipeline extending from the front end of the insertion portion 21 , and is an insertion port for inserting a treatment instrument or the like through the treatment instrument channel from the outside of the endoscope 2 .

[0117] The plurality of operating components 223 are composed of buttons and the like for receiving various operations performed by a user such as an operator, and outputs operation signals corresponding to the various operations to the control device 4 via the universal cable 23. Examples of the various operations include a release operation for instructing the endoscope 2 to take a still image, an operation for switching the observation mode of the endoscope 2 to a normal light observation mode or a special observation mode, and the like.

[0118] The universal cable 23 extends from the operation portion 22 in a direction different from the extending direction of the insertion portion 21, and is provided with a light guide 231 (see Figure 2 ), and the first signal line 232 (refer to Figure 2 ), a second signal line 233 (refer to Figure 2 ) etc. Moreover, Figure 1 As shown, a first connector portion 27 is provided at the base end of the universal cable 23. The first connector portion 27 is connected to the control device 4 in a detachable manner.

[0119] The display device 3 is composed of a display monitor such as a liquid crystal or an organic EL, and displays a display image based on image data subjected to image processing by the control device 4 and various information related to the endoscope 2 under the control of the control device 4 .

[0120] The control device 4 is implemented using a processor as a processing device having hardware such as a GPU, FPGA, or CPU, and a memory as a temporary storage area used by the processor. The control device 4 centrally controls the operation of each part of the endoscope 2 according to a program recorded in the memory.

[0121] The image diagnosis device 5 (CAD) detects (estimates) organs, parts, insertion speed of the insertion portion 21, characteristic regions (abnormal regions or lesion candidate regions) including polyps or cancers, and treatment of the living body by treatment instruments such as forceps or snares captured in the image data, using the image data group input from the control device 4 and the learned model learned in advance using the learning data, and outputs the detection result information that associates the detection result with the detection time to the control device 4. The image diagnosis device 5 is implemented using a processor as a processing device having hardware such as a GPU, FPGA, or CPU, and a memory as a temporary storage area used by the processor.

[0122] [Functional structure of the main parts of the medical support system]

[0123] Next, the functional configuration of the main parts of the endoscope system 100 will be described.

[0124] Figure 4 2 is a block diagram showing the functional configuration of main parts of the endoscope 2 and the control device 4. Hereinafter, the endoscope 2 and the control device 4 will be described in order.

[0125] [Functional structure of endoscope]

[0126] First, the structure of the endoscope 2 will be described.

[0127] like Figure 4 As shown, the endoscope 2 includes an illumination optical system 201, an optical system 202, an imaging element 203, an A / D converter 204, a P / S converter 205, a focus adjustment mechanism 206, and an imaging control unit 207. Here, the illumination optical system 201, the optical system 202, the imaging element 203, the A / D converter 204, the P / S converter 205, the focus adjustment mechanism 206, and the imaging control unit 207 are respectively arranged in the distal end portion 24.

[0128] The illumination optical system 201 is constituted by one or a plurality of lenses and the like, and irradiates the illumination light supplied from the light guide 231 toward the subject.

[0129] The optical system 202 is composed of lenses L1 to L10 and an aperture Di. The optical system 202 forms a subject image on the light receiving surface of the imaging element 203. In addition, the optical system 202 has a focus lens L5 that can move along the optical axis O1 direction among the lenses L1 to L10. The focus lens L5 moves back and forth along the optical axis O1 direction through the focus adjustment mechanism 206. In one embodiment, the optical system 202 is capable of switching a focus position suitable for observation of medium and distant scenes and a focus position suitable for observation of near points. Of course, the optical system 202 may not need to switch the focus position discretely in binary, but may be able to continuously change to multiple desired positions.

[0130] The imaging element 203 is formed by using a CCD or CMOS image sensor, which is formed by configuring any one of the color filters constituting the Bayer arrangement (RGGB) on each pixel of a plurality of pixels configured in a two-dimensional matrix. Under the control of the imaging control unit 207, the imaging element 203 receives the subject image formed by the optical system 202, performs photoelectric conversion and generates a camera image (analog signal). In addition, in one embodiment, the imaging element 203 can also be an imaging element integrally composed of an image sensor and a TOF sensor that obtains a depth map (subject distance information) by a TOF (Time Of Flight) method. In addition, as a structure for generating a depth map, it is not limited to the above-mentioned TOF sensor, and an image sensor including a phase difference sensor can also be used. In the following, the depth map and the camera image are summarized and recorded as image data. The imaging element 203 outputs the image data to the A / D conversion unit 204.

[0131] The A / D converter 204 is configured using an A / D converter circuit, etc. The A / D converter 204 performs A / D conversion processing on analog image data input from the image sensor 203 under the control of the image capture control unit 207 , and outputs the result to the P / S converter 205 .

[0132] The P / S conversion unit 205 is configured using a P / S conversion circuit or the like, and performs parallel / serial conversion on the digital image data input from the A / D conversion unit 204 under the control of the imaging control unit 207 , and outputs the data to the control device 4 via the first signal line 232 .

[0133] In addition, an E / O conversion unit that converts image data into an optical signal may be provided instead of the P / S conversion unit 205, and the image data may be output to the control device 4 through the optical signal. In addition, for example, the image data may be sent to the control device 4 through wireless communication such as Wi-Fi (Wireless Fidelity) (registered trademark).

[0134] The focus adjustment mechanism 206 changes the focus (focus position) of the optical system 101 by moving the focus lens L5 of the optical system 101 along the optical axis O1 under the control of the second control unit 410 or the imaging control unit 207. The focus adjustment mechanism 206 includes a focus driving unit 206a and a detection unit 206b.

[0135] The focus driving unit 206 a is configured using an actuator including a stepping motor or a voice coil motor, for example, and is driven under the control of the second control unit 410 or the imaging control unit 207 to move the focus lens L5 to a predetermined position on the optical axis O1 .

[0136] The detection unit 206b detects focal length information related to the position or focal length of the focus lens L5 in the optical system 202, and outputs the detection result to the second control unit 410. The detection unit 206b is configured using, for example, an encoder or an optical coupler. In addition, the detection unit 206b may also detect the position of the focus lens L5 on the optical axis O1 based on the driving amount of the focus driving unit 206a.

[0137] The imaging control unit 207 is implemented using a TG (Timing Generator), a processor as a processing device having hardware such as a CPU, and a memory as a temporary storage area used by the processor. The imaging control unit 207 controls the operation of the imaging element 203, the A / D conversion unit 204, and the P / S conversion unit 205 according to the setting data received from the control device 4 via the second signal line 233.

[0138] [Structure of control device]

[0139] Next, the configuration of the control device 4 will be described.

[0140] like Figure 4 As shown, the control device 4 includes a condenser lens 401 , a first light source unit 402 , a second light source unit 403 , a light source control unit 404 , an S / P conversion unit 405 , an image processing unit 406 , an input unit 407 , a recording unit 408 , a communication unit 409 , and a second control unit 410 .

[0141] The condenser lens 401 condenses the light emitted from the first light source unit 402 and the second light source unit 403 , and emits the light toward the light guide 231 .

[0142] The first light source unit 402 emits white light (normal light) as visible light under the control of the light source control unit 404, thereby providing the white light as illumination light to the light guide 231. The first light source unit 402 is composed of a collimating lens, a white LED (Light Emitting Diode) lamp, a driver, etc. In addition, as the first light source unit 402, a red LED lamp, a green LED lamp, and a blue LED lamp can also be used to emit light simultaneously to supply white light of visible light. In addition, as the first light source unit 402, a halogen lamp, a xenon lamp, etc. can also be used.

[0143] The second light source unit 403 supplies special light as illumination light to the light guide 231 by emitting special light having a predetermined wavelength band under the control of the light source control unit 404. Here, the special light refers to light used for narrowband light observation (NBI: Narrowband Imaging) based on narrowband light including 390nm to 445nm and 530nm to 550nm. Of course, as special light, in addition to narrowband light, amber light (600nm and 630nm) used for red light observation (RDI: Red dichromatic Imaging) may also be used.

[0144] The light source control unit 404 is implemented using a processor having hardware such as an FPGA or a CPU and a memory as a temporary storage area used by the processor. The light source control unit 404 controls the light emission timing and light emission time of the first light source unit 402 and the second light source unit 403 according to the control data input from the second control unit 410.

[0145] Under the control of the second control unit 410, the S / P conversion unit 405 performs serial / parallel conversion on the image data received from the endoscope 2 via the first signal line 232, and outputs it to the image processing unit 406. In addition, when the endoscope 2 outputs the image data as an optical signal, an O / E conversion unit that converts the optical signal into an electrical signal may be provided instead of the S / P conversion unit 405. Furthermore, when the endoscope 2 transmits the image data by wireless communication, a communication module that can receive wireless signals may be provided instead of the S / P conversion unit 405.

[0146] The image processing unit 406 is implemented using a processor having hardware such as a GPU or FPGA and a memory as a temporary storage area used by the processor. Under the control of the second control unit 410, the image processing unit 406 performs predetermined image processing on the image data of the parallel data input from the S / P conversion unit 405, and outputs it to the display device 3. Here, as the predetermined image processing, demosaicing processing, white balance processing, gain adjustment processing, gamma correction processing, format conversion processing, etc. can be exemplified.

[0147] The input unit 407 is composed of a mouse, a foot switch, a keyboard, buttons, switches, a touch panel, etc., receives user operations performed by a user such as an operator, and outputs an operation signal corresponding to the user operation to the second control unit 410.

[0148] The recording unit 408 is composed of a recording medium such as a volatile memory, a non-volatile memory, an SSD (Solid State Drive), an HDD (Hard Disk Drive), or a memory card. The recording unit 408 records data including various parameters required for the operation of the control device 4 and the endoscope 2. The recording unit 408 includes: a program recording unit 408a that records various programs for operating the endoscope 2 and the control device 4; an image data recording unit 408b that records and stores an image file corresponding to the image data; a learned model recording unit 408c that records the learned model; and a camera parameter recording unit 408d.

[0149] The image data recording unit 408b records, under the control of the second control unit 410, an image data group generated by continuously photographing a plurality of observation sites of the subject through the endoscope 2 and an image (endoscopic image) corresponding to the static image data in association with patient information, etc. Here, the static image data is image data photographed by the imaging element 203 when imaging is instructed by a release signal input from the operation unit 22, and is image data having a higher resolution than the live view images sequentially generated by the imaging element 203.

[0150] The learned model recording unit 408 c records a learned model that uses a plurality of images having different in-focus positions as input parameters and outputs a depth map indicating the distance from the optical system 202 to the subject as an output parameter.

[0151] The camera parameter recording unit 408d records the camera parameters of the optical system 202. Specifically, the camera parameters are conversion coefficients that associate the image plane (the position of the pixel of the imaging element 203) with the object (the position of the subject). For example, it is a conversion matrix that can be obtained by calibrating the camera using a checkered flat plate. In addition, the camera parameter recording unit 408d may also record the value of the point (x, y) on the image corresponding to the depth z for each pixel of the imaging element 203 as a lookup table. In this case, the second control unit 410 described later may also derive the point (x, y) on the image based on the estimated depth z. In addition, the camera parameter recording unit 408d may also record the approximate formula of the point (x, y) on the image corresponding to the depth z for each pixel of the imaging element 203. In this case, the second control unit 410 described later may also derive the point (x, y) on the image based on the estimated depth z.

[0152] The communication unit 409 transmits various information to the image diagnosis device 5 or the learned model generation device 1 under the control of the second control unit 410, and receives various information from the image diagnosis device 5 or the learned model generation device 1 and outputs it to the second control unit 410. Specifically, the communication unit 409 transmits a temporally continuous image data group on which the image processing unit 406 has performed image processing to the image diagnosis device 5, receives detection results including organs, parts, and feature areas detected by the image diagnosis device 5, and outputs them to the second control unit 410. In addition, the communication unit 409 receives the learned model received from the learned model generation device 1 under the control of the second control unit 410 and outputs it to the learned model recording unit 408c. The communication unit 409 is configured using a communication module or the like.

[0153] The second control unit 410 is equivalent to the second processor of the present invention. The second control unit 410 is implemented using a second processor as a processing device having hardware such as an FPGA or a CPU and a memory as a temporary storage area used by the second processor. In addition, the second control unit 410 uniformly controls the various parts constituting the endoscope 2 and the control device 4. The second control unit 410 has a drive control unit 410a, an imaging control unit 410b, an acquisition unit 410c, a derivation unit 410d, a calculation unit 410e, and an output control unit 410f.

[0154] The drive control unit 410 a drives the focus adjustment mechanism 206 to move the focus lens L5 toward the near side or the far side.

[0155] The imaging control unit 410b causes the imaging element 203 to expose and photograph the subject image focused by the focus lens L5 at the near end or the far end. Furthermore, the imaging control unit 410b causes the imaging element 203 to expose and photograph the focused subject image while the focus lens L5 moves from the near end to the far end.

[0156] The acquisition unit 410c acquires a first image generated by the imaging element 203 capturing the subject on the near side of the focus lens L5. Furthermore, the acquisition unit 410c acquires any one or more of a plurality of second images generated by the imaging element 203 capturing the subject during the period of movement by changing the focal length between the near side and the far side of the focus lens L5 by the focus adjustment mechanism 206. Furthermore, the acquisition unit 410c acquires a third image generated by the imaging element 203 capturing the subject on the far side of the focus lens L5.

[0157] The derivation unit 410 d estimates a depth map of the subject using the first image, the second image, and the third image acquired by the acquisition unit 410 c and the learned model recorded by the learned model recording unit 408 c .

[0158] The calculation unit 410 e estimates the shape of the object based on the depth map of the object estimated by the derivation unit 410 d and the camera parameters of the optical system 202 recorded by the camera parameter recording unit 408 d .

[0159] The output control unit 410 f outputs the shape of the subject estimated by the calculation unit 410 e to the display device 3 .

[0160] [Processing by the learned model generation device]

[0161] Next, the processing executed by the learned model generation device 1 will be described. Figure 5 It is a flowchart showing an overview of the processing performed by the learned model generation device 1. Here, the subject distance is the distance from the front end of the optical system 101 to the subject K1. In addition, the subject is described below as a flat subject without bumps and depressions that is directly opposite the optical system 101. The distance range of the flat subject to be learned is wider than the near end and the far end of the focus position. For example, when the near end of the focus position is set to 4 mm and the far end is set to 20 mm, the distance of the subject to be learned is set to 2 mm to 40 mm for description. Therefore, the depth of the subject located at a position closer to the end or farther than the focus position can be estimated.

[0162] like Figure 5 As shown, first, the learned model generation device 1 sets the subject (step S100). Specifically, the learned model generation device 1 sets the subject based on the operation information input by the user via the input unit 13. In this case, in the case of an LCD monitor, the user uses the input unit 13 to input the category content of the pattern and the distance between the LCD monitor and the learned model generation device 1.

[0163] Thereafter, the drive control unit 161 drives the focus adjustment mechanism 103 to move the focus lens L5 to the near end of the optical system 101 (step S101 ).

[0164] Next, the imaging control unit 162 causes the imaging element 102 to capture images at the close end of the focus lens L5 (step S102 ).

[0165] Then, the acquisition unit 163 acquires a first image generated by the imaging element 102 at the close end of the focus lens L5 (step S103 ).

[0166] Next, the drive control unit 161 drives the focus adjustment mechanism 103 to move the focus lens L5 from the near end to the far end (step S104 ).

[0167] Thereafter, the imaging control unit 162 controls the imaging element 102 to perform imaging a plurality of times while the focus lens L5 moves from the near end to the far end (step S105 ).

[0168] Next, the acquisition unit 163 acquires a plurality of second images generated by the imaging element 102 performing multiple shots during the movement period when the focus lens L5 moves from the near end to the far end (step S106). In addition, the acquisition unit 163 acquires all of the plurality of second images generated by the imaging element 102 performing multiple shots during the movement period when the focus lens L5 moves from the near end to the far end, but is not limited thereto, and it is sufficient as long as at least one image is acquired during the movement period when the focus lens L5 moves from the near end to the far end.

[0169] Then, the imaging control unit 162 causes the imaging element 102 to capture images at the far end of the focus lens L5 (step S107 ).

[0170] Next, the acquisition unit 163 acquires a third image generated by the imaging element 102 capturing the image at the far end of the focus lens L5 (step S108 ).

[0171] After that, the determination unit 164 determines whether a predetermined number of images required for learning have been obtained (step S109). Figure 6 to Figure 9 As shown, the determination unit 164 determines whether the plurality of images generated and acquired by photographing the plurality of subjects at the plurality of focus positions respectively exceed a predetermined number (e.g., 10,000 images). When the determination unit 164 determines that the predetermined number of images required for learning has been acquired (step S109: Yes), the learned model generation device 1 moves to step S110 described later. On the other hand, when the determination unit 164 determines that the predetermined number of images required for learning has not been acquired (step S109: No), the learned model generation device 1 returns to step S101.

[0172] In step S110, the acquisition unit 163 acquires a depth map of each subject from an external device or a recording unit 15 via the communication unit 14. The depth map (depth distribution) is obtained by adding a correct value of the distance from the optical system 101 to the subject to each pixel of each subject image. Of course, the acquisition unit 163 may also acquire a depth map input by a user via the input unit 13.

[0173] Next, the learning unit 165 performs learning using a prescribed learning model and learning data (step S111), wherein the learning data is data combining a plurality of images and depth maps of each subject obtained by the acquisition unit 163, each of which has different focus positions. Specifically, the learning unit 165 performs learning using a learning model generated by machine learning using artificial intelligence (AI (Artificial Intelligence)). The method of constructing the learning model used in the learning unit 165 is not particularly limited, and various machine learning methods such as deep learning using neural networks, support vector machines, decision trees, naive Bayes, and k-nearest neighbor methods can be used.

[0174] Figure 6 Schematically shows the learning data set input to the learning model by the learning unit 165. Figure 6 As shown, the learning unit 165 performs learning using a predetermined learning model and learning data to generate a learned model, wherein the learning data is data combining a plurality of images FP1 to FP4 with different focus positions for each subject acquired by the acquisition unit 163 and a depth map. That is, the learning unit 165 generates a learned model that uses the first image, the second image, and the third image as input parameters and outputs the distance information between the subject and the optical system 101 as an output parameter.

[0175] In one embodiment, the depth map and the image may be associated with at least one of the first image at the near end and the second image at the far end among the plurality of images. Figure 7 is a diagram schematically illustrating the relationship between the depth map and the image. Figure 7 In the following, a first image P100, a third image P101, and a second image P102 having different in-focus positions (foci) of the optical system 101 will be described. Figure 7 In the example, the frame W1 of the first image P100 corresponds to the view angle of the third image P101, and the frame W2 corresponds to the view angle of the second image P102. In addition, the area D1 is an area where the depth map can be estimated.

[0176] like Figure 7 As shown, in one embodiment, since the first image P100 and the third image P101 respectively include prescribed features, such as characteristic patterns, a correspondence can be established between the first image P100 and the third image P101, so that the depth map can be associated with other images, such as the second image P102. Therefore, in one embodiment, when associating the depth map, it is desired to associate with an image with a wide viewing angle, such as the first image P100.

[0177] return Figure 5 , continue with the description after step S112.

[0178] In step S102, the output control unit 166 outputs the learned model generated by the learning unit 165 to the learned model recording unit 153 (step S112). In this case, the output control unit 166 may output the learned model to an external device, such as the endoscope system 100 described later, via the communication unit 14. After step S111, the learned model generation device 1 ends the present process.

[0179] [Processing performed by the medical support system]

[0180] Next, the processing executed by the endoscope system 100 will be described. Figure 8 : is a flowchart showing an overview of processing executed by the endoscope system 100 .

[0181] like Figure 8 As shown, the second control unit 410 determines whether an instruction signal for indicating that the observation mode of the endoscope 2 is a measurement mode for measuring the shape of the object is input from the operation unit 22 of the endoscope 2 (step S201). When the second control unit 410 determines that an instruction signal for indicating that the observation mode of the endoscope 2 is a measurement mode for measuring the shape of the object is input from the operation unit 22 of the endoscope 2 (step S201: Yes), the endoscope system 100 shifts to step S202 described later. On the other hand, when the second control unit 410 determines that an instruction signal for indicating that the observation mode of the endoscope 2 is a measurement mode for measuring the shape of the object is not input from the operation unit 22 of the endoscope 2 (step S201: No), the endoscope system 100 shifts to step S204 described later.

[0182] In step S202, the endoscope system 100 executes a measurement mode process for measuring the shape of a predetermined object in the subject. The measurement mode process will be described in detail later.

[0183] Next, the second control unit 410 determines whether an instruction signal for ending the observation by the endoscope 2 is input from the operation unit 22 of the endoscope 2 (step S203). When the second control unit 410 determines that an instruction signal for ending the observation by the endoscope 2 is input from the operation unit 22 of the endoscope 2 (step S203: Yes), the endoscope system 100 ends the present process. On the other hand, when the second control unit 410 determines that an instruction signal for ending the observation by the endoscope 2 is not input from the operation unit 22 of the endoscope 2 (step S203: No), the endoscope system 100 returns to the above-mentioned step S201.

[0184] In step S204, the endoscope system 100 performs normal observation of the subject. Specifically, the second control unit 410 controls the light source control unit 404 to make the first light source unit 402 or the second light source unit 403 emit light, and the imaging element 203 captures the reflected light from the subject to generate image data. Then, the second control unit 410 causes the image processing unit 406 to perform image processing on the image data generated by the imaging element 203 and display it on the display device 3, thereby performing normal observation of the subject. After step S204, the endoscope system 100 transfers to step S203.

[0185] [Overview of measurement mode processing]

[0186] Next, Figure 8 The details of the measurement mode processing described in step S202 will be described. Fig. 9 This is a flowchart showing an overview of the measurement mode processing. Fig.10 2 is a timing chart showing the position of the focus lens L5 of the endoscope 2 on the optical axis O1 and the exposure timing of the imaging element 203 during the measurement mode processing. Fig.10 middle, Fig.10 (a) shows the position of the focus lens L5, and Fig.10 (b) shows the exposure timing of the imaging element 203 .

[0187] like Fig.10 As shown, first, the drive control unit 410a controls the focus adjustment mechanism 206 to stop the AF drive performed by the focus adjustment mechanism 206 (step S301).

[0188] Next, the drive control unit 410a drives the focus adjustment mechanism 206 to move the focus lens L5 toward the near side (step S302), and stops the focus lens L5 at the near end (step S303). Fig.10 As shown in (a), the drive control unit 410a drives the focus adjustment mechanism 206 to move the focus lens L5 to the near side, and stops the focus lens L5 at the near end.

[0189] Then, the imaging control unit 410b causes the imaging element 203 to expose and capture the subject image focused by the focus lens L5 at the near end (step S304). Fig.10 as well as Fig.11 As shown in FIG. 1 , the imaging control unit 410b causes the imaging element 203 to expose and capture the subject image of the subject K1 focused at the near end by the focus lens L5. Fig.12 As shown, the acquisition unit 410c acquires the image data generated by the imaging element 203 as the first image P200.

[0190] Next, the drive control unit 410a drives the focus adjustment mechanism 206 to move the focus lens L5 to the far side (step S305).

[0191] Then, the imaging control unit 410b causes the imaging element 203 to expose the subject image at a predetermined frame rate and to capture the subject image during the movement period of the focus lens L5 from the near end to the far end (step S306). Fig.10 As shown in (b), the camera control unit 410b causes the camera element 203 to expose and shoot the subject image at a predetermined frame rate, such as 60 FPS, during the movement period of the focus lens L5 from the near end to the far end, thereby generating a plurality of image data. Fig.13 As shown, the acquisition unit 410 c acquires any one or more image data among a plurality of image data generated by the imaging element 203 as the second image P201 .

[0192] Next, the drive control unit 410a drives the focus adjustment mechanism 206 to stop the focus lens L5 at the far end (step S307). Fig.10 As shown in (a), the drive control unit 410a drives the focus adjustment mechanism 206 to stop the focus lens L5 at the far end.

[0193] Then, the imaging control unit 410b causes the imaging element 203 to expose the subject image focused at the far end by the focus lens L5 and to shoot (step S308). Fig.10 As shown in (b) of FIG. 1 , the imaging control unit 410b causes the imaging element 203 to expose and shoot the subject image focused at the far end by the focus lens L5. In this case, Fig.14 As shown, the acquisition unit 410c acquires the image data generated by the imaging element 203 as the third image P202.

[0194] Next, the drive control unit 410 a controls the focus adjustment mechanism 206 to move the focus lens L5 to the stop position of the AF drive (step S309 ), and starts the AF drive of the focus adjustment mechanism 206 (step S310 ).

[0195] Thereafter, the derivation unit 410d estimates the depth map of the subject using the first image, the second image, and the third image acquired by the acquisition unit 410c and the learned model recorded by the learned model recording unit 408c (step S311). Specifically, the derivation unit 410d inputs the first image, the second image, and the third image acquired by the acquisition unit 410c as input parameters to the learned model, and estimates the depth map of the subject output by the learned model as an output parameter.

[0196] Next, the calculation unit 410 e estimates the shape of the subject based on the depth map of the subject estimated by the derivation unit 410 d and the camera parameters of the optical system 202 recorded by the camera parameter recording unit 408 d (step S312 ).

[0197] Fig.15 4 is a diagram schematically illustrating a method for estimating the shape of a subject by the calculation unit 410e. Fig.15 As shown, the calculation unit 410e assumes that the position of a specified pixel in the camera element 203 is x_imager, the angle at which the optical axis O1 of the optical system 202 intersects the straight line connecting the position Q1 of the specified subject and the position x of the camera element 203 is θ, and the distance from the optical system 202 to the position Q1 of the specified subject based on the depth map is z, then the following formula (1) holds.

[0198] x=z×tanθ...(1)

[0199] The calculation unit 410 e estimates the shape of the subject using the above-mentioned equation (1) for each pixel of the imaging element 203 .

[0200] Afterwards, the output control unit 410f outputs the shape of the subject estimated by the calculation unit 410e to the display device 3 (step S313). Fig.16 As shown in FIG. 4 , the output control unit 410f outputs an image P203 corresponding to the shape of the subject estimated by the calculation unit 410e. After step S313, the endoscope system 100 returns to step S314. Figure 8 The main routine.

[0201] According to one embodiment described above, the derivation unit 410d estimates the depth map of the subject using the first image, the second image, and the third image acquired by the acquisition unit 410c and the learned model recorded by the learned model recording unit 408c, thereby improving the detection accuracy of the distance from the optical system to the subject serving as the measurement object.

[0202] In addition, according to one embodiment, the calculation unit 410e estimates the shape of the subject based on the depth map of the subject estimated by the derivation unit 410d and the camera parameters of the optical system 202 recorded by the camera parameter recording unit 408d, thereby further improving the estimation accuracy of the three-dimensional shape of the subject as the measurement object.

[0203] In addition, in one embodiment, during the movement of the focus lens L5 from the near end to the far end, the acquisition unit 163 acquires multiple second images constituting a dynamic image generated by the camera element 102, but is not limited to this. For example, a focus sweep image may be used instead of a frame constituting a dynamic image.

[0204] In one embodiment, the acquisition unit 410c, the derivation unit 410d, the calculation unit 410e, and the output control unit 410f are provided in the control device 4, but the present invention is not limited thereto, and the functions of the derivation unit 410d, the calculation unit 410e, and the output control unit 410f may be provided in the image diagnosis device 5. In one embodiment, in addition to the image diagnosis device 5, the functions of the derivation unit 410d, the calculation unit 410e, and the output control unit 410f may be provided in an external server or the like.

[0205] (Variant 1)

[0206] Next, a modification example 1 of an embodiment is described. In modification example 1, the same structure as the endoscope system 100 of the above-mentioned embodiment is provided, but the processing performed is different. Specifically, in the above-mentioned embodiment, the imaging element 203 is exposed to generate the first image and the third image when the focus lens L5 stops at the near end and the far end, but in modification example 1, the imaging element 203 is exposed to generate the first image and the third image when the focus lens L5 reaches the near end and the far end. Therefore, the processing performed by the endoscope system 100 of modification example 1 of an embodiment is described below.

[0207] Fig.17 1 is a timing chart showing the position of the focus lens L5 of the endoscope 2 on the optical axis O1 and the exposure timing of the imaging element 203 during the measurement mode processing. Fig.17 middle, Fig.17 (a) shows the position of the focus lens L5, and Fig.17 (b) shows the exposure timing of the imaging element 203 .

[0208] like Fig.17 As shown in FIG. 1 , the drive control unit 410a drives the focus adjustment mechanism 206 to move the focus lens L5 from the current position to the near side and after reaching the near end, the focus lens L5 is moved linearly to the far side. In this case, the imaging control unit 410b causes the imaging element 203 to expose and shoot the subject image focused by the focus lens L5 at the near end. At this time, the acquisition unit 410c acquires the image data generated by the imaging element 203 as the first image.

[0209] Next, after the focus lens L5 reaches the far end, the drive control unit 410a moves the focus lens L5 to the AF stop position. In this case, the camera control unit 410b causes the camera element 203 to expose and shoot the subject image that is focused during the movement of the focus lens L5 from the near end to the far end. At this time, the acquisition unit 410c acquires the image data generated by the camera element 203 as the second image. Furthermore, the camera control unit 410b causes the camera element 203 to expose and shoot the subject image that is focused by the focus lens L5 at the far end. At this time, the acquisition unit 410c acquires the image data generated by the camera element 203 as the third image.

[0210] In this way, the drive control unit 410a moves the focus lens L5 linearly from the near end to the far end. This can shorten the time required to return from the measurement mode to the normal mode.

[0211] According to the above-described first modification, since the depth map of the subject is estimated, it is possible to improve the accuracy of detecting the distance from the optical system to the subject as the measurement target object.

[0212] (Variant 2)

[0213] Next, a modification 2 of the embodiment will be described. The modification 2 has the same structure as the endoscope system 100 of the embodiment described above, but performs different processes. Therefore, the process performed by the endoscope system 100 of the modification 1 of the embodiment will be described below.

[0214] Fig.18 1 is a timing chart showing the position of the focus lens L5 of the endoscope 2 on the optical axis O1 and the exposure timing of the imaging element 203 during the measurement mode processing. Fig.18 middle, Fig.18 (a) shows the position of the focus lens L5, and Fig.18 (b) shows the exposure timing of the imaging element 203 .

[0215] like Fig.18 As shown in FIG. 1 , the drive control unit 410a drives the focus adjustment mechanism 206 to move the focus lens L5 from the current position to the near side, and after reaching the near end, stops at the near end for a certain period of time, and then moves the focus lens L5 to the far side. In this case, the camera control unit 410b causes the camera element 203 to expose and shoot the subject image focused by the focus lens L5 at the near end. At this time, the acquisition unit 410c acquires the image data generated by the camera element 203 as the first image.

[0216] Next, the drive control unit 410a causes the focus lens L5 to stop at the far end for a certain period of time after reaching the far end, and then moves the focus lens L5 to the AF stop position. In this case, the camera control unit 410b causes the imaging element 203 to expose and shoot the subject image that the focus lens L5 focuses on during the movement from the near end to the far end. At this time, the acquisition unit 410c acquires the image data generated by the imaging element 203 as the second image. Furthermore, the camera control unit 410b causes the imaging element 203 to expose and shoot the subject image that the focus lens L5 focuses on at the far end. At this time, the acquisition unit 410c acquires the image data generated by the imaging element 203 as the third image.

[0217] In this way, the drive control unit 410a moves the focus lens L5 smoothly in a sine wave shape from the near end to the far end, thereby obtaining the first image and the third image in which blurring due to slight movement of the focus lens L5 is suppressed.

[0218] According to the above-described second modification, since the depth map of the subject is estimated, it is possible to improve the detection accuracy of the distance from the optical system to the subject as the measurement target object.

[0219] (Other embodiments)

[0220] By appropriately combining the multiple components disclosed in the endoscope system of one embodiment of the present invention, various inventions can be formed. For example, several components can be deleted from all the components recorded in the endoscope system of the embodiment of the present invention. Furthermore, the components described in the endoscope system of the embodiment of the present invention can also be appropriately combined.

[0221] In the endoscope system according to one embodiment of the present invention, the above-mentioned "unit" can be replaced by "unit", "circuit", etc. For example, the control unit can be replaced by a control unit or a control circuit.

[0222] Furthermore, the program executed by the endoscope system involved in one embodiment of the present invention is provided in the form of file data in an installable or executable form recorded on a computer-readable recording medium such as a CD-ROM, a floppy disk (FD), a CD-R, a DVD (Digital Versatile Disk), a USB medium, a flash memory, etc.

[0223] Furthermore, the program executed by the endoscope system according to one embodiment of the present invention may be stored in a computer connected to a network such as the Internet, and provided by being downloaded via the network.

[0224] In addition, in the description of the flowcharts in this specification, expressions such as "first", "after", and "continue" are used to clearly indicate the chronological relationship between the processes of the steps, but these expressions are not used to uniquely determine the order of the processes required to implement the present invention. That is, the order of the processes in the flowcharts described in this specification can be changed within the scope of no contradiction.

[0225] As mentioned above, several embodiments of the present application have been described in detail based on the drawings, but these are examples, and the present invention can be implemented in other forms in which various modifications and improvements are implemented based on the knowledge of those skilled in the art, including the forms described in the disclosure section of the present invention.

Claims

1. A control device comprising a processor, wherein: The processor obtains a first image, a second image, and a third image from a camera device, The processor derives a depth map of the subject by inputting the first image, the second image, and the third image as input parameters into a learned model, The processor outputs the depth map of the object, The depth map represents the distance from the optical system to the object, and the learned model is a model that outputs the depth map as an output parameter. The imaging device comprises: the optical system capable of changing the focal length; an imaging element for exposing an image formed by the optical system to generate a static image; and a focus adjustment mechanism for changing the focal length by moving the optical system along the optical axis direction of the optical system. The first image is an image generated by the imaging element on the near side of the optical system photographing the subject, the second image is an image generated by the imaging element photographing the subject while the optical system is moved between the near side and the far side by changing the focal length through the focus adjustment mechanism, and the third image is an image generated by the imaging element on the far side of the optical system photographing the subject.

2. The control device according to claim 1, wherein: The second image is at least one frame image in a dynamic image consisting of a group of images captured temporally continuously by the imaging element while the focus adjustment mechanism changes the focal length between the near side and the far side.

3. The control device according to claim 2, wherein: The processor obtains camera parameters of the optical system, The processor estimates a shape of the object based on the camera parameters and a depth map of the object.

4. The control device according to claim 3, wherein: The processor controls the focus adjustment mechanism to stop the optical system at a near end and a far end, respectively.

5. The control device according to claim 3, wherein: The processor controls the focus adjustment mechanism to move the optical system linearly from the near end to the far end.

6. The control device according to claim 3, wherein: The processor controls the focus adjustment mechanism to stop the optical system at the near end and the far end for a certain period of time respectively.

7. A learned model generation device, comprising a processor, wherein: The processor acquires learning data from the camera device, and generates a learned model by learning using the learning data. The imaging device comprises: an optical system capable of changing focal length; an imaging element for exposing an image formed by the optical system to generate a static image; and a focus adjustment mechanism for moving the optical system along an optical axis direction of the optical system to change the focal length. The learning data is data obtained by combining a plurality of first images, a plurality of second images, a plurality of third images, and correct solution values ​​of depth maps related to distances from the optical system to each of a plurality of objects with the plurality of objects, respectively. The plurality of first images are images generated by the imaging element at the near side of the optical system capturing the plurality of objects respectively, the plurality of second images are images generated by the imaging element capturing the plurality of objects respectively while the optical system is moved between the near side and the far side by changing the focal length by the focus adjustment mechanism, and the plurality of third images are images generated by the imaging element at the far side of the optical system capturing the plurality of objects respectively. The learned model is a model that uses the first image, the second image, and the third image as input parameters and outputs a depth map indicating the distance from the optical system to the subject as an output parameter.

8. An endoscope system, wherein: The endoscope system includes: an endoscope capable of photographing the inside of a subject; and a control device including a processor capable of being connected to the endoscope. The endoscope comprises: an optical system capable of changing focal length; an imaging element for exposing an image formed by the optical system to generate a static image; and a focus adjustment mechanism for changing the focal length by moving the optical system along an optical axis direction of the optical system. The processor acquires a first image, a second image, and a third image from the endoscope, and derives a depth map of a subject by inputting the first image, the second image, and the third image as input parameters into a learned model, and outputs the depth map of the subject, wherein the depth map indicates a distance from the optical system to the subject, and the learned model is a model that outputs the depth map as an output parameter. The first image is an image generated by the imaging element on the near side of the optical system photographing the subject, the second image is an image generated by the imaging element photographing the subject while the optical system is moved between the near side and the far side by changing the focal length through the focus adjustment mechanism, and the third image is an image generated by the imaging element on the far side of the optical system photographing the subject.

9. A medical assistance method, executed by a control device having a processor, wherein: The medical assistance method comprises the following steps: The processor obtains a first image, a second image, and a third image from a camera device; The processor derives a depth map of the subject by inputting the first image, the second image, and the third image as input parameters to a learned model; as well as The processor outputs a depth map of the object. The depth map represents the distance from the optical system to the object, and the learned model is a model that outputs the depth map as an output parameter. The imaging device comprises: the optical system capable of changing the focal length; an imaging element for exposing an image formed by the optical system to generate a static image; and a focus adjustment mechanism for changing the focal length by moving the optical system along the optical axis direction of the optical system. The first image is an image generated by the imaging element on the near side of the optical system photographing the subject, the second image is an image generated by the imaging element photographing the subject while the optical system is moved between the near side and the far side by changing the focal length through the focus adjustment mechanism, and the third image is an image generated by the imaging element on the far side of the optical system photographing the subject.

10. A learning method, performed by a learned model generating device having a processor, wherein: The learning method comprises the following steps: The processor obtains learning data from the camera device; and The processor generates a learned model by learning using the learning data, The imaging device comprises: an optical system capable of changing focal length; an imaging element for exposing an image formed by the optical system to generate a static image; and a focus adjustment mechanism for moving the optical system along an optical axis direction of the optical system to change the focal length. The learning data is data obtained by combining a plurality of first images, a plurality of second images, a plurality of third images, and correct solution values ​​of depth maps related to distances from the optical system to each of a plurality of objects with the plurality of objects, respectively. The plurality of first images are images generated by the imaging element at the near side of the optical system capturing the plurality of objects respectively, the plurality of second images are images generated by the imaging element capturing the plurality of objects respectively while the optical system is moved between the near side and the far side by changing the focal length by the focus adjustment mechanism, and the plurality of third images are images generated by the imaging element at the far side of the optical system capturing the plurality of objects respectively. The learned model is a model that uses the first image, the second image, and the third image as input parameters and outputs a depth map indicating the distance from the optical system to the subject as an output parameter.

11. A non-transitory computer-readable recording medium having an executable program recorded thereon, wherein: The program performs the following processing: Acquire a first image, a second image, and a third image from a camera device; Deriving a depth map of the subject by inputting the first image, the second image, and the third image as input parameters into a learned model; as well as outputting a depth map of the subject, The depth map represents the distance from the optical system to the object, and the learned model is a model that outputs the depth map as an output parameter. The imaging device comprises: the optical system capable of changing the focal length; an imaging element for exposing an image formed by the optical system to generate a static image; and a focus adjustment mechanism for changing the focal length by moving the optical system along the optical axis direction of the optical system. The first image is an image generated by the imaging element on the near side of the optical system photographing the subject, the second image is an image generated by the imaging element photographing the subject while the optical system is moved between the near side and the far side by changing the focal length through the focus adjustment mechanism, and the third image is an image generated by the imaging element on the far side of the optical system photographing the subject.

12. A non-transitory computer-readable recording medium having an executable program recorded thereon, wherein: The program performs the following processing: Acquiring learning data from a camera device; and Generate a learned model by learning using the learning data, The imaging device comprises: an optical system capable of changing focal length; an imaging element for exposing an image formed by the optical system to generate a static image; and a focus adjustment mechanism for moving the optical system along an optical axis direction of the optical system to change the focal length. The learning data is data obtained by combining a plurality of first images, a plurality of second images, a plurality of third images, and correct solution values ​​of depth maps related to distances from the optical system to each of a plurality of objects with the plurality of objects, respectively. The plurality of first images are images generated by the imaging element at the near side of the optical system capturing the plurality of objects respectively, the plurality of second images are images generated by the imaging element capturing the plurality of objects respectively while the optical system is moved between the near side and the far side by changing the focal length by the focus adjustment mechanism, and the plurality of third images are images generated by the imaging element at the far side of the optical system capturing the plurality of objects respectively. The learned model is a model that uses the first image, the second image, and the third image as input parameters and outputs a depth map indicating the distance from the optical system to the subject as an output parameter.

Images