Endoscope system, image generation device, and image generation method

Abstract

An endoscope system is provided with: an endoscope having an illumination unit for irradiating an object with illumination light and an imaging unit for imaging the object; a control device that generates a captured image by performing image capture signal processing based on an image capture parameter on an image capture signal acquired from the image capture unit; and an image generation device, in which the control device generates a first captured image in which the subject illuminated by a light amount equal to or less than a limit value is captured, and a second captured image in which the subject illuminated by a light amount equal to or less than a limit value is captured. The second captured image is a captured image obtained by capturing the subject illuminated at a light amount equal to or less than the upper limit value and is brighter than the first captured image, and the image generation device generates an image on the basis of the first captured image and the second captured image, and generates a second captured image on the basis of the first captured image and the second captured image, the second captured image being an image of the subject illuminated at a light amount equal to or less than the upper limit value. And generating an estimated image in which an image of the subject illuminated with an amount of light greater than the upper limit value is captured.

Description

Technical Field

[0001] The present invention relates to an endoscope system, an image generating device and an image generating method. Background Art

[0002] The endoscope system is provided with an illumination unit and an imaging unit at the front end portion, and the illumination unit irradiates the subject with illumination light, and the imaging unit images the subject. It is desirable that the captured image is bright enough to visually confirm the subject in detail.

[0003] When the amount of illumination light emitted from the illumination unit is increased to make the camera image sufficiently bright, the upper limit of the amount of illumination light is set so that the temperature of the front end portion does not exceed the allowable range. On the other hand, when the camera image is made sufficiently bright by camera signal processing such as gain adjustment, noise caused by the camera signal processing that exceeds the allowable range may be generated.

[0004] Patent Document 1 describes an endoscope system that suppresses a temperature rise at the distal end portion of an endoscope by suppressing the emission amount of illumination light to an upper limit value in situations other than when acquiring a still image.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2020-116147 Summary of the invention

[0008] Problem that the invention aims to solve

[0009] However, the endoscope system described in Patent Document 1 may still cause the temperature of the endoscope distal end portion to exceed the allowable range when acquiring a still image.

[0010] Based on the above circumstances, an object of the present invention is to provide an endoscope system, an image generating device, and an image generating method that can suppress a temperature rise at a distal end portion of an endoscope even when acquiring a still image and can provide a sufficiently bright captured image.

[0011] Solutions for solving problems

[0012] In order to solve the above problems, the present invention proposes the following solutions.

[0013] The endoscope system involved in the first embodiment of the present invention comprises: an endoscope, which has an illumination unit for irradiating an object with illumination light and a camera unit for photographing the object; a control device, which performs camera signal processing based on camera parameters on a camera signal acquired from the camera unit to generate a camera image; and an image generating device, wherein the control device generates a first camera image and a second camera image, the first camera image being the camera image photographed when the object is illuminated with a light amount below an upper limit value, the second camera image being the camera image photographed when the object is illuminated with a light amount below the upper limit value and brighter than the first camera image, and the image generating device generates an estimated image that estimates an image of the object photographed when illuminated with a light amount greater than the upper limit value based on the first camera image and the second camera image.

[0014] The image generating device involved in the second mode of the present invention is an image generating device that obtains a camera image by performing camera signal processing based on camera parameters on a camera signal obtained from a camera unit that captures a subject, obtains a first camera image and a second camera image, the first camera image is an image captured by the subject illuminated with a light amount below an upper limit value, the second camera image is an image captured by the subject illuminated with a light amount below the upper limit value and is brighter than the first camera image, and an estimated image is generated based on the first camera image and the second camera image, which estimates the image of the subject captured by being illuminated with a light amount greater than the upper limit value.

[0015] In the image generation method involved in the third embodiment of the present invention, a first camera image and a second camera image are obtained, the first camera image is an image generated by performing camera signal processing on a camera signal of a subject illuminated with a light amount below an upper limit value, the second camera image is an image generated by performing camera signal processing on a camera signal of a subject illuminated with a light amount below the upper limit value, and the second camera image is brighter than the first camera image, and an estimated image that estimates an image of the subject illuminated with a light amount greater than the upper limit value is generated based on the first camera image and the second camera image.

[0016] Effects of the Invention

[0017] According to the endoscope system, the image generating device, and the image generating method of the present invention, even when a still image is acquired, a temperature rise at the distal end portion of the endoscope can be suppressed and a sufficiently bright captured image can be provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a diagram showing an endoscope system according to the first embodiment.

[0019] Figure 2 This is a functional block diagram of the endoscope system.

[0020] Figure 3 2 is a functional block diagram of the imaging control unit 21 of the endoscope system.

[0021] Figure 4 1 is a diagram showing the hardware configuration of the image generating device of the endoscope system.

[0022] Figure 5 This is a functional block diagram of the image generating device.

[0023] Figure 6 2 is a diagram showing the relationship between the first captured image and the second captured image.

[0024] Figure 7 FIG. 1 is a diagram for explaining the generation of a learned model of the endoscope system.

[0025] Figure 8 is a diagram showing the training data used to generate the learned model.

[0026] Fig. 9 It is the control flow chart of the endoscope system.

[0027] Fig.10 : is a diagram showing a modified example of the learned model.

[0028] Fig.11 FIG. 4 is a diagram showing another modified example of the learned model.

[0029] Fig.12 FIG. 1 is a diagram showing a moving image generated by the image generating unit.

[0030] Fig.13 The diagrams show captured images generated by an imaging control unit and still images generated by an image generating unit in the endoscope system according to the second embodiment.

[0031] Fig.14 It is the control flow chart of the endoscope system.

[0032] Fig.15 This is a functional block diagram of an endoscope system according to the third embodiment.

[0033] Fig.16 : is a diagram showing an image generating unit of the endoscope system.

[0034] Fig.17 3 is a diagram showing the relationship between the first special light captured image and the second special light captured image.

[0035] Fig.18FIG. 1 is a diagram showing a modified example of the learned model of the endoscope system.

[0036] Fig.19 is a diagram illustrating the generation of the learned model.

[0037] Fig. 20 is a diagram showing the training data used to generate the learned model.

[0038] Fig.21 It is a diagram showing an image generating unit of an endoscope system according to a fourth embodiment.

[0039] Fig. 22 FIG. 1 is a diagram for explaining the generation of a learned model of the endoscope system. DETAILED DESCRIPTION

[0040] (First Embodiment)

[0041] Reference Figures 1 to 9 An endoscope system 100 according to a first embodiment of the present invention will be described.

[0042] [Endoscope system 100]

[0043] Figure 1 2 is a diagram showing an endoscope system 100 .

[0044] The endoscope system (image generation system) 100 includes an endoscope 1, a control device 2, an image generation device 3, and a display device 4. The control device 2 and the image generation device 3 may be an integrated device (image control device). The display device 4 is a device that displays images generated by the control device 2 and the image generation device 3, various information related to the endoscope system 100, and the like.

[0045] [Endoscope 1]

[0046] The endoscope 1 is a device for observing and treating the body of a patient lying on an operating table T. The endoscope 1 includes an elongated insertion portion 10 inserted into the body of the patient, an operation portion 18 connected to the base end of the insertion portion 10 , and a universal wire 19 extending from the operation portion 18 .

[0047] The insertion section 10 includes a distal end portion 11, a bendable bending portion 12, and a long and flexible flexible tube portion 13. The distal end portion 11, the bending portion 12, and the flexible tube portion 13 are connected in order from the distal end side. The flexible tube portion 13 is connected to the operation portion 18.

[0048] Figure 2 is a functional block diagram of the endoscope system 100 .

[0049] The distal end portion 11 includes an imaging unit 14 , an illumination unit 15 , and a temperature sensor 17 .

[0050] The imaging unit 14 includes an optical system 141 and an imaging element 142 such as a CCD image sensor or a CMOS image sensor (see Figure 3 The imaging unit 14 captures the subject and generates an imaging signal based on the imaging parameter P transmitted from the imaging control unit 21 via the imaging control cable 143 . The imaging signal is transmitted to the control device 2 via the imaging signal cable 144 .

[0051] The illumination unit (white light illumination unit) 15 irradiates the subject with illumination light (white light) transmitted through the light guide 151. The light guide 151 is inserted through the insertion unit 10, the operation unit 18, and the universal cable 19 to be connected to the control device 2. In addition, the illumination unit 15 may also include a light source such as an LED, an optical element such as a fluorescent body having a wavelength conversion function, and the like.

[0052] The temperature sensor 17 is a sensor for detecting the temperature of the lighting unit 15 . The temperature sensor 17 includes, for example, a thermocouple, a thermistor, a temperature resistor, a temperature-sensitive ferrite, a thermal expansion thermometer, etc. The detected temperature of the lighting unit 15 is acquired by the control device 2 .

[0053] The operation unit 18 receives an operation of the endoscope 1. The operation unit 18 includes an escapement knob 181 for controlling the bending portion 12, an air and water supply button 182, a suction button 183, a release button 184, and an observation mode switching button 185. Operations input to the air and water supply button 182, the suction button 183, the release button 184, and the observation mode switching button 185 are acquired by the control device 2. The release button 184 is a push button for inputting an operation of saving a captured image acquired from the imaging unit 14.

[0054] The universal cord 19 connects the endoscope 1 and the control device 2. The universal cord 19 is a cable through which the imaging control cable 143, the imaging signal cable 144, the light guide 151, and the like are inserted.

[0055] [Control device 2]

[0056] The control device 2 is a device that controls the entire endoscope system 100. The control device 2 includes an imaging control unit 21, an illumination control unit 22, a control unit 24, and a recording unit 25.

[0057] Figure 3 This is a functional block diagram of the imaging control unit 21 .

[0058] The imaging control unit 21 performs imaging signal processing based on the imaging parameter P on the imaging signal acquired from the imaging unit 14 to generate a captured image (captured video) D. WThe imaging control unit 21 includes an imaging drive unit 211 , an analog signal processing unit 212 , an AD conversion unit 213 , and a digital signal processing unit 214 .

[0059] The imaging drive unit 211 drives the imaging element 142 of the imaging unit 14 based on the imaging parameter P from the control unit 24. The imaging drive unit 211 controls the exposure of the imaging element 142.

[0060] The analog signal processing unit 212 performs analog signal processing including noise removal and amplification on the imaging signal acquired from the imaging element of the imaging unit 14 based on the imaging parameters P (analog gain, ISO sensitivity, etc.) from the control unit 24 .

[0061] The AD conversion section 213 converts the imaging signal subjected to the analog signal processing by the analog signal processing section 212 into a digital signal (for example, RAW data) based on an instruction from the control section 24 .

[0062] The digital signal processing unit 214 generates a captured image by performing digital signal processing on the digital signal (eg, RAW data) converted by the AD conversion unit 213 based on the imaging parameter P from the control unit 24. The digital signal processing unit 214 performs digital signal processing such as adjusting brightness and contrast on the captured image as necessary.

[0063] The illumination control unit 22 includes a light source such as an LED, and controls the light source based on an imaging parameter P (light quantity, etc.) from the control unit 24 , thereby controlling the light quantity of illumination light transmitted to the illumination unit 15 via the light guide 151 .

[0064] The control unit 24 is a processing circuit (computer) capable of executing a program, including a processor and a memory capable of reading the program. The control unit 24 controls the endoscope system 100 by executing the endoscope control program. The control unit 24 may also include a dedicated circuit. The dedicated circuit refers to a processor independent of the processor of the control unit 24, a logic circuit installed in an ASIC or FPGA, or a combination thereof.

[0065] The control unit 24 specifies the operation parameters including the imaging parameter P to the imaging control unit 21 and the lighting control unit 22, thereby controlling the imaging control unit 21 and the lighting control unit 22 to generate the captured image D. W In addition, the control unit 24 instructs the image generating device 3 to generate an image and instructs the display device 4 to display the image.

[0066] The recording unit 25 is a non-volatile recording medium that stores the above-mentioned program and required data. The recording unit 25 is composed of, for example, a non-volatile memory that can be written to, such as a floppy disk, a magneto-optical disk, a ROM, a flash memory, a removable medium such as a CD-ROM, a storage device such as a hard disk built into the computer system, etc. In addition, the recording unit 25 can also be a storage device set on a cloud server connected to the control device 2 via the Internet.

[0067] [Image generation device 3]

[0068] The image generating device 3 generates a captured image (camera video) based on the captured image (camera video) outputted by the imaging control unit 21 of the control device 2. W Generate a new estimated image E W , and generate the estimated image E W The device displayed on the display device 4. When there is no need to generate a new estimated image E W In this case, the image generating device 3 displays the captured image output by the imaging control unit 21 of the control device 2 on the display device 4.

[0069] Figure 4 It is a diagram showing the hardware configuration of the image generating device 3 .

[0070] The image generating device 3 includes a processor 301, a memory 302 capable of reading a program, a storage unit 303, and an input / output control unit 304. The image generating device 3 is a computer capable of executing a program. The functions of the image generating device 3 are realized by executing the program by the processor 301. At least a part of the functions of the image generating device 3 may also be realized by a dedicated logic circuit installed in an ASIC or FPGA.

[0071] The storage unit 303 is a nonvolatile recording medium that stores the above-mentioned program and necessary data. The storage unit 303 is composed of, for example, a ROM, a hard disk, etc. The program recorded in the storage unit 303 is read into the memory 302 and executed by the processor 301.

[0072] The input / output control unit 304 is connected to the control device 2, the display device 4, an input device (not shown) and a network device (not shown). Based on the control of the processor 301, the input / output control unit 304 sends and receives data and control signals to and from the connected devices.

[0073] The image generation device 3 is not limited to an integrated hardware device. For example, the image generation device 3 may be configured by connecting the separated hardware devices using a communication line on the basis of separating a part as an independent hardware device. For example, the image generation device 3 may also be a cloud system in which the separated storage unit 303 is connected using a communication line.

[0074] The image generation device 3 may also include a structure other than the processor 301, the memory 302, the storage unit 303, and the input / output control unit 304. For example, the image generation device 3 may also include an image operation unit that performs a part or all of the image processing and image recognition processing. By further including the image operation unit, the image generation device 3 can perform specific image processing and image recognition processing at high speed. For example, the image generation device 3 may also include an image processing unit that performs the estimated image E described later. W By further including the inference operation unit, the image generation device 3 can execute the estimated image E at high speed. W The image operation unit and the inference operation unit may be mounted on independent hardware devices connected by a communication line.

[0075] Figure 5 This is a functional block diagram of the image generating device 3 .

[0076] The image generating device 3 includes an image generating unit 31 , an image display unit 32 , and an image storage unit 33 as functional blocks.

[0077] The image generation unit 31 generates a first captured image (first white light captured image) D input from the control device 2 based on the learned model M. W1 and the second camera image (second white light camera image) D W2 To generate an estimated image (white light estimated image) E W .

[0078] Figure 6 The first camera image D W1 With the second camera image D W2 A diagram of the relationship between .

[0079] First camera image D W1 The second captured image D is a captured image of a subject illuminated by a light quantity that is less than the upper limit value U. W2 The first camera image D is a camera image captured by a subject illuminated by a light quantity below the upper limit value U, and is different from the first camera image D due to the camera signal processing based on the camera parameter P (the analog signal processing performed by the analog signal processing unit 212 and / or the digital signal processing performed by the digital signal processing unit 214). W1 Compared to the camera image, the brightness is increased with the increase of noise. The “brightness of the camera image” refers to, for example, 'brightness', 'luminance', and 'luminosity'. In the following description, the first camera image D W1 The brightness of the second camera image D is called the "first brightness B1". W2The brightness of is called "second brightness B2".

[0080] First camera image D W1 and the second camera image D W2 The first camera image D is a captured image of a subject illuminated with substantially the same amount of light. W1 and the second camera image D W2 The images do not necessarily need to be captured images of a subject illuminated with substantially the same amount of light.

[0081] Desired first camera image D W1 and the second camera image D W2 The first captured image D is a captured image of a subject illuminated by a light amount that is less than the upper limit value U and closer to the upper limit value U. By making the light amount closer to the upper limit value U, the first captured image D W1 and the second camera image D W2 The estimated image E can be improved. W In particular, through the second camera image D W2 The noise of the estimated image E is reduced, which can effectively improve W The estimation accuracy.

[0082] Second camera image D W2 It is from the time when the first camera image D is captured W1 The first camera image D is captured after a period T1 has passed. Since the period T1 is a very short period, the first camera image D W1 and the second camera image D W2 The subject photographed in the first camera image D is in substantially the same state. W1 and the second camera image D W2 There is no restriction on the order.

[0083] First camera image D W1 and the second camera image D W2 is the input of the learned model M. In order to improve the estimated image E output by the learned model M W The estimation accuracy of the first camera image D W1 The information contained in the second camera image D W2 Therefore, it is desirable to generate the first camera image D W1 The camera parameters P and the second camera image D are generated W2 The camera parameters P are more different.

[0084] The image signal processing includes, for example, analog signal amplification processing and digital signal amplification processing. In this case, the second image D is generated.W2 The gain of the camera parameter P is greater than the gain of the first camera image D W1 The gain of the imaging parameter P.

[0085] Compared with the amplification of digital signals, the amplification of analog signals is more suitable for improving the estimated image E. W This is because: since the amplification process of the digital signal is performed after the AD conversion in which part of the information of the camera signal is lost, the first camera image D is enlarged. W1 The information contained in the second camera image D W2 On the other hand, compared with the amplification processing of analog signals, the amplification processing of digital signals is more suitable for implementing various and flexible imaging signal processing. Regarding the amplification processing of digital signals, for example, imaging signal processing based on the gains of two imaging parameters P can be performed on a type of imaging signal acquired from the imaging unit 14 to generate a first imaging image D. W1 and the second camera image D W2 .

[0086] Estimated image E W Based on the learned model M and the first camera image D W1 and the second camera image D W2 The generated image is an image that estimates an image of a subject illuminated with a light intensity greater than the upper limit value U.

[0087] The image display unit 32 displays the estimated image E generated by the image generating unit 31. W Output to the display device 4 so that the estimated image E W Displayed on the display device 4.

[0088] The image storage unit 33 stores the estimated image E generated by the image generation unit 31. W In addition, there is no need to record the estimated image E W In this case, the image storage unit 33 is not required.

[0089] [Learning Completed Model M]

[0090] The learned model M is a machine learning model that learns the relationship between an input image and an output image, and is, for example, a machine learning model suitable for image generation, such as a neural network, a simple perceptron, or a multilayer perceptron. The learned model M is generated in advance by supervised learning based on training data. The generation of the learned model M can be implemented by the image generation device 3, or by using another computer with a higher computing power than the image generation device 3.

[0091] Figure 7 2 is a diagram for explaining the generation of the learned model M.

[0092] The training data is the first camera image D used for learning TW1 , the second camera image D for learning TW2 and the third camera image D for learning TW3 As a set of images, it is desirable that the training data contain as many images as possible taken under a variety of conditions.

[0093] The learned model M is based on the input first learning image D TW1 and the second camera image D for learning TW2 To generate the estimated image E W The learned model M is trained to learn the parameters of the learned model M so that the output estimated image E W and the third camera image D for learning TW3 The difference becomes smaller.

[0094] Figure 8 FIG. 1 is a diagram showing training data for generating a learned model M. FIG.

[0095] The first camera image D for learning TW1 This is the first camera image D prepared for learning. W1 The second camera image D is used for learning TW2 The second camera image D is prepared for learning W2 .

[0096] Learning the third camera image D TW3 is a captured image of a subject actually illuminated with a light quantity greater than the upper limit value U. In addition, the third captured image D for learning TW3 The noise generated by the imaging signal processing based on the imaging parameter P (the analog signal processing performed by the analog signal processing unit 212 and / or the digital signal processing performed by the digital signal processing unit 214) and the first imaging image D for learning are TW1 For example, the third camera image D is used for learning. TW3 is the imaging parameter P other than the light intensity and the first imaging image D for learning TW1 The imaging parameters P are roughly the same.

[0097] Learning the third camera image D TW3 The first camera image D is used from the shooting to the learning TW1 and the second camera image D for learning TW2 The first camera image D for learning is captured after a period T2 has passed. Since the period T2 is a very short period, the first camera image D for learning TW1 , the second camera image D for learning TW2and the third camera image D for learning TW3 The subject photographed in the above is in substantially the same state. TW1 , the second camera image D for learning TW2 and the third camera image D for learning TW3 There is no restriction on the order of TW1 , the second camera image D for learning TW2 and the third camera image D for learning TW3 It may also be an image of a stationary subject. In this case, the first camera image D for learning TW1 , the second camera image D for learning TW2 and the third camera image D for learning TW3 The subject photographed in the same state.

[0098] The first camera image D for desired learning TW1 Brightness and the second camera image D for learning TW2 The brightness of the first brightness B1 and the second brightness B2 are roughly the same. This is because the easy-to-learn model M efficiently learns the first camera image D W1 With the second camera image D W2 The relationship between.

[0099] The second camera image D for desired learning TW2 The first camera image D is used from the shooting to the learning TW1 This is because the model M that is easy to learn can efficiently learn the first camera image D W1 With the second camera image D W2 The relationship between.

[0100] By using the above-mentioned training data for supervised learning, a first camera image D is generated in advance. W1 and the second camera image D W2 With the estimated image E W The relationship between the first camera image D is input. W1 and the second camera image D W2 The estimated image E output by the learned model M W The noise contained in the first camera image D W1 The noise generated by camera signal processing is of the same level.

[0101] [Operation of the Endoscope System 100]

[0102] Next, the operation (image generation method) of the endoscope system 100 will be described. Fig. 9 The control flow chart of the endoscope system 100 shown in FIG. 1 is used to describe the control flow chart of the endoscope system 100. When the control unit 24 of the control device 2 detects that the operator has pressed the release button 184, the endoscope system 100 executes step S110.

[0103] <Step S110: Imaging Parameter Adjustment Process>

[0104] In step S110, the control unit 24 of the control device 2 calculates the best imaging parameters P (including light intensity and analog gain) for imaging the subject based on the image captured by the imaging control unit. A known imaging adjustment technique is used to calculate the best imaging parameters P. The endoscope system 100 then executes step S120.

[0105] <Step S120: Light Amount Determination Step>

[0106] In step S120, the control unit 24 of the control device 2 determines whether the calculated light amount exceeds the upper limit U. If the calculated light amount exceeds the upper limit U, the control unit 24 of the control device 2 next executes step S130. If the calculated light amount is equal to or less than the upper limit U, the endoscope system 100 next executes step S160.

[0107] In addition, the upper limit value U may not be a constant value, but a value that changes based on the temperature of the illumination unit 15 acquired by the temperature sensor 17 of the endoscope 1. For example, when the temperature of the illumination unit 15 is lower than a predetermined temperature, the control unit 24 may increase the upper limit value U. In addition, when the temperature of the illumination unit 15 is higher than a predetermined temperature, the control unit 24 may reduce the upper limit value U. In addition, when the endoscope 1 does not have the temperature sensor 17, the control unit 24 may use a temperature estimated from the integrated light amount obtained by multiplying the light amount by the light emission time.

[0108] <Step S130: First Camera Image Generating Step>

[0109] In step S130, the control unit 24 of the control device 2 sets the light intensity of the imaging parameter P to be less than the upper limit value U, and generates a first captured image D W1 The endoscope system 100 then executes step S140 .

[0110] <Step S140: Second Camera Image Generating Step>

[0111] In step S140, the control unit 24 of the control device 2 changes the imaging parameters P except the light intensity, and generates the first captured image D W1 Compared with the second camera image D, which has increased noise but increased brightness W2For example, the control unit 24 increases the analog gain of the imaging parameter P and generates the second imaging image D W2 The control unit 24 of the control device 2 takes the first camera image D W1 After a period T1 has passed, a second camera image D is generated. W2 The endoscope system 100 then executes step S150 .

[0112] <Step S150: Estimated Image Generating Step>

[0113] In step S150, the image generation unit 31 of the image generation device 3 generates the first captured image D input from the image capture control unit 21 of the control device 2 based on the learned model M. W1 and the second camera image D W2 To generate the estimated image E W The image storage unit 33 stores the estimated image E as needed. W The endoscope system 100 then executes step S170 .

[0114] <Step S160: First Camera Image Generating Step>

[0115] In step S160, the control unit 24 and the imaging control unit 21 of the control device 2 generate a first imaging image D W1 Since the light amount calculated in step S110 is less than the upper limit value U, the control unit 24 of the control device 2 sets the light amount to the calculated light amount and generates the first captured image D W1 The endoscope system 100 then executes step S170 .

[0116] <Step S170: Image display process>

[0117] The image display unit 32 of the image generating device 3 displays the estimated image E generated in step S150. W or the first camera image D captured in step S160 W1 The display is displayed on the display device 4. The endoscope system 100 then executes step S180.

[0118] <Step S180: End of determination process>

[0119] In step S180, the control unit 24 of the control device 2 determines whether a pressing operation of the release button 184 is input by the surgeon or the like. When the release button 184 is pressed, the control unit 24 of the control device 2 performs step S110 again.

[0120] According to the endoscope system 100 of this embodiment, even when acquiring a still image, a temperature rise of the distal end portion 11 of the endoscope 1 can be suppressed, and a sufficiently bright captured image can be provided. Specifically, the endoscope system 100 uses only the first captured image D captured of the subject illuminated with a light intensity below the upper limit value U. W1 and the second camera image D W2 Therefore, the light amount of the illumination light emitted from the illumination unit 15 can be suppressed to be below the upper limit value U, and the temperature of the distal end portion 11 of the endoscope 1 can be suppressed to be below the predetermined temperature. On the other hand, the image generating device 3 of the endoscope system 100 can generate the first camera image D based on the first camera image of the subject illuminated with the light amount below the upper limit value U. W1 and the second camera image D W2 , to estimate an estimated image E that estimates an image of a subject illuminated with a light intensity greater than the upper limit value U W That is, the endoscope system 100 can suppress the light amount of the illumination light emitted from the illumination unit 15 to be below the upper limit value U and can present the estimated image E to the user. W , the estimated image E W is larger than the first camera image D W1 The image is bright and contains noise similar to the first camera image D W1 The noise generated by camera signal processing is of the same level.

[0121] The first embodiment of the present invention has been described in detail above with reference to the drawings, but the specific structure is not limited to this embodiment, and design changes within the scope of the present invention are also included. In addition, the components shown in the above embodiment and the modified examples can be appropriately combined and configured.

[0122] (Variant 1)

[0123] Fig.10 FIG. 1 is a diagram showing a learned model M1 which is a modified example of the learned model M. FIG.

[0124] The learned model M1 uses at least a portion of the imaging parameter P (eg, simulation gain) as input data. For example, the learned model M1 also has the first imaging parameter P W1 and the second camera parameter P W2 As input, the first camera parameter P W1 The first camera image D is captured W1 The second imaging parameter P W2 The second camera image D is captured W2By adding at least a part of the imaging parameter P as the input of the learned model M1, the image generation unit 31 can improve the accuracy of the estimated image E. W The imaging parameter P is also added as an input to the training data used for learning the learned model M1.

[0125] (Variant 2)

[0126] Fig.11 FIG. 1 is a diagram showing a learned model M2 as another modified example of the learned model M. FIG.

[0127] The learned model M2 uses the additional auxiliary camera image as input data. For example, the learned model M2 also has W1 The first auxiliary image S is then captured without changing the imaging parameters P other than the light intensity and with the light intensity set to zero. W1 , and then the second camera image D is captured W2 The second auxiliary image S is then captured without changing the imaging parameters P other than the light intensity and with the light intensity set to zero. W2 By adding the auxiliary camera image captured when the light intensity is set to zero as the input of the learned model M2, it is possible to generate an estimated image E that is less susceptible to the influence of pattern noise of the imaging element 142 of the imaging unit 14. W In particular, when a CMOS image sensor, which is difficult to remove noise by correlated double sampling, is used as the imaging element 142, an estimated image E can be generated with higher accuracy by adding an auxiliary imaging image captured with the light amount set to zero as input. W The auxiliary camera images are also added as input to the training data used for learning the learned model M2.

[0128] (Variant 3)

[0129] In the above embodiment, the endoscope system 100 detects the pressing operation of the release button 184 to estimate an estimated image E W However, the endoscope system 100 may continuously generate the estimated image E regardless of whether the release button 184 is pressed (ON). W , thereby generating a continuous estimated image E W A moving image of a frame. Fig.12 is a diagram showing a moving image (continuous estimated image E) generated by the image generation unit 31. W The endoscope system 100 generates an estimated image E by executing a series of steps from step S110 to step S170. W, a series of steps are repeated to generate a motion image (continuous estimated image E W In order to generate a smooth moving image, it is desirable that the endoscope system 100 has a function of generating the estimated image E at, for example, 30 FPS. W processing performance.

[0130] The endoscope system 100 generates an estimated image E each time. W The light amount is adjusted in step S110. Therefore, the light amount changes during the process of generating the moving image. The desired light amount is a value below the upper limit value U and closer to the upper limit value U, but can be set to a value lower than the upper limit value U based on the adjustment result of step S110. In addition, Fig.12 The moving image generation sequence shown is a moving image generation sequence when the light amount calculated in step S120 exceeds the upper limit value U. When the light amount calculated in the process of generating the moving image is less than the upper limit value U, the endoscope system 100 generates the first captured image D in step S160. W1 , and the generated first camera image D W1 Set to one frame of the moving image.

[0131] The endoscope system 100 can suppress the light amount of the illumination light emitted from the illumination unit 15 to be below the upper limit value U and can present the estimated image E to the user. W The continuous motion image, the estimated image E W is larger than the first camera image D W1 The image is bright and contains noise similar to the first camera image D W1 The noise generated by camera signal processing is of the same level.

[0132] (Variant 4)

[0133] In the above-mentioned embodiment, the illumination unit irradiates the subject with white light, however, the illumination light irradiated by the illumination unit is not limited thereto. For example, the illumination unit may irradiate the subject with special light described later.

[0134] (Second Embodiment)

[0135] Reference Figure 13 to Figure 14 The second embodiment of the present invention is described. The endoscope system 100B according to the second embodiment differs from the endoscope system 100 according to the first embodiment only in the control flow. In the following description, the same reference numerals are used for the same configurations as those already described, and duplicate descriptions are omitted.

[0136] The endoscope system 100B displays a captured video (moving image) on the display device 4 , and generates a captured image (still image) when the control unit 24 of the control device 2 detects that the release button 184 is pressed (ON) by the operator. Fig.13 The captured image generated by the image capture control unit 21 and the still image (estimated image E) generated by the image generation unit 31 are shown. W ). Fig.14 When the endoscope system 100B is activated, the endoscope system 100B executes step S210.

[0137] <Step S210: Moving Image Generating Process>

[0138] In step S210, the control unit 24 of the control device 2 calculates the best imaging parameters P (including light intensity and analog gain) for imaging the subject, and continuously generates the second imaging image D. W2 For example, the control unit 24 of the control device 2 generates the second camera image D at 30 FPS. W2 .like Fig.13 As shown in FIG. 1 , the control unit 24 of the control device 2 adjusts the light intensity within a range below the upper limit value U. The control unit 24 of the control device 2 makes the second captured image D W2 The image is displayed on the display device 4 as a captured image (moving image).

[0139] <Step S220: Release Detection Process>

[0140] In step S220, the control unit 24 of the control device 2 detects the pressing operation of the release button 184. Unless the pressing operation of the release button 184 is detected, the control unit 24 of the control device 2 proceeds to step S210. When the pressing operation of the release button 184 is detected, the endoscope system 100B next executes step S230.

[0141] <Step S230: First Camera Image Generating Step>

[0142] The control unit 24 of the control device 2 sets the light amount of the imaging parameter P to be less than the upper limit value U, similarly to step S130 of the first embodiment, and generates the first captured image D W1 For example, the control unit 24 temporarily reduces the analog gain of the imaging parameter P and generates the first imaging image D W1 It is expected that the control unit 24 generates a first camera image D W1 When the control unit 24 generates a first captured image D W1 Then, the imaging control unit 21 is caused to continuously generate the second captured image DW2 and the continuously generated second camera image D W2 The image is displayed as a captured image (moving image) on the display device 4. The endoscope system 100B then executes step S240.

[0143] <Step S240: Estimated Image Generation Step>

[0144] The image generation unit 31 of the image generation device 3 generates the first captured image D generated in step S230 based on the learned model M. W1 And when generating the first camera image D W1 The second camera image D generated before and after W2 To generate the estimated image E W The image storage unit 33 stores the estimated image E as needed. W The endoscope system 100B then executes step S250 .

[0145] <Step S250: Image display process>

[0146] The image display unit 32 of the image generating device 3 displays the estimated image E generated in step S240. W The image display unit 32 of the image generating device 3 may display the second captured image D as a moving image. W2 and the estimated image E W The image display unit 32 of the image generating device 3 may display the estimated image E only during a predetermined period. W The second camera image D displayed as a moving image W2 The interchange method is displayed on the display device 4. The endoscope system 100 then executes step S260.

[0147] <Step S260: End of determination process>

[0148] In step S180, the control unit 24 of the control device 2 determines whether to end the moving image display. If the moving image display is not to be ended, the control unit 24 of the control device 2 performs step S210 again.

[0149] The endoscope system 100B may also continuously generate the estimated image E as in the third modification. W , and generate continuous estimated images E W A moving image of a frame.

[0150] According to the endoscope system 100B according to the present embodiment, even when a still image is acquired during acquisition of a captured video (moving image), a temperature rise in the distal end portion 11 of the endoscope 1 can be suppressed and a sufficiently bright captured image (still image) can be provided.

[0151] The second embodiment of the present invention has been described in detail above with reference to the drawings, but the specific structure is not limited to this embodiment, and design changes within the scope of the present invention are also included. In addition, the components shown in the above embodiment and the modified example can be appropriately combined and configured.

[0152] (Variant 5)

[0153] In the above-described embodiment, the endoscope system 100B generates the second captured images D W2 As a captured image (moving image), a first captured image D generated in response to the pressing operation of the release button 184 is used. W1 Generate estimated image E W As the captured image (still image). However, the method of generating the captured image (moving image) and the still image is not limited to this. The endoscope system 100B may also generate the first captured image D W1 As a captured image (moving image), a second captured image D generated in response to the pressing operation of the release button 184 is used. W2 Generate estimated image E W as a camera image (still image).

[0154] (Third Embodiment)

[0155] Reference Figures 15 to 17 The third embodiment of the present invention is described. Compared with the endoscope system 100 according to the first embodiment, the endoscope system 100C according to the third embodiment can use two observation modes. In the following description, the same reference numerals are attached to the same configuration as the configuration already described, and repeated description is omitted.

[0156] Fig.15 100C is a functional block diagram of the endoscope system 100C.

[0157] The endoscope system 100C includes an endoscope 1C, a control device 2C, an image generating device 3C, and a display device 4 .

[0158] The endoscope 1C is the same as the endoscope 1 of the first embodiment except for the distal end portion 11C. The distal end portion 11C of the endoscope 1C includes an imaging unit 14 , an illumination unit 15 , a special light illumination unit 16 , and a temperature sensor 17 .

[0159] The special light illumination unit 16 irradiates the subject with special light used in NBI (registered trademark. Narrow Band Imaging), RDI (registered trademark. Red Dichromatic Imaging), etc. Here, the special light irradiated by the special light illumination unit 16 is light having a wavelength range different from that of white light. The special light used in NBI is narrow-band light in two wavelength ranges of blue (wavelength of 390nm to 445nm) and green (wavelength of 530nm to 550nm). The special light used in RDI is narrow-band light in three wavelength ranges of red, amber, and green.

[0160] The control device 2C is a device that controls the entire endoscope system 100C and includes an imaging control unit 21C, an illumination control unit 22 , a special light illumination control unit 23 , a control unit 24C, and a recording unit 25 .

[0161] The imaging control unit 21C has the same function as the imaging control unit 21 of the first embodiment, and can generate a special light captured image (special light captured video) D by performing imaging signal processing based on imaging parameters P on an imaging signal of an object irradiated with special light. S .

[0162] The special light illumination control unit 23 controls the light source of the special light based on the imaging parameter P (the light amount of the special light, etc.) from the control unit 24 , thereby controlling the light amount of the special light transmitted to the special light illumination unit 16 .

[0163] The control unit 24C has the same functions as the control unit 24 of the first embodiment, and can switch between the two observation modes (white light observation mode and special light observation mode) based on the operation input of the observation mode switching button 185. In the white light observation mode, the control unit 24C generates a captured image (white light captured image) based on the imaging signal of the subject irradiated with white light from the illumination unit 15. W In the special light observation mode, the control unit 24C generates a special light captured image D based on an image pickup signal of a subject irradiated with special light from the special light illumination unit 16. S .

[0164] The image generating device 3C includes an image generating unit 31C, an image display unit 32 , and an image storage unit 33 as functional blocks.

[0165] Fig.16 It is a diagram showing the image generating unit 31C.

[0166] The image generation unit 31C generates an image based on the learned model MC according to the first captured image D input from the control device 2C.W1 , second camera image D W2 , first special light photographed image D S1 and the second special light photographed image D S2 To generate the estimated image E W .

[0167] Fig.17 The first special light image D S1 and the second special light image D S2 The first special light image D S1 The second special light captured image D is a special light captured image of a subject illuminated by special light of a light quantity less than the upper limit value U. S2 The special light image D is a special light image captured of a subject illuminated by special light of a light quantity below the upper limit value U, and is different from the first special light image D due to image signal processing based on the image parameter P (analog signal processing performed by the analog signal processing unit 212 and / or digital signal processing performed by the digital signal processing unit 214). S1 Compared to special light images, the noise is increased but the brightness is increased.

[0168] Desired first special light captured image D S1 and the second special light photographed image D S2 The first special light captured image D is a captured image of a subject illuminated with a light amount that is less than the upper limit value U and closer to the upper limit value U. By making the light amount closer to the upper limit value U, the first special light captured image D S1 and the second special light photographed image D S2 The estimated image E can be improved. W In particular, the second special light camera image D S2 The noise of the estimated image E is reduced, which can effectively improve W The estimation accuracy.

[0169] First special light captured image D S1 It is from the time when the first camera image D is captured W1 The first camera image D is a special light image captured after a period T3 has passed. Since the period T3 is a very short period, the first camera image D W1 and the first special light photographed image D S1 The subject photographed in the first camera image D is in substantially the same state. W1 and the first special light photographed image D S1 There is no restriction on the order.

[0170] Second special light photographed image D S2In the period from the time of capturing to the second camera image D W2 The special light image D is captured after a period T4 has passed. Since the period T4 is a very short period, the second image D W2 and the second special light photographed image D S2 In addition, the second camera image D is captured. W2 and the second special light photographed image D S2 There is no restriction on the order.

[0171] The learned model MC further includes a first special light captured image D as compared to the learned model M of the first embodiment. S1 and the second special light photographed image D S2 The special light captured image is also added as input to the training data used for learning the learned model MC.

[0172] When the control unit 24C of the control device 2C detects that the operator has pressed the release button 184, the endoscope system 100C generates the first captured image D W1 , second camera image D W2 , first special light photographed image D S1 and the second special light photographed image D S2 , and generate an estimated image E based on these four camera images W .

[0173] The endoscope system 100C may also continuously generate the estimated image E as in the third modification. W , and generate continuous estimated images E W The endoscope system 100C may also generate the second captured image D continuously as in the second embodiment. W2 As the captured image (moving image), the first captured image D generated in response to the pressing operation of the release button 184 is used. W1 , first special light photographed image D S1 and the second special light photographed image D S2 Generate estimated image E W as a camera image (still image).

[0174] According to the endoscope system 100C of the present embodiment, the light amount of the illumination light emitted from the illumination unit 15 and the special light illumination unit 16 can be suppressed to be below the upper limit value U, and the estimated image E can be presented to the user. W , the estimated image E W is larger than the first camera image D W1 The image is bright and contains noise similar to the first camera image D W1The noise generated by camera signal processing is of the same level.

[0175] The special light camera image obtained by the camera signal of the subject illuminated by the special light is added as the input of the learned model MC. Since the two camera images obtained by photographing the subject illuminated by two kinds of illumination light (white light and special light) have different characteristics, the image generation unit 31C can preferably improve the estimated image E. W The estimation accuracy.

[0176] The third embodiment of the present invention has been described in detail above with reference to the drawings, but the specific structure is not limited to this embodiment, and design changes within the scope of the present invention are also included. In addition, the components shown in the above embodiments and modifications can be appropriately combined and configured.

[0177] (Variant 6)

[0178] Fig.18 1 is a diagram showing a learned model MC1 which is a modified example of the learned model MC.

[0179] After learning, the model MC1 outputs the special light estimation image E S .like Fig.17 As shown, the special light estimation image E S is larger than the first special light captured image D S1 The image is bright, and the noise contained in the image is different from the first special light image D S1 In addition, the learned model MC1 can also output an estimated image E W and the special light estimation image E S These two parties.

[0180] Fig.19 This is a diagram for explaining the generation of the learned model MC1.

[0181] The training data is the first camera image D used for learning TW1 , learning the image D captured using the first special light TS1 , the second camera image D for learning TW2 , Learning the image D captured with the second special light TS2 and learning the image D captured using the third special light TS3 An image group that is a set of images.

[0182] Fig. 20 : is a diagram showing training data for generating the learned model MC1.

[0183] Learning the image D captured using the first special light TS1The first special light image D prepared for learning S1 . Learning the image D captured with the second special light TS2 This is the second special light image D prepared for study. S2 .

[0184] Learning to take images D using the third special light TS3 is a captured image actually captured of a subject illuminated by special light having a light quantity greater than the upper limit value U. In addition, the third special light captured image D for learning TS3 The noise generated by the imaging signal processing based on the imaging parameter P (the analog signal processing performed by the analog signal processing unit 212 and / or the digital signal processing performed by the digital signal processing unit 214) and the first special light imaging image D for learning are TS1 For example, the image D captured by the third special light is learned. TS3 is the imaging parameter P other than the light intensity and the first special light imaging image D for learning TS1 The imaging parameters P are roughly the same as the images.

[0185] (Variant 7)

[0186] In the above-described embodiment, the endoscope 1C has two observation modes. However, when the endoscope 1C has three or more observation modes, the learned model MC may be capable of inputting three or more types of captured images.

[0187] (Fourth Embodiment)

[0188] Reference Figure 21 to Figure 22 The fourth embodiment of the present invention is described. An endoscope system 100D according to the fourth embodiment can use two observation modes similarly to the endoscope system 100C according to the third embodiment. In the following description, the same reference numerals are used for the same configurations as those already described, and duplicate descriptions are omitted.

[0189] Compared with the endoscope system 100C according to the third embodiment, the endoscope system 100D includes an image generating unit 31D instead of the image generating unit 31C.

[0190] Fig.21 It is a diagram showing the image generating unit 31D.

[0191] The image generating unit 31D generates the first special light captured image D input from the control device 2C based on the learned model MD. S1 and the second camera image D W2 Generate special light estimation image E S .

[0192] Fig. 22 This is a diagram for explaining the generation of the learned model MD.

[0193] The training data is the first special light image D TS1 , the second camera image D for learning TW2 and learning the image D captured using the third special light TS3 As a set of images, it is desirable that the training data contain as many images as possible taken under a variety of conditions.

[0194] The learned model MD is based on the input first special light image D for learning. TS1 and the second camera image D for learning TW2 To generate the special light estimation image E S The learned model MD is used to learn the parameters of the learned model MD so that the output special light estimated image E S Image D captured with the third special light TS3 The difference becomes smaller.

[0195] Learning to take images D using the third special light TS3 The first special light is used to capture the image D from the shooting to the learning. TS1 and the second camera image D for learning TW2 Since the period T5 is a very short period, the learning first special light image D TS1 , the second camera image D for learning TW2 and learning the image D captured using the third special light TS3 The subject photographed in the above is in substantially the same state. In addition, the first special light image D for learning is photographed. TS1 , the second camera image D for learning TW2 and learning the image D captured using the third special light TS3 There is no restriction on the order.

[0196] When the control unit 24C of the control device 2C detects that the operator has pressed the release button 184, the endoscope system 100D generates the first special light captured image D. S1 and the second camera image D W2 , and generate the special light estimation image E based on these two camera images S .

[0197] The endoscope system 100D may also continuously generate the special light estimated image E as in Modification 3. S , and generate a continuous special light estimation image E SThe endoscope system 100D may also generate the second captured image D continuously as in the second embodiment. W2 As a captured image (moving image), a first special light captured image D generated in response to a pressing operation of the release button 184 is used. S1 Generate special light estimation image E S as a camera image (still image).

[0198] According to the endoscope system 100D according to the present embodiment, the light amount of the illumination light emitted from the illumination unit 15 and the special light illumination unit 16 can be suppressed to be below the upper limit value U, and the special light estimated image E can be presented to the user. S , the special light estimated image E S is larger than the first special light captured image D S1 The image is bright, and the noise contained in the image is similar to the first special light image D S1 The noise generated by the image signal processing is of the same degree. Compared with the endoscope system 100C of the third embodiment, the endoscope system 100D needs fewer image pickup images to generate the estimated image. Therefore, when generating a moving image with the estimated image as a frame as in the modification example 3, the frame rate can be further increased.

[0199] Sometimes, a special light camera image may be darker than a white light camera image, depending on the type of special light. For example, a special light camera image acquired through NBI is darker than a white light camera image. The special light used in NBI is more easily absorbed by the human tissue that is the subject than white light, and is less likely to be reflected by the human tissue. Therefore, when the intensity of the irradiated light is the same, the light intensity received by the imaging element 142 becomes weaker when the special light is irradiated onto the subject than when the white light is irradiated onto the subject. Therefore, when a special light camera image is acquired through NBI, it is difficult to provide a sufficiently bright special light camera image in applications where a white light camera image and a special light camera image are used in combination (for example, a side-by-side display of a white light camera image and a special light camera image). However, the endoscope system 100D is capable of providing a sufficiently bright special light camera image based on the first special light camera image D S1 and the second camera image D W2 To generate and provide a first special light image D S1 Brighter special light estimation image E S .

[0200] The fourth embodiment of the present invention has been described in detail above with reference to the drawings, but the specific structure is not limited to this embodiment, and design changes within the scope of the present invention are also included. In addition, the components shown in the above embodiments and modifications can be appropriately combined and configured.

[0201] (Variant 8)

[0202] In the above-mentioned embodiment, the imaging unit 14 that generates imaging signals is provided in the endoscope 1, and the image generating devices 3 and 3C generate the estimated image E based on the imaging images acquired from the imaging unit 14 of the endoscope 1. However, the imaging signal acquisition source of the image generating device is not limited to the endoscope 1. The image generating device may generate the estimated image E based on the imaging images acquired from other imaging devices such as a camera, a video recorder, an industrial endoscope, a microscope, a robot having an image recognition function, a smartphone, a mobile phone, a smart watch, a tablet terminal, a notebook PC and other mobile devices.

[0203] (Variant 9)

[0204] In the above embodiment, LED is exemplified as the light source provided by the illumination unit 15 and the illumination control unit 22, but the light source is not limited to LED. For example, the light source may be a laser light source including a laser diode, an organic EL, a bulb such as a xenon bulb or a halogen bulb, a combination thereof, or a combination thereof with an optical element such as a phosphor having a wavelength conversion function.

[0205] (Variant 10)

[0206] In the above-mentioned embodiment, the imaging parameter P is the light intensity and gain, but the imaging parameter P is not limited to them. The imaging parameter P may also be the spectral distribution, the aperture value (F value), the shutter speed, the frame rate of the moving image, the optical magnification, the digital magnification, the grayscale in the process of generating the camera image (camera image) in the imaging control unit 21 according to the imaging signal captured by the imaging element 142, the pixel size, the resolution (DPI), etc. In addition, the imaging parameter P may also include the type of organ of the subject (esophagus, stomach, duodenum, etc.). In addition, the type of organ can be determined by a model such as a machine learning model based on the camera image, or it can be determined based on the insertion distance of the front end portion 11 of the endoscope 1 into the body, and it can also be input into the image generation device 3 by the surgical operator.

[0207] It can also be achieved by recording the programs in each embodiment and each of their variants in a computer-readable recording medium, and making the computer system read and execute the program recorded in the recording medium. In addition, the "computer system" is set to include hardware such as OS and peripheral devices. In addition, "computer-readable recording medium" refers to removable media such as floppy disks, optical magnetic disks, ROMs, CD-ROMs, and storage devices such as hard disks built into the computer system. In addition, the "computer-readable recording medium" may also include a medium that dynamically maintains the program in a short period of time, such as a communication line in the case of sending a program via a network such as the Internet, a communication line such as a telephone line, and a medium that maintains the program for a certain period of time, such as a volatile memory inside a computer system that becomes a server or client in this case. In addition, the above program may also be a program for realizing a part of the above functions, and may also be a program that realizes the above functions by combining with a program already recorded in a computer system.

[0208] Description of Reference Numerals

[0209] 100: endoscope system (image generation system); 1: endoscope; 14: imaging unit; 15: lighting unit (white light lighting unit); 16: special light lighting unit; 17: temperature sensor; 2: control device; 3: image generation device; 4: display device; P: imaging parameter; U: upper limit value; D W1 : First camera image (first white light camera image); D W2 : Second camera image (second white light camera image); E W : Estimated image (white light estimated image); D S1 : First camera image (first special light camera image); D S2 : Second camera image (second special light camera image); E S : Estimated image (special light estimation image).

Claims

1. An endoscope system comprising: An endoscope having an illumination unit for irradiating an object with illumination light and an imaging unit for imaging the object; a control device that performs imaging signal processing based on imaging parameters on an imaging signal acquired from the imaging unit to generate a captured image; and an image generating device, in, The control device generates a first camera image and a second camera image, The first captured image is the captured image captured by the subject being illuminated with a light quantity that is equal to or less than an upper limit value, The second captured image is the captured image captured by the subject illuminated with a light quantity equal to or less than the upper limit value, and is brighter than the first captured image. The image generating device generates an estimated image that estimates an image of the subject illuminated with a light intensity greater than the upper limit value, based on the first captured image and the second captured image.

2. The endoscope system according to claim 1, wherein: further comprising a display device for displaying the captured image, The control device continuously generates the second camera image and displays the second camera image on the display device in the form of a moving image, and generates the first camera image as a still image. The image generating device generates the estimated image based on the first captured image and the second captured image generated before or after the first captured image is generated.

3. The endoscope system according to claim 1 or 2, wherein: The image generation device generates the estimated image based on a learned model that performs machine learning in advance on the relationship between the first captured image, the second captured image, and the estimated image.

4. The endoscope system according to claim 3, wherein: The learned model is a model that uses the first camera image prepared for learning, the second camera image prepared for learning, and the third camera image as training data for machine learning. The third camera image is the camera image captured by the subject illuminated with a light amount greater than the upper limit value, and is an image in which noise generated by the camera signal processing is of the same degree as that generated by the camera signal processing in the first camera image.

5. The endoscope system according to claim 3, wherein: The learned model uses at least a portion of the imaging parameters as input data.

6. The endoscope system according to claim 1, wherein: The noise included in the estimated image is of the same level as the noise generated in the first captured image due to the captured image signal processing.

7. The endoscope system according to claim 1, wherein: Regarding the noise generated by the imaging signal processing in the second imaging image, the noise generated by the imaging signal processing in the first imaging image is greater.

8. The endoscope system according to claim 7, wherein: The image signal processing includes amplifying the image signal. A gain of the imaging parameter for generating the second imaging image is greater than a gain of the imaging parameter for generating the first imaging image.

9. The endoscope system according to claim 1, wherein: The control device acquires the temperature of the lighting unit and determines the upper limit value of the light amount so that the temperature becomes equal to or lower than a predetermined temperature.

10. The endoscope system according to claim 1, wherein: The endoscope further includes a special light illumination unit that irradiates special light having a wavelength range different from the wavelength range of the illumination light irradiated by the illumination unit. The control device is capable of performing the imaging signal processing based on the imaging parameter on an imaging signal of the subject irradiated with the special light to generate a special light imaging image. The control device generates a first special light photographed image and a second special light photographed image, The first special light captured image is the special light captured image captured by capturing the subject illuminated by the special light with a light quantity equal to or less than the upper limit value. The second special light captured image is the special light captured image captured by capturing the subject illuminated by the special light having a light quantity equal to or less than the upper limit value, and is brighter than the first special light captured image. The image generating device generates the estimated image based on the first captured image, the second captured image, the first special light captured image, and the second special light captured image.

11. The endoscope system according to claim 10, wherein: The image generating device generates the estimated image based on a learned model obtained by previously performing machine learning on the relationship between the first captured image, the second captured image, the first special light captured image, the second special light captured image, and the estimated image.

12. The endoscope system according to claim 1, wherein: The endoscope further includes a special light illumination unit that irradiates special light having a wavelength range different from the wavelength range of the illumination light irradiated by the illumination unit. The first captured image is an image generated by performing image signal processing on an image signal of the subject irradiated with the special light by the special light illumination unit. The second camera image is an image generated by performing camera signal processing on a camera signal of the subject illuminated by the illumination light of the illumination unit. The estimated image is an image estimated, based on the first captured image and the second captured image, of an image of the subject illuminated by the special light with a light quantity greater than the upper limit value.

13. The endoscope system according to claim 12, wherein: The image generation device generates the estimated image based on a learned model that performs machine learning in advance on the relationship between the first captured image, the second captured image, and the estimated image.

14. The endoscope system according to claim 13, wherein: The learned model is a model that uses the first camera image prepared for learning, the second camera image prepared for learning, and the third camera image as training data for machine learning. The third camera image is the camera image captured by the subject illuminated by the special light with a light amount greater than the upper limit value, and is an image in which noise generated by the camera signal processing is of the same degree as that generated by the camera signal processing in the first camera image.

15. The endoscope system according to claim 1, 10 or 12, wherein: The image generation device continuously generates the estimated images, and generates a moving image using the continuous estimated images as frames and moving images.

16. An image generating device for obtaining a captured image by performing image signal processing based on image parameters on an image signal obtained from an image capturing unit for capturing an image of a subject, wherein: The image generating device acquires a first camera image and a second camera image, wherein the first camera image is a camera image of the subject illuminated with a light quantity below an upper limit value, and the second camera image is a camera image of the subject illuminated with a light quantity below the upper limit value and is brighter than the first camera image. The image generating device generates an estimated image that estimates an image of the subject illuminated with a light intensity greater than the upper limit value, based on the first captured image and the second captured image.

17. The image generating device according to claim 16, wherein: The estimated image is generated based on a learned model that performs machine learning in advance on the relationship between the first captured image, the second captured image, and the estimated image.

18. The image generating device according to claim 17, wherein: The learned model is a model that uses the first camera image prepared for learning, the second camera image prepared for learning, and the third camera image as training data for machine learning. The third camera image is the camera image captured by the subject illuminated with a light amount greater than the upper limit value, and is an image in which noise generated by the camera signal processing is of the same degree as that generated by the camera signal processing in the first camera image.

19. The image generating device according to claim 17, wherein: The learned model uses at least a portion of the imaging parameters as input data.

20. The image generating device according to claim 16, wherein: The noise included in the estimated image is of the same level as the noise generated in the first captured image due to the captured image signal processing.

21. The image generating device according to claim 16, wherein: Regarding the noise generated by the imaging signal processing in the second imaging image, the noise generated by the imaging signal processing in the first imaging image is greater.

22. The image generating device according to claim 21, wherein: The image signal processing includes amplifying the image signal. A gain of the imaging parameter for generating the second imaging image is greater than a gain of the imaging parameter for generating the first imaging image.

23. A method for generating an image, wherein: acquiring a first camera image and a second camera image, wherein the first camera image is an image generated by performing camera signal processing on a camera signal of a subject illuminated with a light amount below an upper limit value, and the second camera image is an image generated by performing camera signal processing on a camera signal of a subject illuminated with a light amount below the upper limit value, and is brighter than the first camera image, An estimated image that estimates an image of the subject illuminated with a light intensity greater than the upper limit is generated based on the first captured image and the second captured image.

24. The image generation method according to claim 23, wherein: The estimated image is generated based on a learned model that performs machine learning in advance on the relationship between the first captured image, the second captured image, and the estimated image.

25. The image generation method according to claim 24, wherein: The learned model is a model that uses the first camera image prepared for learning, the second camera image prepared for learning, and the third camera image as training data for machine learning. The third camera image is an image captured by the subject illuminated with a light amount greater than the upper limit value, and is an image in which noise generated by the camera signal processing is of the same level as that generated by the camera signal processing in the first camera image.

Images