Endoscope system and method of operation thereof

Abstract

The present application provides an endoscope system (10) and a working method thereof, which, in the case where identification processing of a lesion or the like is performed by AI or the like, prompts information of an appropriate treatment instrument in endoscopy. With reference to the identification result (55ep, 55eq) and information (81a, 81b, 82a, 82b, 83a, 83b, 84a, 84b, 85a, 85b) for improving the accuracy of the appropriateness of the treatment instrument, surgical assistance information (60a, 60b, 60c, 60d, 60e, 60f, 60g, 60h, 60i, 60j, 60p, 60q) related to the treatment instrument is calculated, and the examination image (121) and the treatment instrument (125) included in the surgical assistance information are displayed on the display (15).

Description

TECHNICAL FIELD

[0001] The present application relates to an endoscope system that acquires information related to a treatment instrument in an endoscopy, and a working method thereof. BACKGROUND

[0002] In the medical field, a doctor or the like makes a judgment of a treatment method or the like for a lesion portion using an examination image or the like acquired by an endoscope. However, in a case where the experience of a person who makes an image diagnosis such as a doctor is less, or in a case where although the experience is sufficient, an image that becomes a diagnosis object is a rare case and exceeds a professional field, the doctor sometimes cannot accurately make the judgment.

[0003] In recent years, due to the progress of image analysis technology, a medical device that assists in identification and surgery by presenting an identification result of a medical image to a doctor has been invented. For example, in Patent Literature 1, by analysis of a medical image, information related to a treatment instrument used in an endoscopy or the like can be obtained. Also, in Patent Literature 2, a surgery plan can be created from a medical image and a database that stores information of a surgery instrument, a standard treatment step, information of a characteristic portion, patient information, and the like.

[0004] PRIOR ART DOCUMENTS

[0005] PATENT LITERATURE

[0006] Patent Literature 1: International Publication No. 2019 / 008942

[0007] Patent Literature 2: International Publication No. 2019 / 116593 SUMMARY

[0008] PROBLEMS TO BE SOLVED BY THE INVENTION

[0009] As in Patent Literature 1 and Patent Literature 2, in a case where a diagnosis of a lesion portion or the like is made by recognition processing such as image processing or AI (Artificial Intelligence), it is required to improve the reliability of a recognition result obtained by the recognition processing. In recent years, by the recognition processing for mechanical recognition of an observation object by AI or the like, the observation object is automatically diagnosed and a surgery content is predicted. By utilizing such recognition processing, it is possible to compensate for the lack of experience of a doctor, and it is possible to exclude subjective judgment and obtain an objective diagnosis result, thereby providing a higher level and homogeneous medical care.

[0010] However, the diagnostic support device of Patent Literature 1 detects a lesion portion and a treatment instrument included in an endoscope image to perform report creation assistance, and in Patent Literature 2, AI predicts a surgery content from an image of CT (Computed Tomography) or MRI (Magnetic Resonance Imaging), and there is almost no system that predicts a surgery content from an endoscope image in an endoscopy. Therefore, there is a demand for a diagnostic support system that provides information of a surgery content, particularly a treatment instrument, in an endoscopy.

[0011] An object of the present application is to provide an endoscope system and a working method thereof that, in a case where identification processing of a lesion portion or the like is performed by AI or the like, prompts information of an appropriate treatment instrument in an endoscopy.

[0012] Means for solving technical problems

[0013] The endoscope system of the present application is provided with an image control processor. The image control processor performs processing of acquiring an examination image, performing identification processing on the examination image, referring to a result of the identification processing, i.e., an identification result, calculating surgery assistance information related to a treatment instrument, and displaying the examination image and the treatment instrument included in the surgery assistance information on a display.

[0014] It is preferable that the image control processor have a learning model that outputs one or more pieces of surgery assistance information through input of the identification result, the learning model being a learned learning model. It is preferable that the image control processor have a learning model that outputs surgery assistance information with improved appropriateness related to a treatment instrument through input of the identification result and surgery information, the surgery information being one of or obtained by combining one or more pieces of surgery information including inventory information of the treatment instrument, correspondence association information of a lesion and a treatment instrument according to a guideline, i.e., treatment pattern information, peripheral environment information related to an anatomical portion of a throat, an esophagus, a stomach, a small intestine, a large intestine, and a blood vessel course, patient risk determination information of a patient's past history, examination / treatment history, a medicine in use, and a surgeon, the learning model being a learned learning model. It is preferable that the image control processor have a learning model that outputs a treatment expected time through input of surgery assistance information related to a treatment instrument calculated by a surgery assistance information calculation section, the learning model being a learned learning model.

[0015] It is preferable that the learning model, which outputs different surgery assisting information when the input of the inventory information is different, be a learned learning model. It is preferable that the learning model, which outputs different surgery assisting information when the input of the treatment modality information is different, be a learned learning model. It is preferable that the learning model, which outputs different surgery assisting information when the input of the surrounding environment information is different, be a learned learning model.

[0016] It is preferable that the learning model, which outputs different surgery assisting information when the input of the patient risk determination information is different, be a learned learning model. It is preferable that the learning model, which outputs different surgery assisting information when the input of the operator information is different, be a learned learning model.

[0017] It is preferable that the surgery assisting information include the name of the treatment instrument, the appropriateness related to the treatment instrument, and the order of the appropriateness related to the treatment instrument. It is preferable that the display of the surgery assisting information be switchable according to the order of the appropriateness.

[0018] It is preferable that the surgery assisting information be displayed after the surgery assisting information prompting button is pressed. It is preferable that a warning display be performed when the appropriateness of the treatment instrument does not satisfy a certain condition. It is preferable that the certain condition be changeable when a plurality of treatment instruments satisfying the certain condition are selected.

[0019] It is preferable that the image control processor perform the following processing: acquire the examination image, calculate the feature amount of the diagnosed template image stored in the template image storage memory and having a corresponding association established with the treatment instrument, calculate the feature amount of the examination image, calculate the comprehensive matching degree by comparing the feature amount of the examination image with the feature amount of the diagnosed template image, select the template image for which the comprehensive matching degree satisfies a first condition, calculate the appropriateness of the treatment instrument having a corresponding association established with the selected template image with reference to the surgery information including the inventory information of the treatment instrument, output the surgery assisting information including the treatment instrument for which the appropriateness satisfies a second condition, and display the examination image and the treatment instrument included in the surgery assisting information on the display. It is preferable that the examination image be acquired as the template image, and the examination image, the discrimination result related to the examination image, and the treatment instrument be stored in the template image storage memory in a corresponding association.

[0020] It is preferable that the light source processor perform processing of controlling emission of first illumination light and second illumination light having mutually different emission spectra when acquiring an examination image, in a case where a first illumination period during which the first illumination light is emitted and a second illumination period during which the second illumination light is emitted are automatically switched, emitting the first illumination light in a first emission pattern and emitting the second illumination light in a second emission pattern, acquiring a first illumination light image based on the first illumination light, a second illumination light image based on the second illumination light, and an overlay image in which a result of discrimination processing on the second illumination light image is superimposed on a display image based on the first illumination light image, as the examination image. It is preferable that the first emission pattern be either of a first A pattern in which the number of frames of the first illumination period is the same in each first illumination period and a first B pattern in which the number of frames of the first illumination period is different in each first illumination period. It is preferable that the second emission pattern be preferably either of a second A pattern in which the number of frames of the second illumination period is the same in each second illumination period and the emission spectrum of the second illumination light is the same in each second illumination period, a second B pattern in which the number of frames of the second illumination period is the same in each second illumination period and the emission spectrum of the second illumination light is different in each second illumination period, a second C pattern in which the number of frames of the second illumination period is different in each second illumination period and the emission spectrum of the second illumination light is the same in each second illumination period, and a second D pattern in which the number of frames of the second illumination period is different in each second illumination period and the emission spectrum of the second illumination light is different in each second illumination period.

[0021] In a work method of an endoscope system provided with an image control processor, it is preferable that the image control processor perform processing of acquiring an examination image, performing discrimination processing on the examination image, referring to a discrimination result of the discrimination processing, calculating surgery assistance information related to a treatment instrument, and displaying the examination image and the treatment instrument included in the surgery assistance information on a display.

[0022] Effects of Invention

[0023] According to the present application, in a case where diagnosis of a lesion or the like is performed in endoscopic examination, surgery can be assisted by prompting an appropriate treatment instrument based on an examination image. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is an explanatory view that explains the structure of an endoscope system.

[0025] Figure 2 is a block diagram that shows the functions of an endoscope system.

[0026] Figure 3is a graph showing spectra of purple light V, blue light B, green light G, and red light R.

[0027] Figure 4 is a graph showing a spectrum of light emission of the first special light.

[0028] Figure 5 is a graph showing a spectrum of light emission of the second special light.

[0029] Figure 6 is an explanatory diagram showing the first A mode or the second A mode in the overlapping mode.

[0030] Figure 7 is an explanatory diagram showing the first B mode in the overlapping mode.

[0031] Figure 8 is an explanatory diagram showing the second B mode in the overlapping mode.

[0032] Figure 9 is an explanatory diagram showing the second C mode in the overlapping mode.

[0033] Figure 10 is an explanatory diagram showing the second D mode in the overlapping mode.

[0034] Figure 11 is a graph showing spectral transmittances of color filters of an imaging sensor.

[0035] Figure 12 is an explanatory diagram showing a first imaging period and a second imaging period.

[0036] Figure 13 is an explanatory diagram showing a structure of an overlapping image generation section.

[0037] Figure 14 is an explanatory diagram showing, in chronological order, illumination control, discrimination processing, and image display in the overlapping mode.

[0038] Figure 15 is an explanatory diagram showing a control relationship related to a discrimination processing section, a surgery assistance information prompt mode switching section, a first surgery assistance information calculation section, a surgery information storage memory, and a display control section of a processor device of the first embodiment.

[0039] Figure 16 is an explanatory diagram showing information input to a first surgery assistance information calculation model of a processor device of the first embodiment.

[0040] Figure 17 is an explanatory diagram showing that surgery assistance information P is output when a discrimination result P is input to a surgery assistance information calculation model.

[0041] Figure 18is a diagram indicating that the surgery assistance information Q is output when the discrimination result Q is input to the surgery assistance information calculation model.

[0042] Figure 19 is a diagram indicating that the surgery assistance information A is output when the discrimination result P and the inventory information S of the treatment instrument are input to the surgery assistance information calculation model.

[0043] Figure 20 is a diagram indicating that the surgery assistance information B is output when the discrimination result P and the inventory information T of the treatment instrument are input to the surgery assistance information calculation model.

[0044] Figure 21 is a diagram indicating that the surgery assistance information C is output when the discrimination result P and the treatment pattern information S are input to the surgery assistance information calculation model.

[0045] Figure 22 is a diagram indicating that the surgery assistance information D is output when the discrimination result P and the treatment pattern information T are input to the surgery assistance information calculation model.

[0046] Figure 23 is a diagram indicating that the surgery assistance information E is output when the discrimination result P and the surrounding environment information S are input to the surgery assistance information calculation model.

[0047] Figure 24 is a diagram indicating that the surgery assistance information F is output when the discrimination result P and the surrounding environment information T are input to the surgery assistance information calculation model.

[0048] Figure 25 is a diagram indicating that the surgery assistance information G is output when the discrimination result P and the patient risk determination information S are input to the surgery assistance information calculation model.

[0049] Figure 26 is a diagram indicating that the surgery assistance information H is output when the discrimination result P and the patient risk determination information T are input to the surgery assistance information calculation model.

[0050] Figure 27 is a diagram indicating that the surgery assistance information I is output when the discrimination result P and the operator information S are input to the surgery assistance information calculation model.

[0051] Figure 28 is a diagram indicating that the surgery assistance information J is output when the discrimination result P and the operator information T are input to the surgery assistance information calculation model.

[0052] Figure 29 is a diagram indicating a surgery assistance information prompt screen when the appropriateness of the treatment instrument is first.

[0053] Figure 30is a diagram showing a surgery support information presentation screen when the appropriateness of the treatment instrument is the second.

[0054] Figure 31 is a diagram showing a surgery support information presentation screen when a warning display is performed.

[0055] Figure 32 is a diagram showing a surgery support information presentation screen in which a treatment start button is displayed.

[0056] Figure 33 is a diagram showing that a surgery support information P is input to a treatment time calculation model, and a treatment expected time P is output.

[0057] Figure 34 is a diagram showing that a surgery support information Q is input to a treatment time calculation model, and a treatment expected time Q is output.

[0058] Figure 35 is a diagram showing a surgery support information presentation screen in which a treatment expected time is calculated.

[0059] Figure 36 is a diagram showing a surgery support information presentation mode of the first embodiment, and a series of flows of the surgery support information presentation mode.

[0060] Figure 37 is a diagram showing a relationship of information received by a second surgery support information calculation section of a processor device, and information transmitted by the second surgery support information calculation section, with respect to the processor device of the second embodiment.

[0061] Figure 38 is a diagram showing a relationship of an image processing section, a surgery support information presentation mode switching section, a second surgery support information calculation section, a template image storage memory, a surgery information storage memory, and a display control section of the processor device of the second embodiment.

[0062] Figure 39 is a diagram showing a relationship of the processor device, the image processing section, the surgery support information presentation mode switching section, the second surgery support information calculation section, the template image storage memory, the surgery information storage memory, the display control section, a still image acquisition section, and a diagnosis result input section of the second embodiment.

[0063] Figure 40 is a diagram showing a relationship in the second surgery support information calculation section of the processor device of the second embodiment.

[0064] Figure 41 is a diagram showing a surgery support information presentation mode of the second embodiment, and a series of flows of the surgery support information presentation mode. DETAILED DESCRIPTION

[0065] [First Embodiment]

[0066] In Figure 1 The endoscope system 10 has an endoscope 12, a light source device 13, a processor device 14, a display 15, and a UI (User InterFace) 16. The endoscope 12 is optically connected to the light source device 13 and electrically connected to the processor device 14. The endoscope 12 has an insertion section 12a, an operation section 12b, a bending section 12c, and a distal end section 12d. The insertion section 12a is inserted into a body of an observation target. The operation section 12b is provided at a proximal end portion of the insertion section 12a. The bending section 12c and the distal end section 12d are provided at a distal end side of the insertion section 12a. The bending section 12c performs a bending action by operating a bending knob 12e of the operation section 12b. The distal end section 12d is directed toward a desired direction by the bending action of the bending section 12c.

[0067] Further, the bending knob 12e, an observation mode switching switch 12f, a surgery assistance information prompting switch 12g, a still image acquisition instruction switch 12h, and a zoom operation section 12i are provided at the operation section 12b. The observation mode switching switch 12f is used for a switching operation of an observation mode. The surgery assistance information prompting switch 12g is used for prompting of surgery assistance information 123 related to a treatment instrument. The still image acquisition instruction switch 12h is used for an acquisition instruction of a still image of an observation target. The zoom operation section 12i is used for an operation of a zoom lens 42.

[0068] In addition, the endoscope system 10 has three modes of a first illumination observation mode, a second illumination observation mode, and an overlay mode as the observation modes. If the observation mode switching switch 12f is pressed, the mode is switched via an image processing switching section 54. Further, the endoscope system 10 has two modes of a check image display mode and a surgery assistance information prompting mode as display modes. If the surgery assistance information prompting switch 12g is pressed, the display mode is switched via a surgery assistance information prompting mode switching section 56. In addition, the surgery assistance information prompting mode can be used only in a case where the observation mode is the overlay mode. That is, as the modes of the endoscope system 10 according to the present application, there are four modes of a check image display first illumination observation mode, a check image display second illumination observation mode, a check image display overlay mode, and a surgery assistance information prompting overlay mode. Hereinafter, when simply described as a "check image display mode", it means all cases of any three modes of the check image display first illumination observation mode, the check image display second illumination observation mode, and the check image display overlay mode, but it is usually the check image display overlay mode. Further, when simply described as a "surgery assistance information prompting mode", it means the surgery assistance information prompting overlay mode.

[0069] In the examination image display first illumination observation mode, a normal light (first illumination light) such as white light is irradiated to the observation object to perform imaging, and thus a first illumination light image in which colors are natural is displayed on the display 15. In the examination image display second illumination observation mode, a special light (second illumination light) having a wavelength band different from that of the normal light is irradiated to the observation object to perform imaging, and thus a second illumination light image in which a specific structure is emphasized is displayed on the display 15. In the examination image display superimposition mode and the surgery assistance information presentation superimposition mode, the first illumination light and the second illumination light having different emission spectra are switched to perform emission. Also, in addition to performing processing related to AI on an image based on the second illumination light, discrimination processing of obtaining a feature quantity or the like related to the observation object is performed. Then, an overlaid image in which a result of the discrimination processing is superimposed on a display image based on the first illumination light image is displayed on the display 15 (refer to Figure 14 ). In addition, in the present specification, the claims, the drawings, the abstract, the term "examination image" refers to at least any one or all of the first illumination light image, the second illumination light image, and the overlaid image.

[0070] In the examination image display mode, a screen displayed in the three ways of the observation mode is displayed on the display 15. In the surgery assistance information presentation mode, as described later, the examination image 121 and the surgery assistance information 123 related to the treatment instrument are displayed on the surgery assistance information presentation screen 120.

[0071] In the examination image display mode and the surgery assistance information presentation mode, when the user wants to acquire a still image of the examination image, a signal related to a still image acquisition instruction is transmitted to the endoscope 12, the light source device 13, and the processor device 14 by the user operating the still image acquisition instruction switch 12h. When the still image acquisition instruction switch 12h is pressed in the examination image display mode, in the processor device 14, a still image of the observation object is saved in a memory (not shown) during the first illumination period or the second illumination period, or in both. In addition, the saving of the still image in the surgery assistance information presentation mode will be described in detail later.

[0072] The processor device 14 is electrically connected to the display 15 and the UI 16. The display 15 outputs an image of the observation object, information attached to the image of the observation object, and the like. The UI 16 has a keyboard, a mouse, a touchpad, a microphone, and the like, and has a function of accepting an input operation of a function setting or the like. The surgery information storage memory 80 in which surgery information including inventory information of a treatment instrument, treatment mode information, peripheral environment information, patient risk determination information, and surgeon information, an image, information related to the image, and the like are recorded is connected to the processor device 14. In addition, an external memory (not shown) can also be connected to the processor device 14.

[0073] In Figure 2 which, the light source device 13 is provided with a light source part 20 and a light source processor 21 for controlling the light source part. The light source part 20 has, for example, a plurality of semiconductor light sources, and turns them on or off respectively, and controls the light emission amount of each semiconductor light source when it is on, thereby emitting the illumination light that irradiates the observation object. The light source part 20 has LEDs of four colors, V-LED (Violet Light Emitting Diode) 20a, B-LED (Blue Light Emitting Diode) 20b, G-LED (Green Light Emitting Diode) 20c, and R-LED (Red Light Emitting Diode) 20d.

[0074] As Figure 3 shown, the V-LED generates violet light V with a center wavelength of 405 ± 10 nm and a wavelength range of 380 to 420 nm. The B-LED generates blue light B with a center wavelength of 450 ± 10 nm and a wavelength range of 420 to 500 nm. The G-LED generates green light G with a wavelength range of 480 to 600 nm. The R-LED generates red light R with a center wavelength of 620 to 630 nm and a wavelength range of 600 to 650 nm.

[0075] The light source processor 21 controls the V-LED 20a, the B-LED 20b, the G-LED 20c, and the R-LED 20d. The light source processor 21 can emit light by independently controlling each LED 20a to 20d respectively, and independently changing the light amount of the violet light V, the blue light B, the green light G, or the red light R. Also, in the first illumination observation mode, the light source processor 21 controls each LED 20a to 20d to emit white light with a light amount ratio between the violet light V, the blue light B, the green light G, and the red light R becoming Vc:Bc:Gc:Rc. In addition, Vc, Bc, Gc, Rc > 0.

[0076] In addition, as Figure 4 shown, as the second illumination light, a first special light that makes the light amount of the violet light V greater than those of the other blue light B, the green light G, and the red light R can be used. Also, as Figure 5 shown, as the second illumination light, a second special light that makes the light amount of the green light G greater than those of the other violet light V, the blue light B, and the red light R can be used.

[0077] Furthermore, in the second illumination observation mode, the light source processor 21 controls each of the LEDs 20a to 20d to emit a special light with a light intensity ratio of Vs:Bs:Gs:Rs, which is a narrow band of short-wavelength light. This ratio is Vs:Bs:Gs:Rs. The light intensity ratio Vs:Bs:Gs:Rs differs from the light intensity ratio Vc:Bc:Gc:Rc used in the first illumination observation mode and is appropriately set according to the observation objective. For example, when emphasizing superficial blood vessels, it is preferable that Vs is greater than the other Bs, Gs, and Rs; when emphasizing intermediate blood vessels, it is preferable that Gs is greater than the other Vs, Gs, and Rs.

[0078] Furthermore, in the overlapping mode, when automatically switching between the first illumination light and the second illumination light to emit light, the light source processor 21 emits the first illumination light in the first illumination mode and the second illumination light in the second illumination mode. Specifically, the first illumination mode is preferably either mode 1A or mode 1B, such as... Figure 6 As shown, in the first A mode, the number of frames during the first illumination period is the same in each of the first illumination periods, such as... Figure 7 As shown, in the first B mode, the number of frames during the first illumination period varies in each first illumination period. Additionally, in the figure, 'time' indicates the direction of time passage.

[0079] The second luminous mode is preferably any one of mode 2A, mode 2B, mode 2C, and mode 2D, such as... Figure 6 As shown, in the second A mode, the number of frames during the second illumination period is the same in each second illumination period, and the emission spectrum of the second illumination light is the same in each second illumination period, such as... Figure 8 As shown, in the second B mode, the number of frames during the second illumination period is the same in each second illumination period, and the emission spectrum of the second illumination light is different in each second illumination period, such as... Figure 9 As shown, in the second C mode, the number of frames during the second illumination period is different in each second illumination period, and the emission spectrum of the second illumination light is the same in each second illumination period, such as... Figure 10 As shown, in the second D mode, the number of frames during the second illumination period is different in each second illumination period, and the emission spectrum of the second illumination light is different in each second illumination period. Additionally, the emission spectrum of the first illumination light may be the same or different in each first illumination period.

[0080] Here, it is preferable that the first illumination period is longer than the second illumination period, and preferably the first illumination period is two or more frames. For example, in Figure 6In this case, when the first light emission mode is set to the first A mode and the second light emission mode is set to the second A mode (the number of frames of the second illumination period: same, the emission spectrum of the second illumination light: same), the first illumination period is set to 2 frames and the second illumination period is set to 1 frame. The first illumination light is used to generate a display image displayed on the display 15, and thus it is preferable to obtain a bright image by irradiating the observation object with the first illumination light.

[0081] For example, the first illumination light is preferably white light. On the other hand, the second illumination light is used for discrimination processing, and thus it is preferable to obtain an image suitable for discrimination processing by irradiating the observation object with the second illumination light. For example, when discrimination processing is performed based on shape information of a plurality of blood vessels having different depths from the mucosal surface, it is preferable to use violet light V, blue light B, green light G, and red light R as the second illumination light. At this time, when the second light emission mode is set to the second A mode (the number of frames of the second illumination period: same, the emission spectrum of the second illumination light: same) or the second C mode (the number of frames of the second illumination period: different, the emission spectrum of the second illumination light: same), it is preferable to use any one of the violet light V, the blue light B, the green light G, and the red light R. On the other hand, when the second light emission mode is set to the second B mode (the number of frames of the second illumination period: same, the emission spectrum of the second illumination light: different) or the second D mode (the number of frames of the second illumination period: different, the emission spectrum of the second illumination light: different), it is preferable to emit at least two of the violet light V, the blue light B, the green light G, and the red light R in a specific order during the second illumination period. The details of the second light emission mode will be described later. Figure 14 In this case, the three types of light, violet light V, green light G, and red light R, are sequentially emitted.

[0082] Here, when both the first special light and the second special light are used as the second illumination light, the second light emission mode can be set to the second B mode or the second D mode, and the first special light and the second special light can be alternately emitted.

[0083] In addition, the details of the first and second light emission modes, which are the switching modes of the first and second illumination periods, are determined based on the imaging control of the imaging sensor 43 by the imaging processor 44, and thus will be described later. Furthermore, a frame refers to a unit including at least a period from a specific timing to the completion of signal readout in the imaging sensor 43.

[0084] For example, when shape information of blood vessels in a range of 50 μm in depth from the mucosal surface, blood vessels in a range of 200 μm in depth from the mucosal surface, and blood vessels in a range of 600 μm in depth from the mucosal surface are acquired, and discrimination processing is performed based on the shape information of these superficial, middle, and deep blood vessels, it is preferable to use violet light V that emphasizes the superficial blood vessels, green light G that emphasizes the middle blood vessels, and red light R that emphasizes the deep blood vessels.

[0085] In addition, in the present specification, the case where the ratio of the light quantity of at least one semiconductor light source is 0 (zero) is included. Therefore, the case where any one or two or more of the semiconductor light sources are not turned on is included. For example, in the case where the light quantity ratio among the violet light V, the blue light B, the green light G, and the red light R is 1 : 0 : 0 : 0, the case where only one of the semiconductor light sources is turned on and the other three are not turned on is also included in the light quantity ratio.

[0086] As shown in FIG. 1, the light emitted from each of the LEDs 20a to 20d is incident on the light guide 23 via a light path combining portion 22 composed of a mirror and a lens, and the like. The light guide 23 propagates the light from the light path combining portion 22 to the distal end portion 12d of the endoscope 12. Figure 2 An illumination optical system 30a and an imaging optical system 30b are provided at the distal end portion 12d of the endoscope 12. The illumination optical system 30a has an illumination lens 31, and the illumination light propagated by the light guide 23 is irradiated on an observation object via the illumination lens 31. The imaging optical system 30b has an objective lens 41, a zoom lens 42, and an imaging sensor 43. The light from the observation object generated by the irradiation of the illumination light is incident on the imaging sensor 43 via the objective lens 41 and the zoom lens 42. Thereby, an image of the observation object is imaged on the imaging sensor 43. The zoom lens 42 is a lens for magnifying the observation object, and is moved between a telephoto end and a wide angle end by operating a zoom operation portion 12i.

[0087] The imaging sensor 43 is a color sensor of a primary color system, and has three kinds of pixels of a B pixel (blue pixel) having a blue color filter, a G pixel (green pixel) having a green color filter, and an R pixel (red pixel) having a red color filter. As shown in FIG. 2, the blue color filter BF mainly transmits light of a blue wavelength band, and specifically, transmits light of a wavelength band of 380 to 560 nm. The transmittance of the blue color filter BF reaches a peak near a wavelength of 460 to 470 nm. The green color filter GF mainly transmits light of a green wavelength band, and specifically, transmits light of a wavelength band of 460 to 620 nm. The red color filter RF mainly transmits light of a red wavelength band, and specifically, transmits light of a wavelength band of 580 to 760 nm.

[0088] Figure 11 As shown in FIG. 2, the blue color filter BF mainly transmits light of a blue wavelength band, and specifically, transmits light of a wavelength band of 380 to 560 nm. The transmittance of the blue color filter BF reaches a peak near a wavelength of 460 to 470 nm. The green color filter GF mainly transmits light of a green wavelength band, and specifically, transmits light of a wavelength band of 460 to 620 nm. The red color filter RF mainly transmits light of a red wavelength band, and specifically, transmits light of a wavelength band of 580 to 760 nm.

[0089] ​Furthermore, the image sensor 43 is preferably a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor). The image processor 44 controls the image sensor 43. Specifically, the image processor 44 reads out the signal from the image sensor 43 and outputs an image signal from the image sensor 43. In the first illumination observation mode, when white light is exposed to the image sensor 43, the image processor 44 reads out the signal, thereby outputting an image signal Bc from the B pixel, an image signal Gc from the G pixel, and an image signal Rc from the R pixel of the image sensor 43. In the second illumination observation mode, when special light is exposed to the image sensor 43, the image processor 44 reads out the signal, thereby outputting an image signal Bs from the B pixel, an image signal Gs from the G pixel, and an image signal Rs from the R pixel of the image sensor 43.

[0090] In overlapping mode, such as Figure 12 As shown, during the first illumination period, the camera processor 44 reads out signals while exposing the camera sensor 43 to the first illumination light, thereby outputting a first image signal from the camera sensor 43. The period during which the first image signal is output is defined as the first imaging period. The first image signal includes a B1 image signal output from a B pixel, a G1 image signal output from a G pixel, and an R1 image signal output from an R pixel. Furthermore, during the second illumination period, the camera processor 44 reads out signals while exposing the camera sensor 43 to the second illumination light, thereby outputting a second image signal from the camera sensor 43. The period during which the second image signal is output is defined as the second imaging period. The second image signal includes a B2 image signal output from a B pixel, a G2 image signal output from a G pixel, and an R2 image signal output from an R pixel.

[0091] like Figure 2 As shown, the CDS / AGC (Correlated Double Sampling / Automatic Gain Control) circuit 45 performs correlation double sampling (CDS) or automatic gain control (AGC) on the analog image signal obtained from the camera sensor 43. The image signal after passing through the CDS / AGC circuit 45 is converted into a digital image signal by the A / D (Analog / Digital) converter 46. The digital image signal after A / D conversion is input to the processor device 14.

[0092] As for the acquisition and display of the inspection image via the image processing section 55 and the display control section 61, the following-described configuration and operation method of the processor device 14 are common in the inspection image display mode and the surgery assistance information presentation mode.

[0093] In the processor device 14, the program in the program memory is operated by the central control section 70 constituted by the image control processor, whereby the functions of the image acquisition section 50, the DSP (Digital Signal Processor) 52, the noise reduction section 53, the image processing switching section 54, the image processing section 55, and the display control section 61 are realized. In the superimposition mode, the image control processor performs image processing based on the first image signal or the second image signal, and performs control over the display 15.

[0094] The image acquisition section 50 acquires a color image input from the endoscope 12. The color image includes a blue signal (B image signal), a green signal (G image signal), and a red signal (R image signal) output from the B pixel, the G pixel, and the R pixel of the imaging sensor 43. The acquired color image is sent to the DSP 52. The DSP 52 performs various signal processing such as defect correction processing, offset processing, gain correction processing, matrix processing, gamma conversion processing, demosaicing processing, and YC conversion processing on the received color image.

[0095] The matrix processing for improving color reproducibility is performed on the image signal of each color after the gain correction processing. Thereafter, the luminance and the chroma of the color image are adjusted by the gamma conversion processing. The demosaicing processing (also called isotropic processing or synchronization processing) is performed on the color image after the matrix processing, and a signal of a missing color for each pixel is generated by interpolation. By the demosaicing processing, all the pixels have signals of RGB colors. The DSP 52 performs the YC conversion processing on the color image after the demosaicing processing, and outputs a luminance signal Y, a color difference signal Cb, and a color difference signal Cr to the noise reduction section 53.

[0096] In addition, when the first special light is used as the second illumination light, the color difference expansion processing for expanding the color difference between the normal portion and the abnormal portion (lesion portion or the like) included in the observation object can be performed on the second image signal. The second image signal after the color difference expansion processing can be subjected to the discrimination processing.

[0097] The noise reduction section 53 performs noise reduction processing based on, for example, a moving average method or a median filter method on the color image on which the demosaicing processing or the like has been performed by the DSP 52. The color image after the noise reduction is input to the image processing switching section 54.

[0098] The image processing switch section 54 switches the transmission destination of the image signal from the noise reduction section 53 to any one of the first illumination light image generation section 55a, the second illumination light image generation section 55b, and the superimposition image generation section 55c in the image processing section 55 according to the set mode. Specifically, when the check image display first illumination observation mode is set, the image signal from the noise reduction section 53 is input to the first illumination light image generation section 55a. When the check image display second illumination observation mode is set, the image signal from the noise reduction section 53 is input to the second illumination light image generation section 55b. When the superimposition mode is set, the image signal from the noise reduction section 53 is input to the superimposition image generation section 55c.

[0099] After the next paragraph, the first illumination light image image processing, the second illumination light image image processing, and the superimposition image display control processing performed on the image signal transmitted from the noise reduction section 53 to the image processing section 55 via the image processing switch section 54 are described.

[0100] In the case of the check image display first illumination observation mode, the first illumination light image generation section 55a performs first illumination light image image processing on the input one-frame amount of Rc image signal, Gc image signal, and Bc image signal. The first illumination light image image processing includes a 3 x 3 matrix processing, a color conversion processing such as a gray scale conversion processing, a three-dimensional LUT (Look Up Table) processing, a color emphasis processing, a structure emphasis processing such as a spatial frequency emphasis, and the like. The Rc image signal, Gc image signal, and Bc image signal on which the first illumination light image image processing has been performed are input to the display control section 61 as a first illumination light image via the surgery assistance information prompting mode switch section 56. In the check image display first illumination observation mode, the first illumination light image is not input to the first surgery assistance information calculation section 60 described later, but is directly input to the display control section 61 (refer to FIG. 2). Figure 15 ) The display control section 61 displays the first illumination light image on the display 15.

[0101] When the second illumination observation mode is used to check the image display, the second illumination image generation unit 55b performs image processing for the second illumination image on the input 1-frame amount of Rs image signal, Gs image signal, and Bs image signal. The second illumination image processing includes 3×3 matrix processing, grayscale conversion processing, color conversion processing such as three-dimensional LUT (Look Up Table) processing, color emphasis processing, and structural emphasis processing such as spatial frequency emphasis. The Rs image signal, Gs image signal, and Bs image signal that have undergone second illumination image processing are used as the second illumination image and input to the display control unit 61 via the surgical assistance information prompt mode switching unit 56. In the second illumination observation mode of the check image display, the second illumination image is directly input to the display control unit 61 without passing through the first surgical assistance information calculation unit 60 (described later). Figure 15 The display control unit 61 displays the first illumination image on the display 15.

[0102] When checking the image display overlay mode, Figure 13 Overlapping images are generated in the overlapping image generation unit 55c shown. The overlapping image generation unit 55c includes a display image generation unit 55d, a discrimination processing unit 55e, and an overlap processing unit 55f. The display image generation unit 55d performs the same first illumination image processing as described above on the input one-frame quantity of R1 image signals, G1 image signals, and B1 image signals. The R1 image signals, G1 image signals, and B1 image signals after the first illumination image processing are used as display images. Furthermore, the discrimination processing unit 55e performs discrimination processing on the input specific frame quantity of R2 image signals, G2 image signals, and B2 image signals. The overlap processing unit 55f generates overlapping images by performing overlapping image processing such as overlaying the discrimination processing result, i.e., the discrimination result, onto the display image.

[0103] The display image or overlay image is input to the display control unit 61 via the surgical assistance information prompt mode switching unit 56. The display control unit 61 performs overlay image display control processing to display the display image or overlay image on the display 15. In addition, the same overlay image display control processing is performed in the surgical assistance information prompt overlay mode described later.

[0104] The following explains the control process for displaying overlapping images. For example... Figure 14As shown, in a case where the first light emission mode is set to the first A mode and the second light emission mode is set to the second B mode (frame number of the second illumination period: same, emission spectrum of the second illumination light: different), 2-frame amount of white light W as the first illumination light is irradiated to the observation target, and 1-frame amount of purple light V, green light G, and red light R as the second illumination light is respectively irradiated to the observation target during the emission period of the white light W, the first image signal obtained by the illumination with the white light is subjected to the first illumination light image image processing, and thus a display image is obtained.

[0105] On the other hand, the second image signals (R2 image signal, G2 image signal, B2 image signal) obtained by the illumination with the purple light V are subjected to the discrimination processing to obtain discrimination results V. Similarly, the second image signals (R2 image signal, G2 image signal, B2 image signal) obtained by the illumination with the green light G are subjected to the discrimination processing to obtain discrimination results G. Also, the second image signals (R2 image signal, G2 image signal, B2 image signal) obtained by the illumination with the red light R are subjected to the discrimination processing to obtain discrimination results R. After the discrimination processing related to the red light R is completed, these discrimination results V, G, R are displayed as the aggregated discrimination results T in the display image. In addition, the discrimination results V, G, R can be individually displayed in the display image, and the discrimination result obtained by combining at least two of the discrimination results V, G, R can also be displayed as an overlay image.

[0106] The above display image or the overlay image is input to the display control section 61 via the surgery assistance information prompting mode switching section 56. The display control section 61 displays the display image or the overlay image in the display 15 according to a specific display condition. As the specific display condition, for example, there is a case where the discrimination results are detected for a lesion portion having a high progression degree. At this time, when the specific display condition is not satisfied, the display control section 61 displays the display image in the display 15, and when the specific display condition is satisfied, the display control section 61 displays the overlay image in the display 15.

[0107] In addition, in the first embodiment, the first image signal based on the first illumination light is used for the display image, and the second image signal based on the second illumination light is used only for the discrimination processing and is not used for the display to the display 15, but the second image signal can also be used for the display to the display 15. At this time, the display image based on the first image signal and the display image based on the second image signal are switched and displayed in the display 15. As for the display or non-display of the image based on the second image signal to the display 15, it is preferable that appropriate setting be made through the UI 16.

[0108] Next, the calculation of the surgery assistance information related to the treatment instrument based on the discrimination results will be described in detail. As shown in FIG. 6, the discrimination results V, G, R are obtained for each of the three colors of the second illumination light, and the discrimination results V, G, R are aggregated to obtain the aggregated discrimination results T. The aggregated discrimination results T are used for the calculation of the surgery assistance information related to the treatment instrument. Figure 15As shown, the discrimination processing section 55e outputs a discrimination result by performing, in association with the calculation of the feature quantity related to the second illumination light image, the setting of the region of interest related to the second illumination light image, and the determination of the diagnostic result, and so on, and transmits it to the display control section 61. Here, the discrimination result that is specifically output is the diagnostic result and the first illumination light image, the second illumination light image, or the superimposed image in which the region of interest is emphasized. (In the present application, the "diagnostic result" is one of the "discrimination result" and is a concept included in the "discrimination result".) The diagnostic result is output from the input of the second illumination light image to the discrimination model 55eM according to the feature quantity related to the second illumination light image. The emphasis display of the region of interest will be described later.

[0109] The discrimination result output from the discrimination processing section 55e is output as a result of the input of the image signal transmitted from the noise reduction section 53 to the discrimination processing section 55e via the image processing switching section 54 to the discrimination model 55eM mounted on the discrimination processing section 55e as a learning model in machine learning. The learning includes supervised learning, unsupervised learning, reinforcement learning, deep reinforcement learning, learning using a neural network, deep learning, and so on.

[0110] It is preferable to classify the feature quantity according to at least any one of the observation object being located in the superficial layer, the middle layer, or the deep layer. The feature quantity is preferably the shape, the color, or a value obtained from these shapes and colors of the observation object. As items of the feature quantity, for example, there are the blood vessel density, the blood vessel shape, the number of branches of the blood vessel, the thickness of the blood vessel, the length of the blood vessel, the tortuosity of the blood vessel, the depth of the blood vessel, the gland duct shape, the gland duct opening shape, the length of the gland duct, the tortuosity of the gland duct, and color information. The feature quantity is preferably at least one of these or a value obtained by combining two or more of these. In addition, the items of the feature quantity are not limited to these and can be appropriately added according to the use case.

[0111] Also, the diagnostic result preferably includes any one or more of whether it is a lesion or a normal portion, an active phase or a remission phase, an international disease classification, a UICC TNM (Union for International Cancer Control Tumor Lymph Nodes Metastasis) classification, a TNM (Tumor Lymph Nodes Metastasis) classification, a Dukes classification, other classifications, a diagnostic criterion, a guideline, a diagnostic name based on a textbook and an atlas, a kind, a type, a degree of progression, and a stage.

[0112] In addition, as to the display or non-display of the discrimination result displayed together with the image to the display 15, it is preferable that it can be appropriately set by the UI 16.

[0113] Following this paragraph, the surgical assistance information prompting mode is disclosed. When the surgical assistance information prompting switch 12g is pressed, the surgical assistance information prompting mode switching unit 56 switches from the examination image display overlay mode to the surgical assistance information prompting overlay mode to switch to the surgical assistance information prompting mode.

[0114] right Figure 15 The switching of the surgical assistance information prompt mode and an overview of the surgical assistance information prompt mode will be explained. Although the default display mode is the examination image display mode, if the user presses the surgical assistance information prompt switch 12g, the surgical assistance information prompt mode switching unit 56 will be operated to switch to the surgical assistance information prompt mode, and the calculated identification result will be sent to the first surgical assistance information calculation unit 60.

[0115] Next, the first surgical assistance information calculation unit 60 outputs surgical assistance information 123 related to the treatment device based on the identification result. The surgical assistance information 123 calculated here can be limited to surgical assistance information that meets specific conditions. Specifically, surgical assistance information 123 related to treatment devices for which an appropriateness value is calculated to be higher than a predetermined value is selected. Finally, the inspection screen and the treatment device included in the calculated surgical assistance information 123 are displayed on one screen as a surgical assistance information prompt screen 120, set to a state that the user can visually recognize (see reference). Figure 29 With the above structure, during endoscopic examinations, the user can switch to the surgical assistance information prompt mode at any time and display the appropriate treatment instruments corresponding to the identification results of the examination images as surgical assistance information 123.

[0116] In the first embodiment, the surgical assistance information 123 (reference) output by the first surgical assistance information calculation unit 60 Figure 29 As Figure 15 and Figure 16 The results shown are obtained from the discrimination processing unit 55e within the overlapping image generation unit 55c and the second illumination image, which are input to the surgical assistance information calculation model 60M, which serves as a learning model in machine learning, mounted on the first surgical assistance information calculation unit 60, and then output. Information calculated by the first surgical assistance information calculation unit 60 in the second embodiment will be described later.

[0117] In this embodiment, the identification result output by the identification model 55eM in the identification processing unit 55e is sent to the first surgical assistance information calculation unit 60. If an identification result is input, the surgical assistance information calculation model 60M of the first surgical assistance information calculation unit 60 outputs surgical assistance information 123 related to the treatment instrument. Preferably, the surgical assistance information calculation model 60M is a learned model that outputs one or more pieces of surgical assistance information 123 related to the treatment instrument based on the input of the identification result. Furthermore, it is preferred that the surgical assistance information calculation model 60M is a model that has been learned to output different pieces of surgical assistance information 123 depending on the input of the identification result. For example, the surgical assistance information calculation model 60M is preferably as follows: Figure 17 and Figure 18 As shown, when the identification result output by the identification processing unit 55e is identification result P55ep, surgical assistance information P60p is output; when the identification result output by the identification processing unit 55e is identification result Q55eq, surgical assistance information Q60q is output.

[0118] As a specific example, such as Figure 17 As shown, when the identification result P55ep input to the surgical assistance information calculation model 60M is "long diameter 30mm, not reaching the submucosal layer, pedunculated, polyp", the output surgical assistance information P60p is "treatment method: polyp removal, most appropriate treatment instrument: polyp removal device". On the other hand, as Figure 18 As shown, when the identification result Q55eq input to the surgical assistance information calculation model 60M is "long diameter 50mm, not reaching the submucosal layer, sessile, stage I cancer", the output surgical assistance information Q60q is "treatment method: ESD, most appropriate treatment instrument: disposable high-frequency electrosurgical unit". Through this structure, one or more appropriate treatment instruments can be suggested to the user based on the identification result, reducing surgical risks, especially in cases of insufficient user experience or rare cases, and shortening treatment time.

[0119] The surgical assistance information 123 output by the surgical assistance information calculation model 60M contains the treatment method for the lesion output by the first surgical assistance information calculation unit 60. Specific examples include endoscopic polypectomy (polypectomy, polypectomy with a vibrator), endoscopic mucosal resection (EMR), endoscopic submucosal dissection (ESD), endoscopic ultrasound-guided fine needle aspiration, endoscopic mucosal cauterization, etc., but are not limited to these.

[0120] As the information input to the surgical assistance information calculation model 60M of the first surgical assistance information calculation unit 60, it is preferable that, in addition to the identification result output by the identification model 55eM, surgical information stored in the surgical information storage memory 80 is also input. At this time, the surgical assistance information calculation model 60M, mounted on the first surgical assistance information calculation unit 60, improves the appropriateness related to the treatment instrument by inputting the identification result and the surgical information stored in the surgical information storage memory 80, and outputs surgical assistance information 123. The surgical assistance information calculation model 60M preferably performs pre-learning based on the surgical information stored in the surgical information storage memory 80 (described later). With the above structure, the identification result and other information can be combined to suggest a treatment instrument with higher appropriateness to the user, further reducing surgical risks and shortening treatment time.

[0121] The surgical information storage memory 80 stores surgical information including inventory information 81 of treatment instruments, treatment mode information 82 (correspondence between lesions and treatment instruments according to guidelines), surrounding environment information 83 (related to the anatomical locations of the pharynx, esophagus, stomach, small intestine, large intestine, etc., and the course of blood vessels), patient risk assessment information 84 (including the patient's past medical history, examination / treatment history, and medications in use), and surgeon information 85. It also stores template images used in the second embodiment, identification results associated with the template images, and information about the treatment instruments. Regarding the surgical information input to the surgical assistance information calculation model 60M, it is preferable to input one or more pieces of surgical information in combination to improve the appropriateness related to the treatment instruments when outputting surgical assistance information.

[0122] The first surgical assistance information calculation unit 60 preferably has a surgical assistance information calculation model 60M that has been learned to output different surgical assistance information 123 when the input of the identification result is common and the input of the inventory information 81 of the treatment instrument is different.

[0123] For example, such as Figure 19 and Figure 20 As shown, the identification result output from the identification processing unit 55e and input to the first surgical assistance information calculation unit 60 is a common identification result P55ep, while the inventory information of the treatment instruments input from the surgical information storage memory 80 is different inventory information S81a of the treatment instruments. Figure 19 ) or inventory information of disposal equipment T81b ( Figure 20 In the case of inputting inventory information S81a for surgical instruments, the surgical assistance information calculation model 60M outputs surgical assistance information A60a. Figure 19 When the inventory information T81b of the treatment instruments is input, the surgical assistance information calculation model 60M outputs surgical assistance information B60b. Figure 20 ).

[0124] As a specific example, such as Figure 19 As shown, when the most appropriate treatment device derived from the input of the identification result P55ep is a disposable high-frequency scissors, and the inventory information S81a of the input treatment device shows that the inventory of disposable high-frequency scissors is 10, the surgical assistance information A60a indicates that the most appropriate treatment device is the disposable high-frequency scissors. On the other hand, as... Figure 20 As shown in the input surgical instrument inventory information T81b, when the inventory of disposable high-frequency forceps is 0, the surgical assistance information B60b suggests that other surgical instruments (e.g., disposable high-frequency hemostatic forceps) are the most appropriate surgical instruments. Furthermore, the threshold for setting different inventory levels for surgical assistance information 123 can be set automatically or manually by the user.

[0125] Furthermore, the inventory information 81 of the treatment instruments can be accessed through an external management system (not shown). The inventory quantity of the treatment instruments can also be displayed on the surgical assistance information prompt screen 120 (described later). Additionally, if the surgical assistance information 123 that should have been output is changed to surgical assistance information 123 related to other treatment instruments due to inventory falling below a threshold, a warning can be displayed on the surgical assistance information prompt screen 120 (described later) indicating that the inventory of the treatment instrument that should have been output is below the threshold. This structure prevents unexpected surgical delays caused by prompting for the treatment instrument with the highest suitability but no inventory. Furthermore, it encourages users to consume treatment instruments manufactured by the company.

[0126] The first surgical assistance information calculation unit 60 preferably has a surgical assistance information calculation model 60M that has been learned to output different surgical assistance information 123 when the input of the identification result is common and the input of the treatment mode information 82 is different.

[0127] For example, such as Figure 21 and Figure 22 As shown, the identification result output from the identification processing unit 55e and input to the first surgical assistance information calculation unit 60 is a common identification result P55ep, while the treatment mode information input from the surgical information storage memory 80 is different treatment mode information S82a. Figure 21 ) or disposal mode information T82b ( Figure 22 In the case of inputting treatment mode information S82a, the surgical assistance information calculation model 60M outputs surgical assistance information C60c. Figure 21 When the treatment mode information T82b is input, the surgical assistance information calculation model 60M outputs surgical assistance information D60d. Figure 22 ).

[0128] As a specific example, such asFigure 21 As shown, in a case where the most appropriate treatment tool derived from the input of the discrimination result P55ep is a polyp resecting snare, as the input treatment mode information S82a, the treatment tool recommended for use in the 2019 version of the gastric cancer treatment guidelines is a polyp resecting snare, as the surgery assistance information C60c, it is prompted that the most appropriate treatment tool is a polyp resecting snare. On the other hand, as shown, as the input treatment mode information T82b, the treatment tool recommended for use in the 2020 version of the gastric cancer treatment guidelines is a disposable high-frequency cutting forceps, the new guidelines are recognized as information for further improving the appropriateness, as the surgery assistance information D60d, it is prompted that the most appropriate treatment tool is a disposable high-frequency cutting forceps. In addition, with respect to the treatment mode information 82, the surgery information storage 80 can be automatically or manually updated by the user. Figure 22 As shown, in a case where the most appropriate treatment tool derived from the input of the discrimination result P55ep is a polyp resecting snare, as the input treatment mode information S82a, the treatment tool recommended for use in the 2019 version of the gastric cancer treatment guidelines is a polyp resecting snare, as the surgery assistance information C60c, it is prompted that the most appropriate treatment tool is a polyp resecting snare. On the other hand, as shown, as the input treatment mode information T82b, the treatment tool recommended for use in the 2020 version of the gastric cancer treatment guidelines is a disposable high-frequency cutting forceps, the new guidelines are recognized as information for further improving the appropriateness, as the surgery assistance information D60d, it is prompted that the most appropriate treatment tool is a disposable high-frequency cutting forceps. In addition, with respect to the treatment mode information 82, the surgery information storage 80 can be automatically or manually updated by the user.

[0129] Also, the referred treatment mode information can be displayed on the surgery assistance information prompting screen 120 described later. With the above structure, it is possible to prompt the treatment tool that refers to the latest guidelines at the time of treatment.

[0130] The 1st surgery assistance information calculation section 60 preferably has a surgery assistance information calculation model 60M that has been learned in such a manner that different surgery assistance information 123 is outputted when the input of the discrimination result is common and the input of the surrounding environment information 83 is different.

[0131] For example, as shown in Figure 23 and Figure 24 As shown, in a case where the discrimination result outputted by the discrimination processing section 55e and inputted to the 1st surgery assistance information calculation section 60 is the common discrimination result P55ep, and the surrounding environment information inputted from the surgery information storage 80 is the mutually different surrounding environment information S83a( Figure 23 ) or the surrounding environment information T83b( Figure 24 ), in a case where the surrounding environment information S83a is inputted, the surgery assistance information calculation model 60M outputs the surgery assistance information E60e( Figure 23 ), and in a case where the surrounding environment information T83b is inputted, the surgery assistance information calculation model 60M outputs the surgery assistance information F60f( Figure 24 ).

[0132] As a specific example, as shown in Figure 23 in a case where the most appropriate treatment tool derived from the input of the discrimination result P55ep is a monopolar hemostatic clamp, as the input surrounding environment information S83a, the current examination site is the gastric fundus, as the surgery assistance information E60e, it is prompted that the most appropriate treatment tool is a monopolar hemostatic clamp. On the other hand, as shown in Figure 24As shown, when the current examination site is a site where blood vessels are numerous in the rectum as the input peripheral environment information T83b, the most appropriate treatment tool is a bipolar hemostatic forceps as the surgery assistance information F60f is prompted.

[0133] Further, regarding the peripheral environment information 83, the surgery information storage memory 80 can be updated automatically or manually by the user. Also, an examination site input section (not shown) can be provided to input the examination site by the user in advance. With the above structure, the best treatment tool can be prompted according to the anatomical site to be operated on and the progress of blood vessels, and the safety of treatment can be improved.

[0134] The first surgery assistance information calculation section 60 preferably has a surgery assistance information calculation model 60M that is learned in a manner that outputs different surgery assistance information 123 when the input of the discrimination result is common and the input of the patient risk determination information 84 is different.

[0135] For example, as shown in FIG. 6A, when the discrimination result output by the discrimination processing section 55e and input to the first surgery assistance information calculation section 60 is a common discrimination result P55ep, and the patient risk determination information input from the surgery information storage memory 80 is mutually different patient risk determination information S84a (P55ep, S84a) or patient risk determination information T84b (P55ep, T84b), the surgery assistance information calculation model 60M outputs surgery assistance information G60g (G60g, S84a) or surgery assistance information H60h (H60h, T84b) as the most appropriate treatment tool. Figure 25 Figure 26 As shown in FIG. 6A, when the discrimination result output by the discrimination processing section 55e and input to the first surgery assistance information calculation section 60 is a common discrimination result P55ep, and the patient risk determination information input from the surgery information storage memory 80 is mutually different patient risk determination information S84a (P55ep, S84a) or patient risk determination information T84b (P55ep, T84b), the surgery assistance information calculation model 60M outputs surgery assistance information G60g (G60g, S84a) or surgery assistance information H60h (H60h, T84b) as the most appropriate treatment tool. Figure 25 Figure 26 As shown in FIG. 6A, when the discrimination result output by the discrimination processing section 55e and input to the first surgery assistance information calculation section 60 is a common discrimination result P55ep, and the patient risk determination information input from the surgery information storage memory 80 is mutually different patient risk determination information S84a (P55ep, S84a) or patient risk determination information T84b (P55ep, T84b), the surgery assistance information calculation model 60M outputs surgery assistance information G60g (G60g, S84a) or surgery assistance information H60h (H60h, T84b) as the most appropriate treatment tool. Figure 25 Figure 26 As shown in FIG. 6A, when the discrimination result output by the discrimination processing section 55e and input to the first surgery assistance information calculation section 60 is a common discrimination result P55ep, and the patient risk determination information input from the surgery information storage memory 80 is mutually different patient risk determination information S84a (P55ep, S84a) or patient risk determination information T84b (P55ep, T84b), the surgery assistance information calculation model 60M outputs surgery assistance information G60g (G60g, S84a) or surgery assistance information H60h (H60h, T84b) as the most appropriate treatment tool.

[0136] As shown in FIG. 6A, when the discrimination result output by the discrimination processing section 55e and input to the first surgery assistance information calculation section 60 is a common discrimination result P55ep, and the patient risk determination information input from the surgery information storage memory 80 is mutually different patient risk determination information S84a (P55ep, S84a) or patient risk determination information T84b (P55ep, T84b), the surgery assistance information calculation model 60M outputs surgery assistance information G60g (G60g, S84a) or surgery assistance information H60h (H60h, T84b) as the most appropriate treatment tool. Figure 25 Figure 26 As shown in FIG. 6A, when the discrimination result output by the discrimination processing section 55e and input to the first surgery assistance information calculation section 60 is a common discrimination result P55ep, and the patient risk determination information input from the surgery information storage memory 80 is mutually different patient risk determination information S84a (P55ep, S84a) or patient risk determination information T84b (P55ep, T84b), the surgery assistance information calculation model 60M outputs surgery assistance information G60g (G60g, S84a) or surgery assistance information H60h (H60h, T84b) as the most appropriate treatment tool.

[0137] ​​​​Furthermore, the patient risk assessment information 84 can utilize information from an external electronic medical record or management system (not shown). Additionally, patient information can be displayed on the surgical assistance information prompt screen 120, described later. Through this structure, appropriate treatment instruments can be prompted based on the patient information. Furthermore, it can prevent unexpected medical accidents caused by patient risk.

[0138] The first surgical assistance information calculation unit 60 preferably has a surgical assistance information calculation model 60M that has been learned to output different surgical assistance information 123 when the input of the identification result is the same and the input of the surgeon information 85 is different.

[0139] For example, such as Figure 27 and Figure 28 As shown, the identification result output from the identification processing unit 55e and input to the first surgical assistance information calculation unit 60 is a common identification result P55ep, while the surgeon information input from the surgical information storage memory 80 is different surgeon information S85a. Figure 27 ) or surgeon information T85b ( Figure 28 In the case of inputting surgeon information S85a, the surgical assistance information calculation model 60M outputs surgical assistance information I60i. Figure 27 When the surgeon's information T85b is input, the surgical assistance information calculation model 60M outputs surgical assistance information J60j. Figure 28 ).

[0140] As a specific example, such as Figure 27 As shown, if the hemostat manufactured by Company A is the most suitable treatment device derived from the input of the identification result P55ep, and the hemostat manufactured by Company A has been used 50 times and the hemostat manufactured by Company B has been used 5 times as the input surgeon information S85a, then as surgical aid information G60g, it is indicated that the hemostat manufactured by Company A is the most suitable treatment device. On the other hand, as... Figure 28 As shown, when the hemostat manufactured by Company A has been used 5 times and the hemostat manufactured by Company B has been used 50 times, as the surgical aid information H60h, it is indicated that the hemostat manufactured by Company B is the most appropriate treatment instrument.

[0141] Furthermore, the surgeon information 85 can be set automatically or manually by the user. The surgeon information 85 can utilize information from an external management system (not shown) using IDs, or a surgeon information input section (not shown) can be set up for the user to pre-enter surgeon information. Through this structure, appropriate treatment instruments can be suggested based on the surgeon's expertise and experience.

[0142] The surgery information input from the surgery information storage 80 to the first surgery assistance information calculating section 60 is one of inventory information of a treatment instrument, treatment mode information, surrounding environment information, a patient's anamnesis, patient risk determination information, surgeon information, or surgery information obtained by combining one or more of them, and preferably a combination of a plurality of the surgery information. This is to improve the appropriateness of the treatment instrument related to the surgery assistance information 123 output from the first surgery assistance information calculating section 60. Note that the surgery information is not limited to the above.

[0143] The surgery assistance information 123 output from the first surgery assistance information calculating section 60 preferably includes the name of the candidate of the treatment instrument and the appropriateness of the treatment instrument. The name of the treatment instrument refers to a general name such as "disposable high-frequency scissors forceps" and a trade name or a brand name (see Figure 29 ). Although only the general name is illustrated, it is assumed that the brand name and the general name are displayed by default. Also, the appropriateness of the treatment instrument is a score indicating the degree of appropriateness of the treatment instrument. The score is given by the surgery assistance information calculating model 60M to a plurality of treatment instruments, and the final score is reflected in the surgery assistance information 123. Also, the surgery assistance information 123 further preferably includes the name of the treatment instrument, the diagnosis result of the examination image, the treatment method, the appropriateness of the treatment instrument, the rank of the appropriateness of the treatment instrument, and the image 125 of the treatment instrument. The surgery assistance information 123 output from the first surgery assistance information calculating section 60 is sent to the display control section 61.

[0144] The rank of the appropriateness of the treatment instrument is a rank given in accordance with the appropriateness of the treatment instrument output from the first surgery assistance information calculating section 60. For example, when four treatment instruments whose appropriateness is 95, 90, 80, and 40, respectively, are output, the treatment instrument whose appropriateness is 95 is ranked first, the treatment instrument whose appropriateness is 90 is ranked second, the treatment instrument whose appropriateness is 80 is ranked third, and the treatment instrument whose appropriateness is 40 is ranked fourth.

[0145] The first surgery assistance information calculating section 60 can limit the treatment instrument output as the surgery assistance information 123 to a treatment instrument that satisfies a specific condition. The specific condition is a threshold value for the appropriateness of the treatment instrument. For example, assuming that a treatment instrument whose appropriateness is 95 (first), a treatment instrument whose appropriateness is 90 (second), a treatment instrument whose appropriateness is 80 (third), and a treatment instrument whose appropriateness is 40 (fourth) are to be output, the specific condition is set to "80 or more". Under this condition, the treatment instrument whose appropriateness is 40 does not satisfy the specific condition, and thus is not sent to the display control section 61, on the other hand, the treatment instruments whose appropriateness is 95, 90, and 80 satisfy the specific condition, and thus are sent to the display control section 61 as a plurality of the surgery assistance information 123.

[0146] The specific condition can be changed, and can be set automatically by the surgery assistance information calculation model 60M or manually by the user. With the above structure, the treatment instrument to be prompted can be selected according to the value of the appropriateness of the treatment instrument. For example, when a plurality of treatment instruments are selected by the first surgery assistance information calculation section 60, and the number of selected treatment instruments is larger than expected, by increasing the threshold value of the appropriateness of the treatment instrument, only the treatment instrument with higher appropriateness can be prompted.

[0147] In the surgery assistance information prompting mode, the display control section 61 sets the examination image 121 and the surgery assistance information 123 containing the treatment instrument satisfying the specific condition as one screen, and displays it as the surgery assistance information prompting screen 120 on the display 15. Also, as shown in Figure 29 As the surgery assistance information 123, it is preferable to set the name of the treatment instrument, the diagnosis result of the examination image, the treatment method, the appropriateness of the treatment instrument, the rank of the appropriateness of the treatment instrument, and the image 125 of the treatment instrument and the examination image 121 as one screen, and display it as the surgery assistance information prompting screen 120. With the above structure, the user can be prompted with the appropriate treatment instrument for the lesion displayed in the examination image 121. Also, it becomes easy to transfer information between the user and a third person who is observing the display 15 at the same time.

[0148] On the surgery assistance information prompting screen 120, as shown in Figure 29 , the attention region in the examination image 121 is highlighted by labeling it with, for example, a circular frame surrounding the attention region and the symbol I1. In the first embodiment, as described above, this highlighting is performed by the discrimination model 55eM of the discrimination processing section 55e.

[0149] As the display form of this highlighting, it is not limited to Figure 29 the example shown above, and the shape, color, size, thickness of the frame, or the display using a pseudo color or a pattern can be appropriately changed. Also, when there are a plurality of attention regions in the examination image 121, different shapes, colors, sizes, thicknesses of the frame, or the like can be used for each attention region, or different colors or patterns can be used for the display. A number or a character, or the like, can be marked on each highlighted display. Also, when the user is made aware that the lesion needs extra attention, display using a warning color such as yellow or red can be performed. Also, it is preferable that it can also be set not to perform the highlighting.

[0150] In addition, when there are a plurality of regions of interest in the inspection image 121, it is preferable to select each region of interest, and the surgery assistance information 123 related to each region of interest is displayed on the surgery assistance information presentation screen 120. At this time, it is preferable to make the emphasized display for the region of interest corresponding to the surgery assistance information 123 currently displayed and the emphasized display for the other regions of interest different, so that it can be determined which region of interest the surgery assistance information 123 is displayed for. For example, the shape, color, size, thickness of the emphasized display can be changed, or it can be made to flicker.

[0151] Also, when it is desired to switch the region of interest for which the surgery assistance information 123 is displayed, it is preferable to be able to operate via the UI 16 on the display 15. For example, a click by a mouse pointer directed at the region of interest for which the surgery assistance information 123 is displayed, a tap on a touch panel can be used as a trigger for switching. As described later, the surgery assistance information presentation switch 12g can also be used, and the method of switching is not limited to the above.

[0152] In the surgery assistance information presentation screen 120, by default, the surgery assistance information 123 related to the treatment instrument with the highest rank of appropriateness is displayed. In the surgery assistance information presentation screen 120, the surgery assistance information 123 related to the treatment instrument with the highest rank of appropriateness is displayed by default. Figure 29 In the surgery assistance information presentation screen 120, as an operation icon for switching the surgery assistance information 123 in the display to the surgery assistance information 123 related to the treatment instrument with a lower rank of appropriateness, a lower direction rank switching icon 126a is displayed, and as an operation icon for switching the surgery assistance information 123 in the display to the surgery assistance information 123 related to the treatment instrument with a higher rank of appropriateness, an upper direction rank switching icon 126b is displayed (see FIG. 6). Figure 30 The lower direction rank switching icon 126a is provided to the right of the surgery assistance information 123, and the upper direction rank switching icon 126b is provided to the left of the surgery assistance information 123. The lower direction rank switching icon 126a is represented by a triangular icon, and faces the right direction. The upper direction rank switching icon 126b is represented by a triangular icon, and faces the left direction. By operating the lower direction rank switching icon 126a, the surgery assistance information 123 related to the treatment instrument with a lower rank of appropriateness is switched to. On the other hand, by operating the upper direction rank switching icon 126b, the surgery assistance information 123 related to the treatment instrument is switched to the surgery assistance information related to the treatment instrument with a higher rank. In addition, the shape and display form of the lower direction rank switching icon 126a and the upper direction rank switching icon 126b are not limited to this.

[0153] As for the operation of the lower-order direction switching icon 126a and the upper-order direction switching icon 126b, it is preferable that the user operate using a mouse or a touch panel of the UI 16 or the like. Also, it is possible to operate in combination with the UI 16 and the surgery assistance information prompting switch 12g. For example, if the user selects either the lower-order direction order switching icon 126a or the upper-order direction order switching icon 126b via the mouse or the touch panel of the UI 16, and then presses the surgery assistance information prompting switch 12g after activating the selected icon, the display corresponding to the activated icon is switched.

[0154] Next, the switching of the surgery assistance information 123 related to the treatment instrument using the lower-order direction switching icon 126a and the upper-order direction switching icon 126b will be described. For example, as shown in Figure 29 if the surgery assistance information prompting switch 12g is pressed after the lower-order direction switching icon 126a is activated in a case where the surgery assistance information 123 related to the treatment instrument whose appropriateness order is first is displayed, the surgery assistance information 123 related to the treatment instrument whose appropriateness order is second is switched, whereby the display of the treatment instrument name, the treatment method, the appropriateness of the treatment instrument, the appropriateness order, and the image 125 of the treatment instrument is switched, and the screen shown in Figure 30 is displayed. That is, it is also possible to cause the surgery assistance information prompting switch 12g to function as a trigger switch that switches the display of the appropriate treatment instrument related to the discrimination result.

[0155] In addition, it is also possible to newly provide an order switching button in the endoscope 12 or the UI 16.

[0156] In addition, when the display of the surgery assistance information 123 of the order before or after is desired to be switched, it is possible to operate via the UI 16 on the display 15. For example, it is possible to use a click, a tap, or a sliding action of a mouse pointer aimed at a region of interest where the surgery assistance information 123 is displayed as a trigger for the switching, but the present technology is not limited thereto. With the above structure, the user can compare the treatment instruments whose appropriateness is high according to the appropriateness order.

[0157] In a case where there is no surgery assistance information 123 related to a treatment instrument whose appropriateness of the treatment instrument satisfies a certain condition, or in a case where the appropriateness of the treatment instrument is lower than a set threshold value, the 1st surgery assistance information calculation section 60 transmits an instruction to perform a warning display to the display control section 61, and a warning display 130 is performed on the surgery assistance information prompting screen 120. For example, as shown in Figure 31As illustrated, it can be displayed as "Unable to present treatment instrument." The warning display is not limited to this method. For example, in a case where the appropriateness of the treatment instrument is lower than a set threshold, the character color of the appropriateness of the treatment instrument can be changed, or it can be displayed with flickering, or in a thick frame. It can also be warned with a lamp (not illustrated) or sound. With the above structure, in the absence of an appropriate treatment instrument, or in a case where the appropriateness of the treatment instrument is low, the user can be prevented from performing a treatment with low safety.

[0158] On the surgery assistance information presentation screen 120, the user selects one of the one or more treatment instrument-related surgery assistance information 123, and presses the treatment start button 152, and the treatment instrument-related surgery assistance information 123 calculated by the first surgery assistance information calculation section 60 is input to the treatment time calculation section 62, and the treatment expected time is output.

[0159] Specifically, if the user selects the treatment start button 152 indicated by "Treatment Start" on the surgery assistance information presentation screen 120 as illustrated, Figure 32 the treatment instrument-related surgery assistance information 123 in the currently displayed is input to the treatment time calculation model 62M of the treatment time calculation section 62, and the treatment time calculation model 62M outputs the treatment expected time related to the selected treatment instrument. At this time, the treatment expected time 140 is displayed. For example, it is preferable that, as illustrated, Figure 33 when the surgery assistance information 123 output by the first surgery assistance information calculation section 60 is the surgery assistance information P60p, the treatment time calculation model 62M outputs the treatment expected time P62p, as illustrated, Figure 34 when the surgery assistance information 123 output by the first surgery assistance information calculation section 60 is the surgery assistance information Q60q, the treatment time calculation model 62M outputs the treatment expected time Q62q.

[0160] With the above structure, the user can be presented with the treatment expected time when using the treatment instrument selected by the user, and the estimated surgery time can be notified. In addition, regarding the treatment expected time, the treatment expected time 140 can be displayed on the surgery assistance information presentation screen 120 when the appropriate treatment instrument is initially presented to the user. At this time, one or more surgery assistance information 123 output by the first surgery assistance information calculation section 60 is directly input to the treatment time calculation model 62M of the treatment time calculation section 62. In addition, the procedure required for using the treatment instrument, such as marking of the lesion or water injection into the submucosal layer, or the instrument such as a syringe, a needle, a reagent for staining, or the amount of a local anesthetic, can be displayed together with the treatment expected time 140.

[0161] In Figure 35On the surgery assistance information prompt screen 120, a consent button 150 indicated by "Yes" and a denial button 151 indicated by "No" for the user to consent to or deny the use of the treatment instrument displayed as the surgery assistance information 123, a treatment start button 152, an examination image saving button 153, and a screen shot button 154 are displayed. The user can input the content of the treatment instrument prompted on the surgery assistance information prompt screen 120 to the processor device 14 using the consent button 150. Also, the user can input the content of the treatment instrument prompted on the surgery assistance information prompt screen 120 not to be used to the processor device 14 using the denial button 151. The surgery assistance information prompt mode can be ended and switched to the examination image display mode by pressing the consent button 150 or the denial button 151. The result of the input by the user is transmitted to the surgery assistance information calculation model 60M (not shown). With the above structure, the surgery assistance information calculation model 60M can be caused to learn using the result of the input and output of the surgery assistance information calculation model 60M.

[0162] If the user presses the treatment start button 152 after the treatment expected time 140 is displayed and actually starts the treatment, the treatment start time can be recorded in the storage (not shown). With the above structure, the treatment start time can be recorded in the examination recording system and the electronic medical record or the like. Also, as described later, when the examination image 121 is acquired as a still image, the treatment instrument and the time taken for the actual treatment output by the treatment time calculation model 62M are associated and input again to the treatment time calculation model 62M, and the treatment time calculation model 62M can be caused to learn.

[0163] In Figure 35 On the surgery assistance information prompt screen 120, if the user presses the examination image saving button 153, the still image acquisition unit 63 acquires the examination image 121 as a still image and transmits it to the diagnosis result input unit 64. The diagnosis result input unit 64 associates the examination image 121 and the discrimination result calculated by the discrimination processing unit 55e and stores them in the storage (not shown) or the surgery information storage 80. Alternatively, instead of the examination image saving button 153, the still image acquisition instruction switch 12h can be used as a trigger to acquire the examination image. With the above structure, the examination image can be recorded in the examination recording system and the electronic medical record or the like.

[0164] A screenshot button 154 is displayed on the surgical assistance information prompt screen 120. If the user presses the screenshot button 154, the surgical assistance information prompt screen 120 displayed on the monitor 15 can be captured as a still image and saved in the memory (not shown) or the surgical information storage memory 80. With the above structure, the surgical assistance information prompt screen 120 can be recorded in the storage medium such as the examination recording system or electronic medical record, and the collection of case records can be easily carried out during the examination.

[0165] Along Figure 36 The flowchart shown illustrates the switching of the surgical assistance information prompting mode and a series of processes in the first embodiment. Normally, in the examination image display mode, an examination image is acquired (step S101). The examination image is input to the identification processing unit 55e, and the identification processing unit 55e outputs an identification result related to the examination image (step S102). At this time, if the surgical assistance information prompting switch 12g is pressed, the surgical assistance information prompting mode switching unit 56 is operated to switch to the surgical assistance information prompting mode (step S103). If the surgical assistance information prompting switch 12g is not pressed, in the examination image display mode, the examination image is sent to the display control unit 61 (step S104). By pressing the surgical assistance information prompting switch 12g, the identification result and the surgical information in the surgical information storage memory 80 are input to the first surgical assistance information calculation unit 60 (step S105). The first surgical assistance information calculation unit 60 outputs surgical assistance information 123 that meets specific conditions and sends it to the display control unit 61 (step S106). Next, the user selects surgical assistance information 123 related to any treatment instrument (step S107). Through this operation, the surgical assistance information 123 selected by the user is input into the treatment time calculation unit 62 (step S108), and the expected treatment time is output (step S109). Through the above control, finally, the examination image 121, surgical assistance information 123, and expected treatment time 140 are sent to the display control unit 61 and displayed on the display unit.

[0166] [Second Implementation]

[0167] Following this paragraph, the surgical assistance information prompting mode of the second embodiment will be described. In the second embodiment, as... Figure 38 As shown, instead of the first surgical assistance information calculation unit 60 of the first embodiment, a template matching type second surgical assistance information calculation unit 200 that does not use a learning model is provided. A template image storage memory 202 is connected to the second surgical assistance information calculation unit 200. The template image storage memory 202 is used for template matching with the second illumination image and storing the completed diagnostic template image that is associated with the treatment instrument. In addition, the second embodiment and...Figure 15 The first embodiment is the same as shown (the surgical information storage memory 80 is the same as the first embodiment).

[0168] The template image stored in the template image storage memory 202 is a white light image or a special light image. The diagnosed white light image or the diagnosed special light image, which serves as the template image after diagnosis, is preferably compared after color matching of the image's hue, tone, etc., so that it can be compared with the second illumination light image or the overlapping image.

[0169] As a treatment instrument that establishes a corresponding association with the diagnosed template image, it is preferably a treatment instrument used by a doctor or other personnel to observe the object in the template image. For example, if the object observed in the template image is a pedunculated polyp, the treatment instrument that establishes a corresponding association with the template image is preferably a polyp remover or the like. Furthermore, if the object observed in the template image contains bleeding, a hemostatic forceps or the like is preferably used to stop the bleeding.

[0170] The diagnostic results that are associated with the template image after diagnosis are the same as the diagnostic results output in the identification processing unit 55e of the first embodiment, so they are omitted from the description.

[0171] The following describes in detail the surgical assistance information prompting mode of the second embodiment. If the surgical assistance information prompting switch 12g is operated to switch to the surgical assistance information prompting mode, then as follows... Figure 39 As shown, the second illumination image is sent from the image processing unit 55 to the feature calculation unit 203 of the second illumination image within the second surgical assistance information calculation unit 200. The diagnosed template image is sent from the template image storage memory 202 to the feature calculation unit 204 of the template image within the second surgical assistance information calculation unit 200. The calculated feature values ​​of the second illumination image and the diagnosed template image are sent to the comprehensive matching degree calculation unit 205. Then, a comprehensive matching degree is calculated between the feature values ​​of the second illumination image and the feature values ​​of the template image.

[0172] Specific examples of feature quantity calculation and overall matching degree calculation in the second surgical assistance information calculation unit 200 will be explained. Values ​​related to the feature quantity items are calculated for both the second illumination image and the diagnosed template image. Specifically, the feature quantity calculation unit 203 of the second illumination image calculates the feature quantity related to the shape and branching of the superficial blood vessels as A1, the feature quantity related to the uniformity of the superficial blood vessels as A2, etc., and the feature quantity calculation unit 204 of the template image calculates the feature quantity related to the shape and branching of the superficial blood vessels as a1, the individual feature quantity related to the uniformity of the superficial blood vessels as a2, etc. The feature quantities related to each feature quantity item are sent to the overall matching degree calculation unit 205.

[0173] In the comprehensive matching degree calculation unit 205, firstly, the values ​​of A1 and a1 are compared regarding the similarity related to the shape and branching of the superficial blood vessels. Figure 39 In the diagram (labeled "A1 vs a1"), the matching degree is calculated as α1. The matching degree (here, α1) related to a feature quantity such as "shape and branching of superficial blood vessels" is called the individual matching degree. Next, the sum of the individual matching degrees is calculated as the overall matching degree. The overall matching degree between the second illumination image or the overlapping image and the template image is calculated as the value αβγ.

[0174] The template image selection unit 207 selects a template image whose overall matching degree meets the first condition. The first condition is preferably an overall matching degree of at least the threshold used for the first condition. The appropriateness calculation unit 209, referring to the surgical information storage memory 80, calculates the appropriateness of the treatment instruments that are associated with the selected template image. For example, ... Figure 40 As shown, when three template images K, L, and M are selected by the template image selection unit 207, the suitability of the treatment instruments X, Y, and Z corresponding to each template image K, L, and M is calculated by referring to the inventory information of treatment instruments X, Y, and Z contained in the surgical information storage memory 80. Furthermore, the surgical information, including the inventory information of treatment instruments, is the same as in the first embodiment, so its description is omitted.

[0175] An example is given where the appropriateness of a surgical instrument is calculated solely based on inventory information. For instance, such as... Figure 40 As shown, in the inventory information of the surgical information storage memory 80, when the inventory of treatment device X is "0", the inventory of treatment device Y is "10", and the inventory of treatment device Z is "5", the appropriateness YP of treatment device Y is calculated to be the highest, and the appropriateness XP of treatment device X is the lowest. The appropriateness ZP of treatment device Z is set as a value between appropriateness YP and appropriateness XP.

[0176] The surgery assistance information output unit 211 outputs surgery assistance information including a treatment instrument whose appropriateness satisfies the second condition. The second condition is preferably a condition in which the appropriateness is equal to or higher than a second condition threshold value. In Figure 40 the case, the treatment instruments Y and Z whose appropriateness is equal to or higher than the appropriateness ZP are regarded as appropriate treatment instruments, and surgery assistance information including the treatment instruments Y and Z is output. With respect to the output surgery assistance information, the same as in the first embodiment, the surgery assistance information is displayed on the surgery assistance information presentation screen 120 together with the examination screen. The display form of the surgery assistance information presentation screen 120 is the same as in the first embodiment, and thus the description is omitted.

[0177] In the surgery assistance information presentation screen 120 of the second embodiment Figure 35 If the user presses the examination image saving button 153, the still image acquisition unit 63 acquires the examination image 121 as a still image and sends it to the diagnosis result input unit 64. As shown in Figure 39 the diagnosis result input unit 64 can associate the examination image 121 and the discrimination result calculated by the discrimination processing unit 55e with respect to the examination image and save them in the template image storage memory 202. Alternatively, instead of the examination image saving button 153, a still image acquisition instruction switch 12h can be used as a trigger for acquiring the examination image. With the above-described configuration, it is possible to add a template image to the surgery information storage memory 80.

[0178] With respect to the display of the attention region on the surgery assistance information presentation screen 120, the switching of the display screen according to the order of the appropriateness, the warning display in the case where there is no surgery assistance information 123 related to a treatment instrument satisfying a specific condition or the appropriateness of the treatment instrument is lower than a set threshold value, the agree button 150, the disagree button 151, the treatment start button 152 at the time of performing a treatment, the examination image saving button 153, the screen shot button 154, and the switching of the display by the use of the order switching icon, the same as in the first embodiment, and thus the description is omitted.

[0179] Along with Figure 41The flowchart shown explains a series of processes of the surgery assisting information presentation mode of the second embodiment. This is generally the examination image display mode. At this time, if the surgery assisting information presentation switch 12g is pressed, the surgery assisting information presentation mode switching section 56 is operated to switch to the surgery assisting information presentation mode (step S201). In the case where the surgery assisting information presentation switch 12g is not pressed, it is the examination image display mode, and the superimposed image is sent to the display control section 61 (step S202). In the case of the surgery assisting information presentation mode, the template image is sent from the template image storage memory 202 to the feature amount calculation section 204 of the template image of the second surgery assisting information calculation section 200 (step S203), and the feature amount related to the template image is calculated by the feature amount calculation section 204 of the template image (step S204). Also, the second illumination light image is sent from the superimposed image generation section 55c to the feature amount calculation section 203 of the second illumination light image of the second surgery assisting information calculation section 200 (step S205), and the feature amount related to the second illumination light image is calculated by the feature amount calculation section 203 of the second illumination light image (step S206).

[0180] Next, the feature amount related to the template image and the feature amount related to the second illumination light image are sent to the comprehensive matching degree calculation section 205, and the individual matching degree and the comprehensive matching degree are calculated (step S207). Next, the template image for which the comprehensive matching degree satisfies the first condition is selected (step S208). Next, the appropriateness of the treatment instrument for which a corresponding association is established with the selected template image is calculated with reference to the surgery information including the inventory information of the treatment instrument (step S209). Next, the surgery assisting information including the treatment instrument for which the appropriateness satisfies the second condition is output (step S210). The superimposed image and the treatment instrument included in the surgery assisting information are displayed on the display 15 (step 211).

[0181] In addition, in the above-described embodiments, the medical image processing apparatus of the present application is applied to an endoscope system that acquires endoscope images as medical images, but of course, it can be applied to various endoscope systems such as a capsule endoscope, and the medical image processing apparatus of the present application can be applied to other various medical image apparatuses that acquire X-ray images, CT images, MRI images, ultrasonic wave images, pathological images, PET (Positron Emission Tomography) images, and the like as medical images.

[0182] In the above-described embodiments, the hardware structure of the processing units (processing units) in which the processor 21 for the light source, the processor 44 for the imaging, the image acquisition section 50, the DSP 52, the noise reduction section 53, the image processing switching section 54, the authentication processing section 55e, the central control section 70, and the like execute various processes is various processors as shown below. Among the various processors, there are a general-purpose processor, that is, a CPU (Central Processing Unit) that executes software (programs) to function as various processing units, a processor that can change the circuit structure after manufacture, that is, a programmable logic device (PLD) such as an FPGA (Field Programmable Gate Array), a processor having a circuit structure designed specifically to execute various processes, that is, an application-specific circuit, and the like.

[0183] One processing unit can be constituted by one of these various processors, or can be constituted by a combination of two or more processors of the same kind or different kinds (for example, a combination of a plurality of FPGAs or CPUs and FPGAs). Also, one processor can constitute a plurality of processing units. As an example in which one processor constitutes a plurality of processing units, first, there is a mode in which a computer such as a client and a server is constituted by a combination of one or more CPUs and software, and the processor functions as a plurality of processing units. Second, there is a mode in which a system on chip (SoC) or the like is used, and a processor that realizes the functions of the entire system including a plurality of processing units by one IC (Integrated Circuit) chip is used. In this way, various processing units are constituted by one or more of the above-described various processors as the hardware structure.

[0184] Furthermore, the hardware structure of these various processors is, more specifically, a circuit (circuitry) in which circuit elements such as semiconductor elements are combined. Also, the hardware structure of the storage section is a storage device such as an HDD (hard disc drive) or an SSD (solid state drive).

[0185] Symbol Explanation

[0186] 10-Endoscope system, 12-Endoscope, 12a-Insertion section, 12b-Operation section, 12c-Bending section, 12d-Tip section, 12f-View mode switching switch, 12g-Surgical assistance information presentation switch, 12h-Still image acquisition instruction switch, 12i-Zoom operation section, 13-Illumination device, 14-Processor device, 15-Display, 16-UI, 20-Illumination section, 20a-V-LED, 20b-B-LED, 20c-G-LED, 20d-R-LED, 21-Processor for illumination, 22-Optical path combining section, 23-Light guide, 30a-Illumination optical system, 30b-Imaging optical system, 31-Illumination lens, 41-Objective lens, 42-Zoom lens, 43-Imaging sensor, 44-Processor for imaging, 45-CDS / AGC circuit, 46-A / D converter, 50-Image acquisition section, 52-DSP, 53-Noise reduction section, 54-Image processing switching section, 55-Image processing section, 55a-1st illumination light image generation section, 55b-2nd illumination light image generation section, 55c-Overlapped image generation section, 55d-Image for display generation section, 55e-Discrimination processing section, 55eM-Discrimination model, 55ep-Discrimination result P, 55eq-Discrimination result Q, 55f-Overlapping processing section, 56-Surgical assistance information presentation mode switching section, 60-1st surgical assistance information calculation section, 60M-Surgical assistance information calculation model, 60a-Surgical assistance information A, 60b-Surgical assistance information B, 60c-Surgical assistance information C, 60d-Surgical assistance information D, 60e-Surgical assistance information E, 60f-Surgical assistance information F, 60g-Surgical assistance information G, 60h-Surgical assistance information H, 60i-Surgical assistance information I, 60j-Surgical assistance information J, 60p-Surgical assistance information P, 60q-Surgical assistance information Q, 61-Display control section, 62-Treatment time calculation section, 62M-Treatment time calculation model, 62p-Treatment expected time P, 62q-Treatment expected time Q, 63-Still image acquisition section, 64-Diagnosis result input section, 70-Central control section, 80-Surgical information storage memory, 81-Inventory information of treatment instrument, 81a-Inventory information of treatment instrument S, 81b-Inventory information of treatment instrument T, 82-Treatment mode information, 82a-Treatment mode information S, 82b-Treatment mode information T, 83-Peripheral environment information, 83a-Peripheral environment information S, 83b-Peripheral environment information T, 84-Patient risk determination information, 84a-Patient risk determination information S, 84b-Patient risk determination information T, 85-Surgeon information, 85a-Surgeon information S, 85b-Surgeon information T, 120-Surgical assistance information presentation screen, 121-Inspection image, 123-Surgical assistance information, 125-Image of treatment instrument, 126a-Icon for rank switching in lower direction, 126b-Icon for rank switching in upper direction,130 - warning display, 140 - treatment expected time, 150 - agree button, 151 - negative button, 152 - treatment start button, 153 - inspection image save button, 154 - screen shot button, 200 - 2nd surgery assistance information calculation section, 202 - template image storage memory, 203 - 2nd illumination light image feature amount calculation section, 204 - template image feature amount calculation section, 205 - comprehensive matching degree calculation section, 207 - template image selection section, 209 - appropriateness calculation section, 211 - surgery assistance information output section.

Claims

1. An endoscope system comprising: an image control processor configured to: acquire an examination image; perform a discrimination process on the examination image; calculate surgery assistance information related to a treatment instrument, with reference to a result of the discrimination process; and display the examination image and the treatment instrument included in the surgery assistance information on a display, the image control processor has a learning model, the learning model is a learned learning model, the learning model outputs one or more of the surgery assistance information by improving appropriateness related to the treatment instrument through input of the discrimination result and surgery information, the surgery information being one of surgery information including the following information or being obtained by combining one or more of the following information: inventory information of the treatment instrument; correspondence information of a lesion and the treatment instrument according to a guideline, that is, treatment pattern information; peripheral environment information related to an anatomical site of a throat, an esophagus, a stomach, a small intestine, a large intestine, and a course of a blood vessel; patient risk determination information of a patient's past history, examination / treatment history, and a medicine in use; and surgeon information, the appropriateness of the treatment instrument is a score indicating a degree of appropriateness of the treatment instrument.

2. The endoscope system according to claim 1, wherein the learning model outputs different surgery assistance information when input of the inventory information of the treatment instrument is different.

3. The endoscope system according to claim 1, wherein the learning model outputs different surgery assistance information when input of the treatment pattern information is different.

4. The endoscope system according to claim 1, wherein the learning model outputs different surgery assistance information when input of the peripheral environment information is different.

5. The endoscope system according to claim 1, wherein the learning model outputs different surgery assistance information when input of the patient risk determination information is different.

6. The endoscope system according to claim 1, wherein the learning model outputs different surgery assistance information when input of the surgeon information is different.

7. The endoscope system according to claim 1, wherein the learning model outputs a treatment anticipation time through input of the surgery assistance information related to the treatment instrument.

8. The endoscope system according to any one of claims 1 to 7, wherein the surgery assistance information is displayed after a surgery assistance information prompting button is pressed.

9. The endoscope system according to any one of claims 1 to 7, wherein the surgery assistance information includes a name of the treatment instrument, appropriateness related to the treatment instrument, and a rank of the appropriateness related to the treatment instrument.

10. The endoscope system according to claim 9, wherein display of the surgery assistance information can be switched according to the rank of the appropriateness.

11. The endoscope system according to claim 9, wherein ​ ​ ​ ​ ​ In a case where the propriety related to the treatment instrument does not satisfy a certain condition, a warning display is performed.

12. The endoscope system according to claim 11, wherein In a case where a plurality of the treatment instruments satisfying the certain condition are selected, the certain condition is changeable.

13. The endoscope system according to claim 1, wherein The endoscope system is provided with a light source processor, The light source processor performs the following processing: controlling light emission of first illumination light and second illumination light whose emission spectra are different from each other; in a case of automatically switching a first illumination period in which the first illumination light is emitted and a second illumination period in which the second illumination light is emitted, emitting the first illumination light in a first light emission mode and emitting the second illumination light in a second light emission mode; and acquiring a first illumination light image based on the first illumination light, a second illumination light image based on the second illumination light, and an overlay image in which a result of the discrimination processing for the second illumination light image is overlaid on a display image based on the first illumination light image as the examination image.

14. A working method of an endoscope system provided with an image control processor, wherein The image control processor performs the following processing: acquiring an examination image; performing discrimination processing on the examination image; calculating surgery assistance information related to a treatment instrument with reference to a result of the discrimination processing, i.e., a discrimination result; and displaying the examination image and the treatment instrument included in the surgery assistance information on a display, The image control processor has a learning model, The learning model is a learned learning model, The learning model outputs one or more of the surgery assistance information by improving propriety related to the treatment instrument with input of the discrimination result and surgery information, which is one of surgery information including the following information or obtained by combining one or more thereof: inventory information of the treatment instrument; correspondence association information of a lesion and the treatment instrument according to a guideline, i.e., treatment pattern information; peripheral environment information related to an anatomical site of a throat, an esophagus, a stomach, a small intestine, a large intestine, and a course of a blood vessel; patient risk determination information of a patient's past history, examination / treatment history, and a medical product in use; and surgeon information, The propriety of the treatment instrument is a score indicating a degree of appropriateness of the treatment instrument.

Images