Information support device, endoscope system, information support method, and non-temporary information storage medium

Abstract

To realize information support for reducing a burden on an operator.SOLUTION: An information support device 31 includes a processor 311 that processes an endoscopic image captured by an endoscope device 21 that images the inside of a subject. The processor 311 sets a specific region in the endoscopic image for a region of interest, recognizes a portion to note in the subject captured in the endoscopic image, recognizes a treatment tool captured in the endoscopic image, calculates a distance between the portion to note and the treatment tool, and causes a notification unit 34 to notify specific information in a case where the distance is equal to or less than a specific threshold value and the portion to note is located outside the region of interest.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information support device, an endoscope system, an information support method, and a non-transitory information storage medium. [Background technology]

[0002] BACKGROUND ART Conventionally, an information support system that supports a surgeon is known (see, for example, Patent Document 1). The information support system (surgery support system) described in Patent Document 1 supports the surgeon by issuing a warning to inform him / her that the treatment tool is approaching a critical area in the living tissue during surgery. The critical area corresponds to a dangerous area that the treatment tool must not come into contact with. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-233240 Summary of the Invention [Problem to be solved by the invention]

[0004] Meanwhile, a surgeon performs a treatment while focusing on the area surrounding the tip of a treatment tool and recognizing a site of interest that is close to the treatment tool. For this reason, a technology that constantly issues a warning when a treatment tool approaches a site of interest, such as the information support system described in Patent Document 1, issues a warning even when the site of interest is within the area the surgeon is focusing on, which may interfere with the treatment by the surgeon. Meanwhile, because the surgeon does not focus on areas outside the area of ​​interest, the surgeon may accidentally bring the treatment tool into contact with a site of interest that is outside the area of ​​interest. Therefore, there is a demand for a technology that can provide appropriate information to the surgeon and realize information support that reduces the burden on the surgeon.

[0005] The present invention has been made in consideration of the above, and aims to provide an information support device, an endoscopic system, an information support method, and a non-transitory information storage medium that can realize information support that reduces the burden on the surgeon. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the information support device of the present invention is an information support device equipped with a processor that processes endoscopic images captured by an endoscopic device that captures images inside a subject, wherein the processor sets a specific area in the endoscopic image as a region of interest, recognizes a site of interest in the subject that appears in the endoscopic image, recognizes a treatment tool that appears in the endoscopic image, calculates the distance between the site of interest and the treatment tool, and causes an alarm unit to notify specific information if the distance is equal to or less than a specific threshold and the site of interest is located outside the region of interest.

[0007] An endoscope system according to the present invention includes the above-described information support device and an endoscope device that captures images of the inside of the subject.

[0008] The information support method of the present invention is an information support method executed by an information support device, which sets a specific area in an endoscopic image captured by an endoscopic device that captures images inside a subject as a region of interest, recognizes a region of interest in the subject that appears in the endoscopic image, recognizes a treatment tool that appears in the endoscopic image, calculates the distance between the region of interest and the treatment tool, and causes an alarm unit to notify specific information if the distance is equal to or less than a specific threshold and the region of interest is located outside the region of interest.

[0009] The non-transitory information storage medium of the present invention is a non-transitory information storage medium that stores an information assistance program to be executed by an information assistance device, and the information assistance program causes the information assistance device to perform the following processes: set a specific area in an endoscopic image captured by an endoscopic device that captures images inside a subject as a region of interest, recognize a region of interest in the subject that appears in the endoscopic image, recognize a treatment tool that appears in the endoscopic image, calculate the distance between the region of interest and the treatment tool, and cause an alarm unit to notify specific information if the distance is equal to or less than a specific threshold and the region of interest is located outside the region of interest. [Effects of the Invention]

[0010] According to the information support device, endoscope system, information support method, and non-transitory information storage medium of the present invention, it is possible to realize information support that reduces the burden on doctors. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing a configuration of an endoscope system according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing the functions of the second processor. [Figure 3] FIG. 3 is a flowchart showing an information support method according to an embodiment. [Figure 4] FIG. 4 is a diagram illustrating the information support method. [Figure 5] FIG. 5 is a diagram illustrating the information support method. [Figure 6] FIG. 6 is a diagram illustrating the information support method. [Figure 7] FIG. 7 is a diagram illustrating a first modification of the embodiment. [Figure 8] FIG. 8 is a diagram illustrating a second modification of the embodiment. [Figure 9] FIG. 9 is a diagram illustrating a second modification of the embodiment. [Figure 10] FIG. 10 is a diagram illustrating a second modification of the embodiment. [Figure 11] FIG. 11 is a diagram illustrating a third modification of the embodiment. [Figure 12] FIG. 12 is a diagram illustrating a third modification of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, a mode for carrying out the present invention (hereinafter referred to as an embodiment) will be described with reference to the drawings. Note that the present invention is not limited to the embodiment described below. Furthermore, in the description of the drawings, the same parts are given the same reference numerals.

[0013] [Configuration of endoscope system] FIG. 1 is a block diagram showing the configuration of an endoscope system 1 according to an embodiment. The endoscope system 1 is used in the medical field to provide information support to an operator while observing the inside of a subject. As shown in FIG. 1, the endoscope system 1 includes an observation system 2 and an information support system 3.

[0014] [Configuration of observation system] The observation system 2 is a system for observing the inside of a subject. As shown in Fig. 1, the observation system 2 includes an endoscope device 21, a light source device 22, a first display device 23, a first input device 24, and a control device 25.

[0015] As shown in FIG. 1, the endoscope device 21 includes an insertion section 211 and an imaging section 212. The insertion section 211 is configured by a so-called rigid endoscope. That is, the insertion section 211 is rigid or at least partially flexible, has an elongated shape, and is inserted into a subject. The endoscope device 21 according to this embodiment is configured by a 3D scope. The insertion section 211 is configured by a twin-lens relay type or monocular split-pupil type scope described below.

[0016] In a twin-lens relay scope, two parallel optical paths are arranged within the scope. Each of the two optical paths has an optical system. The twin-lens relay scope takes in and emits observation light for the right and left eyes, each of which has a parallax between them, through the two optical systems (see, for example, Japanese Patent Application Laid-Open No. 6-160731).

[0017] Furthermore, in a monocular split-pupil scope, one optical path is provided within the scope. An optical system is disposed in this optical path. Furthermore, a pupil division unit that divides the light beam within the pupil into two regions is provided at the pupil position of the optical system. In a monocular split-pupil scope, the optical system takes in observation light, and the pupil division unit separates the observation light into observation light for the right and left eyes, which have a parallax between them, and outputs the separated light (see, for example, Japanese Patent Application Laid-Open No. 6-59199).

[0018] Although not specifically shown in the drawings, the imaging unit 212 includes a right-eye imaging unit that captures the right-eye observation light (subject image) emitted from the insertion unit 211, and a left-eye imaging unit that captures the left-eye observation light (subject image) emitted from the insertion unit 211. These right- and left-eye imaging units are configured with a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) that receives the subject image and converts it into an electrical signal, and generate an image signal by capturing the subject image. Note that, hereinafter, the image signal generated by the imaging unit 212 will be referred to as a captured image.

[0019] The light source device 22 supplies observation light to be irradiated onto the subject to a light guide (not shown) under the control of the control device 25. Examples of the observation light include white light including a visible wavelength band, excitation light that excites a fluorescent agent such as indocyanine green, and narrow band light used in NBI (Narrow Band Imaging). The observation light supplied to the light guide passes through the insertion section 211 and is irradiated onto the subject from the tip of the insertion section 211.

[0020] The first display device 23 is an LCD (Liquid Crystal Display) or an EL (Electro Luminescence) display, and displays a display image generated by the control device 25 under the control of the control device 25.

[0021] The first input device 24 is configured using a keyboard, a mouse, a switch, a touch panel, etc., and accepts user operations by a user such as a surgeon. Then, the first input device 24 outputs an operation signal to the control device 25 in accordance with the user operation.

[0022] As shown in FIG. 1, the control device 25 includes a first processor 251, a first storage unit 252, and a first external interface 253.

[0023] The first processor 251 is configured to include a controller such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array), and controls the operation of the entire observation system 2. For example, the first processor 251 acquires a captured image generated by the imaging section 212 and generates a display image (hereinafter referred to as an endoscopic image) by performing predetermined image processing on the captured image. The first processor 251 then displays the endoscopic image on the first display device 23. The endoscopic image is output to the outside via a first external interface 253.

[0024] The first storage unit 252 stores various programs executed by the first processor 251, information necessary for the processing of the first processor 251, and the like.

[0025] The first external interface 253 is an interface that connects to an external device so as to be able to send and receive information.

[0026] [Configuration of information support system] The information support system 3 is a system that supports users such as surgeons. As shown in Fig. 1, the information support system 3 includes an information support device 31, a second display device 32, a second input device 33, and a speaker 34.

[0027] As shown in FIG. 1, the information support device 31 includes a second processor 311, a second storage unit 312, and a second external interface 313.

[0028] FIG. 2 is a block diagram showing the functions of the second processor 311. As shown in FIG. The second processor 311 includes a controller such as a CPU or MPU, or an integrated circuit such as an ASIC or FPGA, and controls the overall operation of the information support system 3. As shown in Fig. 2, the second processor 311 has the functions of an image acquisition unit 311A, an attention region recognition unit 311B, a display control unit 311C, a surgical scene recognition unit 311D, a treatment tool recognition unit 311E, a region of interest setting unit 311F, a distance calculation unit 311G, and a notification control unit 311H. Details of each of the functions 311A ​​to 311H in the second processor 311 will be described in the "Information Support Method" section below.

[0029] The second storage unit 312 stores various programs (including the information assistance program according to the present invention) executed by the second processor 311, and information necessary for the processing of the second processor 311. Examples of the information necessary for the processing of the second processor 311 include first to third trained models and first and second relationship information. Details of the first to third trained models and the first and second relationship information will be described in the "Information Assistance Method" section below.

[0030] The second external interface 313 is an interface that connects to an external device so as to be able to send and receive information to and from the external device. For example, the second external interface 313 receives an endoscopic image from the observation system 2 via the first external interface 253.

[0031] The second display device 32 is an LCD or EL display, and displays a display image (hereinafter referred to as an information support image) generated by the display control unit 311C under the control of the display control unit 311C.

[0032] The second input device 33 is configured using a keyboard, a mouse, a switch, a touch panel, etc., and accepts user operations by a user such as a surgeon. The second input device 33 then outputs an operation signal to the information support device 31 in accordance with the user operation.

[0033] The speaker 34 corresponds to the notification unit according to the present invention, and notifies specific information by voice under the control of the notification control unit 311H.

[0034] [Information support method] Next, an information support method executed by the information support device 31 will be described. Fig. 3 is a flowchart showing an information support method according to an embodiment. Figs. 4 to 6 are diagrams illustrating the information support method. Fig. 4 is a diagram showing first related information. Fig. 5 is a diagram showing second related information. Fig. 6 is a diagram showing an example of an information support image F displayed on the second display device 32.

[0035] The endoscope system 1 is already set in the following state. That is, the insertion section 211 is inserted into the subject, and the observation region of the observation system 2 is a region inside the subject. That is, an endoscopic image that has been generated by the imaging section 212 and subjected to predetermined image processing by the first processor 251 is displayed on the first display device 23. Then, the surgeon performs treatment on the living tissue inside the subject while checking the endoscopic image displayed on the first display device 23.

[0036] First, the image acquisition unit 311A ​​acquires an endoscopic image from the observation system 2 via the first and second external interfaces 253 and 313 (step S1).

[0037] After step S1, the attention region recognition unit 311B recognizes an attention region reflected in the endoscopic image acquired in step S1 by performing an estimation process on the endoscopic image using the first trained model stored in the second storage unit 312 (step S2). In the case of colon surgery, examples of the attention region include the ureter, hypogastric nerve, and artery.

[0038] Here, the first trained model is a trained model obtained by repeatedly executing a training process on the trained model using a plurality of sets of training images and training data, each set being a training image and training data. The training images are captured images of the inside of a subject. The training data are data annotated with the type, correct position, and size of the region of interest in the training images. The trained model used in the training process is, for example, a convolutional neural network (CNN). The first trained model includes a weight file (training parameters) having weight values ​​and bias values ​​of the CNN.

[0039] After step S2, the display control unit 311C generates an information support image F for the endoscopic image acquired in step S1, in which a specific color is applied to the region of the attention area recognized in step S2 in the endoscopic image (step S3). For ease of explanation, the information support image F shown in FIG. 6 shows the region Ar1 of the attention area in the endoscopic image F1 shaded in diagonal lines. The information support image F is then displayed on the second display device 32.

[0040] After step S3, the surgical scene recognition unit 311D recognizes the current surgical scene being performed by the surgeon by performing an estimation process on the endoscopic image acquired in step S1 using the second trained model stored in the second memory unit 312 (step S4).

[0041] Here, the second trained model is a trained model obtained by repeatedly executing a training process on the trained model using multiple sets of training images and training data, each set being a training image and training data. The training images are captured images of the inside of a subject. The training data is data annotated with the type of surgical scene in the training image, which is obtained when the training image is captured. The trained model used in the training process is, for example, a CNN. The second trained model includes a weight file (training parameters) having weight values ​​and bias values ​​of the CNN.

[0042] After step S4, the treatment tool recognition unit 311E recognizes the treatment tool reflected in the endoscopic image acquired in step S1 by performing an estimation process on the endoscopic image (step S5) using the third trained model stored in the second storage unit 312. Examples of the treatment tool include an energy device that applies ultrasonic vibration, high-frequency current, heat, or the like to the biological tissue to exfoliate or separate the biological tissue, a linear stapler that sutures the biological tissue, and grasping forceps that grasp the biological tissue.

[0043] Here, the third trained model is a trained model obtained by repeatedly executing a training process on the trained model using multiple sets of training images and training data, each set consisting of a training image and training data. The training images are captured images of the inside of a subject. The training data is data annotated with the type, correct position, and size of a treatment tool in the training images. The trained model used in the training process is, for example, a CNN. The third trained model includes a weight file (learning parameters) having weight values ​​and bias values ​​for the CNN.

[0044] The neural network used in the learning process for generating the first to third trained models described above is not limited to CNN, and other neural networks may be used. For example, neural networks such as DNN (Deep Neural Network), Transformer, and GAN (Generative Adversarial Network) may be used as appropriate. Furthermore, various well-known learning algorithms may be used as machine learning algorithms in neural networks. For example, a supervised learning algorithm using backpropagation may be used.

[0045] After step S5, region of interest setting unit 311F sets a specific region in the endoscopic image acquired in step S1 as a region of interest (step S6).

[0046] Specifically, the region of interest setting unit 311F uses the first relationship information or the second relationship information stored in the second storage unit 312 in step S6.

[0047] When the region of interest setting mode is set to the first setting mode by an input operation to the second input device 33 by a user such as an operator, the region of interest setting unit 311F uses the first relationship information.

[0048] Here, the first relationship information is information that associates a surgical scene with a region of interest, as shown in FIG. 4. Specifically, the surgical scenes of "tissue dissection" and "tissue cutting" are each associated with a region of interest of a "specific region centered on the tip of the energy device." The surgical scene of "intestinal resection" is associated with a region of interest of a "specific region centered on the tip of the linear stapler." Furthermore, surgical scenes other than "tissue dissection," "tissue cutting," and "intestinal resection" are associated with a region of interest of "the entire screen." In this embodiment, the tip of a treatment tool (e.g., an energy device, a linear stapler, etc.) is, for example, the foremost end point of the treatment side of the treatment tool. Note that the tip of the treatment tool is not limited to this, and a predetermined point at the end of the treatment side of the treatment tool can be set.

[0049] For example, assume that in step S4, "tissue peeling" or "tissue dissection" is recognized as the surgical scene. In this case, in step S6, the region of interest setting unit 311F refers to the first relationship information and identifies a region of interest that is a "specific region centered on the tip of the energy device" corresponding to the surgical scene of "tissue dissection" or "tissue cutting." Then, the region of interest setting unit 311F sets, in the endoscopic image acquired in step S1, a specific region centered on the tip of the energy device, which is the treatment tool recognized in step S5, as a region of interest. For convenience of explanation, in the information support image F shown in FIG. 6, an elliptical region Ar2 centered on the tip P1 of the energy device D1 captured in the endoscopic image F1 is illustrated as the region of interest. Note that the elliptical region Ar2 is not actually displayed in the information support image F.

[0050] Also, for example, assume that in step S4, "intestinal resection" is recognized as the surgical scene. In this case, in step S6, region of interest setting unit 311F refers to the first relationship information and identifies a region of interest of "a specific region centered on the tip of the linear stapler" corresponding to the surgical scene of "intestinal resection." Then, region of interest setting unit 311F sets, as a region of interest, a specific region centered on the tip of the linear stapler, which is the treatment tool recognized in step S5, in the endoscopic image acquired in step S1.

[0051] Furthermore, for example, it is assumed that in step S4, a surgical scene other than "tissue peeling," "tissue dissection," and "intestinal resection" is recognized. In this case, in step S6, the region of interest setting unit 311F refers to the first relationship information and specifies a region of interest of "the entire screen" corresponding to a surgical scene other than "tissue peeling," "tissue dissection," and "intestinal resection." Then, the region of interest setting unit 311F sets the entire endoscopic image acquired in step S1 as a region of interest.

[0052] Furthermore, when the region of interest setting mode is set to the second setting mode by an input operation to the second input device 33 by a user such as an operator, the region of interest setting unit 311F uses the second relationship information.

[0053] Here, the second relationship information is information that associates a treatment tool with a region of interest, as shown in Fig. 5. Specifically, an "energy device" treatment tool is associated with a region of interest of a "specific region centered on the tip of the energy device." Also, a "linear stapler" treatment tool is associated with a region of interest of a "specific region centered on the tip of the linear stapler." Furthermore, treatment tools other than an "energy device" and a "linear stapler" are associated with a region of interest of the "entire screen."

[0054] For example, assume that an "energy device" is recognized as the treatment tool in step S5. In this case, in step S6, the region of interest setting unit 311F refers to the second relationship information and identifies a region of interest that is a "specific region centered on the tip of the energy device" corresponding to the "energy device" treatment tool. Then, the region of interest setting unit 311F sets, in the endoscopic image acquired in step S1, a specific region centered on the tip of the energy device, which is the treatment tool recognized in step S5, as a region of interest. For convenience of explanation, in the information support image F shown in FIG. 6, an elliptical region Ar2 centered on the tip P1 of the energy device D1 captured in the endoscopic image F1 is illustrated as a region of interest. Note that the elliptical region Ar2 is not actually displayed in the information support image F.

[0055] Also, for example, assume that a "linear stapler" is recognized as the treatment tool in step S5. In this case, in step S6, region of interest setting unit 311F refers to the second relationship information and identifies the region of interest of a "specific region centered on the tip of the linear stapler" corresponding to the treatment tool of "linear stapler." Then, region of interest setting unit 311F sets, as a region of interest, the specific region centered on the tip of the linear stapler, which is the treatment tool recognized in step S5, in the endoscopic image acquired in step S1.

[0056] Furthermore, for example, it is assumed that a treatment tool other than an "energy device" or a "linear stapler" is recognized in step S5. In this case, in step S6, region of interest setting unit 311F refers to the second relationship information and identifies a region of interest of "the entire screen" corresponding to a treatment tool other than "energy device" and "linear stapler." Then, region of interest setting unit 311F sets the entire endoscopic image acquired in step S1 as a region of interest.

[0057] In step S5, there may be cases where multiple treatment tools are recognized. In this case, in step S6, region of interest setting unit 311F refers to the second relationship information and identifies the region of interest corresponding to the treatment tool with the highest priority among the multiple recognized treatment tools. The order of priority is, for example, "energy device," "linear stapler," and "other treatment tools."

[0058] After step S6, the distance calculation unit 311G calculates the distance between the actual position of the attention area on the observation object corresponding to the attention area recognized in step S2 and the actual position of the treatment tool on the observation object corresponding to the treatment tool recognized in step S5 (step S7).

[0059] In this embodiment, as described above, the endoscope device 21 is configured with a 3D scope. Therefore, in step S7, the distance calculation unit 311G uses the relative displacement amount in the images of the same subject between the captured images simultaneously captured from different viewpoints in the right-eye and left-eye imaging units of the imaging unit 212. Then, the distance calculation unit 311G calculates the three-dimensional coordinates of the actual site of interest on the observation object (hereinafter referred to as the first three-dimensional coordinates) based on the principle of triangulation. Similarly, the distance calculation unit 311G calculates the three-dimensional coordinates of the actual treatment tool on the observation object (hereinafter referred to as the second three-dimensional coordinates) based on the principle of triangulation. Then, the distance calculation unit 311G calculates, for example, the shortest distance between the position of the actual site of interest on the observation object and the position of the actual treatment tool on the observation object based on the first and second three-dimensional coordinates. The actual position of the treatment tool on the observation object can be the position of the tip of the treatment tool.

[0060] After step S7, the notification control unit 311H determines whether a condition is met (step S8), which is that the distance calculated in step S7 is equal to or less than a specific threshold, and the attention part that is the target of calculating the distance is located outside the region of interest set in step S6.

[0061] If it is determined that the condition is not satisfied (step S8: No), the information support device 31 returns to step S1.

[0062] On the other hand, if it is determined that the condition is met (step S8: Yes), the notification control unit 311H causes the speaker 34 to notify specific information indicating a warning by voice (step S9). After that, the information support device 31 returns to step S1.

[0063] 6, the area Ar1 of the attention part is located outside the region of interest (elliptical area Ar2). When the distance between the actual position of the attention part on the observation object corresponding to the area Ar1 of the attention part and the actual position of the tip P1 of the energy device D1 on the observation object corresponding to the tip P1 of the energy device D1 becomes equal to or less than a specific threshold, specific information indicating a warning is announced by voice from the speaker 34.

[0064] According to the present embodiment described above, the following effects are achieved. In the information support device 31 according to this embodiment, the second processor 311 sets a specific region in an endoscopic image as a region of interest. The second processor 311 also recognizes a region of interest in the subject that appears in the endoscopic image. The second processor 311 also recognizes a treatment tool that appears in the endoscopic image. The second processor 311 also calculates the distance between the region of interest and the treatment tool. If the distance is equal to or less than a specific threshold and the region of interest is located outside the region of interest, the second processor 311 causes the speaker 34 to announce specific information. Therefore, when a region of interest is present within the region of interest, the operator is not warned, and treatment by the operator is not hindered. Also, when a region of interest is present outside the region of interest and the treatment tool approaches the region of interest, the operator is warned, so it is possible to avoid the treatment tool coming into contact with the region of interest that the operator is not paying attention to. Therefore, the information support device 31 according to this embodiment can provide information support that reduces the burden on the surgeon.

[0065] Furthermore, in the information support device 31 according to this embodiment, the second processor 311 sets a region of interest based on a treatment tool that appears in the endoscopic image. Specifically, a case will be described in which the treatment tool that appears in the endoscopic image is a treatment tool (e.g., an energy device or a linear stapler) that is problematic or likely to cause problems if it comes into contact with a region of interest. In this case, the second processor 311 sets a region of interest that is centered on a specific position (the tip in this embodiment) of the treatment tool. This allows the region of interest to be set appropriately. Furthermore, if the treatment tool that appears in the endoscopic image is other than an energy device or a linear stapler, the second processor 311 sets the entire endoscopic image as the region of interest. In other words, for a treatment tool such as grasping forceps that is problematic or unlikely to cause problems if it comes into contact with a region of interest, the entire endoscopic image is set as the region of interest, so that unnecessary warnings are not given to the surgeon and the procedure by the surgeon is not hindered.

[0066] Furthermore, in the information support device 31 according to this embodiment, the second processor 311 recognizes a surgical scene based on an endoscopic image and sets a region of interest based on the surgical scene. Specifically, when the surgical scene is a surgical scene in which tissue is dissected, a surgical scene in which tissue is cut, or a surgical scene in which an intestine is resected, the second processor 311 sets a region of interest centered on a specific position of the energy device or linear stapler (the tip in this embodiment) as the region of interest. This allows the region of interest to be set appropriately. Furthermore, when the surgical scene is not a surgical scene in which tissue is dissected, a surgical scene in which tissue is cut, or a surgical scene in which an intestine is resected, the second processor 311 sets the entire endoscopic image as the region of interest. In other words, when a surgical scene uses a grasping forceps or the like that does not cause any problems even if it comes into contact with the target area, the entire endoscopic image is set as the region of interest, so that unnecessary warnings are not given to the surgeon and the procedure by the surgeon is not hindered.

[0067] (Other embodiments) Although the embodiments for carrying out the present invention have been described above, the present invention should not be limited to only the above-described embodiments. In the above-described embodiment, the information support device according to the present invention is mounted on the endoscope system 1 in which the insertion section 211 is configured by a rigid endoscope, but this is not limiting. For example, the information support device according to the present invention may be mounted on an endoscope system in which the insertion section 211 is configured by a flexible endoscope.

[0068] In the above-described embodiment, the distance calculation unit 311G calculated the distance between the actual position of the area of ​​interest on the observation object and the actual position of the treatment tool on the observation object based on the principle of triangulation, since the endoscopic device 21 is composed of a 3D scope, but this is not limited to this.

[0069] For example, if the endoscopic device 21 is not a 3D scope, the distance calculation unit 311G may calculate the distance between the actual position of the area of ​​interest on the object of observation and the actual position of the treatment tool on the object of observation, as shown below. The distance calculation unit 311G estimates the first and second three-dimensional coordinates by performing estimation processing using a specific trained model on the two-dimensional image generated by the imaging unit 212. Then, the distance calculation unit 311G calculates the distance between the actual position of the attention area on the observation object and the actual position of the treatment tool on the observation object. In addition, when the imaging element constituting the imaging unit 212 of the endoscopic device 21 is composed of an imaging element including phase difference pixels, the distance calculation unit 311G calculates the distance between the actual position of the attention area on the observation object and the actual position of the treatment tool on the observation object based on pixel information corresponding to the phase difference pixels. Furthermore, the distance calculation unit 311G converts the number of pixels between the position of the attention area on the display image on the first display device 23 or the second display device 32 and the position of the treatment tool into a distance.

[0070] In the above-described embodiment, the display control unit 311C displays the information support image F on the second display device 32, but this is not limiting. For example, the display control unit 311C may transmit the information support image F to the observation system 2 via the first and second external interfaces and display the information support image F on the first display device 23.

[0071] Incidentally, when a specific period of time has passed since the power supplied to the energy device was switched from on to off, the residual heat in the treatment portion of the energy device disappears, and there is no problem even if the treatment portion comes into contact with the area of ​​interest. Therefore, in the above-described embodiment, the second processor 311 may switch the region of interest to the entire endoscopic image after a specific period of time has elapsed since the power supplied to the energy device was switched from on to off.

[0072] Incidentally, if the treatment tool captured in the endoscopic image is an energy device and the power supplied to the energy device is turned off, there is no problem even if the energy device comes into contact with the area of ​​interest. Therefore, in the above-described embodiment, if the treatment tool reflected in the endoscopic image is an energy device and the power supplied to the energy device is turned off, the second processor 311 may set the entire endoscopic image as the region of interest.

[0073] In the above-described embodiment, the speaker 34 is employed as the notification unit according to the present invention, but this is not limiting. A notification unit other than the speaker 34 may be employed as long as it can notify specific information using at least one of an image, light, sound, and vibration. For example, the first display device 23 or the second display device 32 can be exemplified as a notification unit that notifies specific information using an image. Furthermore, an LED (Light Emitting Diode) or the like provided adjacent to the second display device 32 or the like can be exemplified as a notification unit that notifies specific information using light. Furthermore, a vibration member provided in the insertion section 211 or the gripping portion of the treatment tool can be exemplified as a notification unit that notifies specific information using vibration.

[0074] In addition, in the above-described embodiment, the following modified examples 1 to 3 may be adopted.

[0075] (Variation 1) 7 is a diagram for explaining a first modified example of the embodiment. Specifically, FIG. 7 corresponds to FIG. In the above-described embodiment, an elliptical region Ar2 is employed as the region of interest according to the present invention, but this is not limiting. The shape of the region of interest according to the present invention is not limited to an ellipse, and other shapes may be used. In this case, it is preferable that the shape of the region of interest be a shape that is rotationally symmetric about a specific angle with the center of the region of interest as the central axis. In the example of FIG. 7, a rectangular region Ar2 is employed as the region of interest according to the present invention. Furthermore, the size of the region of interest according to the present invention in the endoscopic image F1 may be changeable by a user, such as a surgeon, performing an input operation on the second input device 33.

[0076] Even when the configuration of the present modified example 1 described above is adopted, the same effects as those of the above-described embodiment are achieved.

[0077] (Variation 2) 8 to 10 are diagrams illustrating a second modification of the embodiment. Specifically, FIG. 8 corresponds to FIG. 4 and shows first relationship information according to the second modification. FIG. 9 corresponds to FIG. 5 and shows second relationship information according to the second modification. FIG. 10 corresponds to FIG. 6 and shows an example of an information support image F according to the second modification.

[0078] In the above-described embodiment, the first and second relationship information shown in Figures 4 and 5 may be replaced with the first and second relationship information according to this modified example 2 shown in Figures 8 and 9.

[0079] The first relation information according to the present modified example 2 is information in which surgical scenes and regions of interest are associated with each other, similar to the above-described embodiment. Specifically, surgical scenes requiring special attention that may cause problems or are likely to cause problems if a treatment tool comes into contact with a region of interest (for example, surgical scenes of "tissue peeling," "tissue dissection," and "intestinal resection") are associated with regions of interest in the "center of the screen," where the surgeon is expected to pay attention. Furthermore, surgical scenes other than "tissue peeling," "tissue dissection," and "intestinal resection" (surgical scenes in which no problems occur or are unlikely to cause problems even if a treatment tool comes into contact with a region of interest) are associated with regions of interest in the "entire screen."

[0080] For example, assume that in step S4, "tissue peeling," "tissue dissection," or "intestinal resection" is recognized as the surgical scene. In this case, in step S6, the region of interest setting unit 311F refers to the first relationship information and identifies a region of interest in the "center of the screen" corresponding to the surgical scene of "tissue exfoliation," "tissue dissection," or "intestinal resection." Then, the region of interest setting unit 311F sets a specific region centered on the center point of the screen of the endoscopic image acquired in step S1 as the region of interest in the center of the screen. For convenience of explanation, the information support image F shown in FIG. 10 illustrates an elliptical region Ar2 centered on the center point C of the screen of the endoscopic image F1 as the region of interest. Note that the elliptical region Ar2 is not actually displayed in the information support image F.

[0081] Also, for example, it is assumed that in step S4, a surgical scene other than "tissue peeling," "tissue dissection," and "intestinal resection" is recognized. In this case, in step S6, the region of interest setting unit 311F refers to the first relationship information and specifies a region of interest of "the entire screen" corresponding to a surgical scene other than "tissue peeling," "tissue dissection," and "intestinal resection." Then, the region of interest setting unit 311F sets the entire endoscopic image acquired in step S1 as a region of interest.

[0082] Furthermore, the second relationship information according to this modification 2 is information in which treatment tools and regions of interest are associated with each other, similar to the above-described embodiment. Specifically, the treatment tools "energy device" and "linear stapler" are each associated with a region of interest in the "center of the screen" where the surgeon is expected to focus their attention. Furthermore, treatment tools other than the "energy device" and the "linear stapler" are associated with a region of interest in the "entire screen."

[0083] For example, assume that an "energy device" or a "linear stapler" is recognized as the treatment tool in step S5. In this case, in step S6, the region of interest setting unit 311F refers to the second relationship information and identifies a region of interest in the "screen center" corresponding to the treatment tool, "energy device" or "linear stapler." Then, the region of interest setting unit 311F sets a specific region centered on the screen center of the endoscopic image acquired in step S1 as the region of interest. For convenience of explanation, the information support image F shown in FIG. 10 illustrates an elliptical region Ar2 centered on the screen center C of the endoscopic image F1 as the region of interest. Note that the elliptical region Ar2 is not actually displayed in the information support image F.

[0084] Also, for example, it is assumed that a treatment tool other than an "energy device" or a "linear stapler" is recognized in step S5. In this case, in step S6, region of interest setting unit 311F refers to the second relationship information and identifies a region of interest of "the entire screen" corresponding to a treatment tool other than "energy device" and "linear stapler." Then, region of interest setting unit 311F sets the entire endoscopic image acquired in step S1 as a region of interest.

[0085] In step S5, there may be cases where multiple treatment tools are recognized. In this case, in step S6, region of interest setting unit 311F refers to the second relationship information and identifies the region of interest corresponding to the treatment tool with the highest priority among the multiple recognized treatment tools. The order of priority is, for example, "energy device," "linear stapler," and "other treatment tools."

[0086] Even when the configuration of the present modified example 2 described above is adopted, the same effects as those of the above-described embodiment are achieved.

[0087] In the above-described second modification, the "screen center" is used as the region of interest corresponding to the surgical scene of "tissue exfoliation," "tissue resection," or "intestinal resection," and the treatment tool of "energy device" or "linear stapler" in the first and second relationship information, but this is not limitative. The region of interest corresponding to the surgical scene or treatment tool is not limited to the screen center C, and may be a specific region centered on another position on the endoscopic image F1.

[0088] Incidentally, when a specific period of time has passed since the power supplied to the energy device was switched from on to off, the residual heat in the treatment portion of the energy device disappears, and there is no problem even if the treatment portion comes into contact with the area of ​​interest. Therefore, in the above-described second modification, the second processor 311 may switch the region of interest to the entire endoscopic image after a specific period of time has elapsed since the power supplied to the energy device was switched from on to off.

[0089] Incidentally, if the treatment tool captured in the endoscopic image is an energy device and the power supplied to the energy device is turned off, there is no problem even if the energy device comes into contact with the area of ​​interest. Therefore, in the above-mentioned variant example 2, if the treatment tool reflected in the endoscopic image is an energy device and the power supplied to the energy device is off, the second processor 311 may set the entire endoscopic image as the region of interest.

[0090] (Variation 3) 11 and 12 are diagrams illustrating a third modification of the embodiment. Specifically, Fig. 11 is a block diagram showing the configuration of an information support system 3 according to the third modification. Fig. 12 is a diagram corresponding to Fig. 2 and is a block diagram showing the functions of a second processor 311 according to the second modification.

[0091] In the above-described embodiment, the method for setting a region of interest of the present invention is not limited to the setting method described in the above-described embodiment, and it is also possible to set a specific region centered on the position of the surgeon's line of sight on the screen of the second display device 32 (the position at which the surgeon is focusing) as the region of interest, as in this variant example 3.

[0092] Specifically, in the information assistance system 3 according to the third modification, as shown in FIG. 11, a gaze detection device 35 is added to the information assistance system 3 described in the above embodiment.

[0093] The gaze detection device 35 is installed adjacent to the second display device 32, such as below the second display device 32. The gaze detection device 35 includes a small camera or the like that photographs the eyeball from the front direction, and detects in real time the position coordinates of the surgeon's gaze on the screen of the second display device 32 (hereinafter referred to as gaze position coordinates) by tracking the position of the pupil of the eyeball. The gaze detection device 35 then outputs the gaze position coordinates to the information support device 31.

[0094] In step S6, the region of interest setting unit 311F according to the present modification 3 sets a specific region in the endoscopic image centered on the gaze position coordinates output from the gaze detection device 35 as a region of interest. That is, in the present modification 3, the first and second relationship information are not stored in the second storage unit 312. Also, as shown in FIG. 12, the second processor 311 according to the present modification 3 does not have the function of the surgical scene recognition unit 311D compared to the second processor 311 described in the above-mentioned embodiment.

[0095] Even when the configuration of the third modified example described above is adopted, the same effects as those of the above-described embodiment are achieved. [Explanation of symbols]

[0096] 1. Endoscopy system 2. Observation System 3 Information Support System 21 Endoscopic equipment 22 Light source device 23 First display device 24 First Input Device 25 Control device 31 Information support equipment 32 Second display device 33 Secondary Input Device 34 Speaker (announcement section) 35 Eye gaze detection device 211 Insertion part 212 Imaging unit 251 First Processor 252 First Memory Unit 253 First External Interface 311 Second Processor 311A Image acquisition unit 311B Attention part recognition unit 311C Display control unit 311D Surgical Scene Recognition Unit 311E Treatment tool recognition unit 311F Region of interest setting section 311G distance calculation section 311H Notification control unit 312 Second memory unit 313 Second External Interface Ar1,Ar2 area C Screen center point D1 Energy Device F. Information Support Image F1 Endoscopic image P1 tip

Claims

1. An information support device including a processor that processes an endoscopic image captured by an endoscopic device that captures images inside a subject, The processor: setting a specific region in the endoscopic image as a region of interest; Recognizing a region of interest in the subject captured in the endoscopic image; Recognizing a treatment tool captured in the endoscopic image; Calculating the distance between the noted portion and the treatment tool; An information support device that causes a notification unit to notify specific information when the distance is equal to or less than a specific threshold and the attention part is located outside the region of interest.

2. The processor: The information support device according to claim 1 , wherein the region of interest is set based on the treatment tool captured in the endoscopic image.

3. The processor: The information support device according to claim 1 , wherein a specific treatment tool that appears in the endoscopic image is recognized from among a plurality of treatment tools used during surgery.

4. The processor: The information support device according to claim 2, wherein, when the treatment tool captured in the endoscopic image is an energy device, a region having a specific position of the energy device as a center is set as the region of interest.

5. The processor: The information support device according to claim 4 , wherein the region of interest is switched to the entire endoscopic image after a specific period of time has elapsed since the power supplied to the energy device was switched from on to off.

6. The processor: The information support device according to claim 2, wherein when the treatment tool captured in the endoscopic image is an energy device and the power supplied to the energy device is off, the entire endoscopic image is set as the region of interest.

7. The processor:

3. The information support device according to claim 2, wherein when the treatment tool captured in the endoscopic image is a linear stapler, the region of interest is set to a region having a center at a specific position of the linear stapler.

8. The processor: The information support device according to claim 2, wherein when the treatment tool captured in the endoscopic image is other than an energy device or a linear stapler, the entire endoscopic image is set as the region of interest.

9. The processor: The information support device according to claim 2, wherein when the treatment tool captured in the endoscopic image is an energy device or a linear stapler, a region centered on the center of the endoscopic image is set as the region of interest.

10. The processor: Recognizing a surgical scene based on the endoscopic image; The information support device according to claim 1 , wherein the region of interest in the endoscopic image is set based on the surgical scene.

11. The processor: The information support device according to claim 10, wherein when the surgical scene is a surgical scene in which tissue is dissected or cut, the region of interest is set to be an area centered on a specific position of the energy device, which is the treatment tool.

12. The processor: The information support device according to claim 10, wherein when the surgical scene is a surgical scene for resecting an intestinal tract, the region of interest is set to be an area having a center at a specific position of a linear stapler, which is the treatment tool.

13. The processor: The information support device according to claim 10 , wherein when the surgical scene is not a surgical scene in which tissue is dissected or cut, or a surgical scene in which an intestinal tract is resected, the entire endoscopic image is set as the region of interest.

14. The processor: The information support device according to claim 10, wherein when the surgical scene is a surgical scene in which tissue is dissected or cut, or a surgical scene in which an intestine is resected, an area centered on the image center of the endoscopic image is set as the region of interest.

15. The region of interest is 2. The information support device according to claim 1, wherein the shape is rotationally symmetric at a specific angle with the center of the region of interest as the central axis.

16. The processor: The information support device according to claim 1 , wherein the notification unit notifies the specific information by at least one of an image, light, sound, and vibration.

17. The information support device according to claim 1; and an endoscope device that captures images of the inside of the subject.

18. a first display device that displays the endoscopic image; The endoscope system according to claim 17, further comprising: a second display device that displays an image processed by the information support device.

19. An information assistance method executed by an information assistance device, A specific region in an endoscopic image captured by an endoscopic device that captures images of the inside of a subject is set as a region of interest; Recognizing a region of interest in the subject captured in the endoscopic image; Recognizing a treatment tool captured in the endoscopic image; Calculating the distance between the noted portion and the treatment tool; An information assistance method that causes a notification unit to notify specific information when the distance is equal to or less than a specific threshold and the attention part is located outside the region of interest.

20. A non-transitory information storage medium that stores an information assistance program to be executed by an information assistance device, The information assistance program causes the information assistance device to execute the following processes: A specific region in an endoscopic image captured by an endoscopic device that captures images of the inside of a subject is set as a region of interest; Recognizing a region of interest in the subject captured in the endoscopic image; Recognizing a treatment tool captured in the endoscopic image; Calculating the distance between the noted portion and the treatment tool; A non-transitory information storage medium that causes a notification unit to notify specific information when the distance is equal to or less than a specific threshold and the attention portion is located outside the region of interest.

Images