Endoscope auxiliary information generation device, endoscope auxiliary information generation method, endoscope auxiliary information generation program, inference model training method, and endoscope auxiliary system

Abstract

This endoscope auxiliary information generation device comprises: an image acquisition unit that sequentially acquires a plurality of images of the inside of a living body, wherein the images are obtained using an imaging unit of an endoscope; a target area detection unit that detects a target area having a specific feature in the images; a relative position change detection unit that detects a change in the relative position between the imaging unit and an object being imaged by the imaging unit on the basis of the images; and an auxiliary information generation unit that generates auxiliary information for guiding the operation of the endoscope in accordance with a continuous change in the relative position.

Description

Endoscope auxiliary information generating device, endoscope auxiliary information generating method, endoscope auxiliary information generating program, inference model learning method, and endoscope auxiliary system

[0001] The present invention relates to an endoscope auxiliary information generating device, an endoscope auxiliary information generating method, an endoscope auxiliary information generating program, an inference model learning method, and an endoscope auxiliary system.

[0002] Endoscopes have been widely used in the medical and industrial fields for a long time. For example, in the medical field, an operator can view an endoscopic image of the inside of a subject displayed on a display device, identify a lesion, and then perform treatment on the lesion using a treatment tool.

[0003] In recent years, in order to prevent a surgeon from overlooking a lesion, an endoscope system has been used that extracts a region of interest, such as a lesion, from a medical image and notifies the surgeon of information related to the extracted region of interest. For example, Patent Literature 1 proposes an endoscope system that detects a region of interest from a medical image and changes the display mode according to the detected position of the region of interest, thereby preventing the region of interest from being overlooked.

[0004] JP 2022-103441 A

[0005] When the relative position between the imaging unit of the endoscope and the target area to be imaged changes unintentionally due to the patient's breathing, changing of body position, etc., the surgeon is likely to lose sight of the region of interest. However, the conventional endoscope system described in Patent Document 1 and the like does not present the cause of the lost region of interest or a way to recover from the lost region of interest.

[0006] The present invention has been made in consideration of the above circumstances, and aims to provide an endoscopic auxiliary information generating device, an endoscopic auxiliary information generating method, an endoscopic auxiliary information generating program, an inference model learning method, and an endoscopic auxiliary system that can present causes and recovery measures for losing sight of an area of ​​interest such as a lesion.

[0007] In order to solve the above problems, the present invention proposes the following means: An endoscopic auxiliary information generating device according to a first aspect of the present invention includes an image acquisition unit that sequentially acquires a plurality of images of the inside of a living body obtained by an imaging unit of an endoscope, an attention area detection unit that detects an attention area having a specific feature in the images, a relative position change detection unit that detects a change in the relative position between the imaging unit and an object imaged by the imaging unit based on the images, and an auxiliary information generation unit that generates auxiliary information that guides operation of the endoscope in accordance with successive changes in the relative position.

[0008] The endoscopic auxiliary information generating device, endoscopic auxiliary information generating method, endoscopic auxiliary information generating program, inference model learning method, and endoscopic auxiliary system of the present invention can present causes and recovery measures for losing sight of an area of ​​interest such as a lesion.

[0009] 1 is a diagram showing an endoscopic system according to an embodiment. FIG. 1 is a functional block diagram of the endoscopic system. FIG. 2 is a diagram showing an inference model. FIG. 3 is a diagram explaining training data. FIG. 4 is a flowchart showing the operation of the endoscopic system. FIG. 5 is an example of information related to a case and information related to a patient's condition. FIG. 6 is a diagram showing an example of disappearance of a region of interest when a change in relative position is mainly based on the movement of a living organism. FIG. 7 is a diagram showing an example of disappearance of a region of interest when a change in relative position is mainly based on the movement of a living organism. FIG. 8 is a diagram showing an example of disappearance of a region of interest when a change in relative position is mainly based on the movement of an endoscope. FIG. 9 is a diagram showing an example of disappearance of a region of interest when a change in relative position is mainly based on the movement of an endoscope. FIG. 10 is a diagram showing an example of disappearance of a region of interest when a change in relative position is mainly based on the movement of an endoscope. FIG. 11 is a diagram showing an example of disappearance of a region of interest when a change in relative position is mainly based on the movement of an endoscope. FIG. 12 is a diagram showing an example of disappearance of a region of interest when a change in relative position is mainly based on the movement of an endoscope. 1 is a diagram illustrating an example in which information regarding the patient's posture is used as second auxiliary information; FIG. 2 is a control flowchart for detecting the direction of the patient's body; FIG. 3 is a diagram illustrating an example in which information regarding the direction of the tip of the endoscope is used as second auxiliary information; and FIG. 4 is a control flowchart for displaying the history of the direction of the tip.

[0010] An endoscope system 500 according to an embodiment of the present invention will be described with reference to FIGS. 1 to 21. FIG.

[0011] [Endoscope system 500] Fig. 1 is a diagram showing an endoscope system 500. The endoscope system 500 includes an endoscope 100, an image processing processor device 200, a light source device 300, and a display device 400. The image processing processor device 200 and the light source device 300 may be an integrated device (image control device).

[0012] The light source device 300 has a light source 310 such as an LED, and controls the light source to control the amount of illumination light transmitted to the endoscope 100 via the light guide 161. This allows for illuminating and observing the wall surface even in a dark lumen where there is no natural light. Portions not reached by the illumination light appear black in the captured image D.

[0013] The display device 400 is a device that displays images generated by the image processing device 200 and various information related to the endoscope system 500. The display device 400 is, for example, a liquid crystal monitor or a head-mounted display.

[0014] [Endoscope 100] The endoscope 100 is a device for observing and treating the inside of the body of a patient lying on, for example, an operating table T. The endoscope 100 includes an elongated insertion section 110 that is inserted into the patient's body, an operation section 180 that is connected to the base end of the insertion section 110, and a universal cord 190 that extends from the operation section 180.

[0015] The insertion section 110 has a tip section 120, a bending section 130 that can be bent freely, and a long, flexible flexible tube section 140. The tip section 120, the bending section 130, and the flexible tube section 140 are connected in this order from the tip side. The flexible tube section 140 is connected to an operation section 180. Note that the image processing processor 200 does not need to be a single device as shown in the figure, and may be a distributed device whose functions are distributed across multiple devices. Some of the distributed devices may be located in different locations via a network.

[0016] 2 is a functional block diagram of the endoscope system 500. The distal end portion 120 has an imaging unit 150, an illumination unit 160, and a sensor 170.

[0017] The imaging unit 150 has an optical system, an imaging element that converts optical signals into electrical signals, and an AD conversion circuit that converts analog signals output by the imaging element into digital signals. The imaging unit 150 captures an image of a subject and generates an imaging signal. The imaging signal is acquired by the image processing processor device 200 via an imaging signal cable 151.

[0018] The illumination unit 160 irradiates the subject with illumination light transmitted by the light guide 161. The light guide 161 is connected to the light source device 300 by inserting the insertion section 110, the operation section 180, and the universal cord 190. The illumination unit 160 may include a light source such as an LED, an optical element such as a phosphor having a wavelength conversion function, or the like.

[0019] The sensor 170 detects the position of the tip portion 120 and the speed and direction of the tip portion 120. The sensor 170 is, for example, an acceleration sensor, a gyro sensor, a direction sensor, or a combination of these sensors. The output of the sensor 170 is acquired by the image processing processor unit 200 via a signal cable 171.

[0020] The operation unit 180 accepts operations for the endoscope 100. The operation unit 180 has an ankle knob 181 that controls the bending section 130, an air / water supply button 182, a suction button 183, and a release button 184. The ankle knob 181 is a rotary handle that bends the bending section 130. Operations input to the air / water supply button 182, the suction button 183, and the release button 184 are acquired by the image processing processor 200. The release button 184 is a push button that inputs an operation to save an image acquired from the imaging unit 150.

[0021] The universal cord 190 connects the endoscope 100 and the image processor device 200. The universal cord 190 is a cable through which the imaging signal cable 151, the light guide 161, the signal cable 171, etc. are inserted.

[0022] [Image processing processor device 200] As shown in Figure 2, the image processing processor device (endoscopic auxiliary information generation device) 200 includes a control unit 210, an image acquisition unit 220, an image recording unit 230, a region of interest detection unit 240, an insertion position detection unit 250, a relative position change detection unit 260, an auxiliary information generation unit 270, and an image synthesis unit 290.

[0023] The image processor device 200 is a computer capable of executing programs and including a processor such as a CPU, a memory, a recording unit, etc. The functions of the image processor device 200 are realized by the processor executing the programs. At least some of the functions of the image processor device 200 may be realized by a dedicated logic circuit implemented in an ASIC or FPGA.

[0024] The image processing processor device 200 may further include components other than the processor, memory, and recording unit. For example, the image processing processor device 200 may further include an image calculation unit that performs part or all of the image processing and image recognition processing. By including the image calculation unit, the image processing processor device 200 can execute specific image processing and image recognition processing at high speed. The image calculation unit may be a calculation unit provided in a cloud server connected via the Internet.

[0025] The recording unit is a non-volatile recording medium that stores the above-mentioned program and data necessary for executing the program. The recording unit is composed of, for example, a writable non-volatile memory such as a flexible disk, a magneto-optical disk, a ROM, or a flash memory, a portable medium such as a CD-ROM, or a storage device such as a hard disk or SSD built into a computer system. The recording unit may also be a storage device provided in a cloud server connected via the Internet.

[0026] The above program may be provided by a "computer-readable recording medium" such as a flash memory. The program may be transmitted from a computer storing the program to a memory or recording unit via a transmission medium or by transmission waves in the transmission medium. A "transmission medium" that transmits the program is a medium that has the function of transmitting information. Media that have the function of transmitting information include networks (communication networks) such as the Internet and communication lines (communication lines) such as telephone lines. The above program may realize some of the functions described above. Furthermore, the above program may be a difference file (difference program). The above function may be realized by combining a program already recorded on the computer with a difference program.

[0027] At least a part of the image processing processor device 200 may be a device separate from the image processing processor device 200. The separate device may be a computing device provided in a cloud server connected via the Internet.

[0028] The control unit 210 controls the entire image processing device 200. The control unit 210 also acquires information about the case in which the endoscopic system 500 is used (type of endoscope 100, patient information) from an in-hospital system or the like. The control unit 210 may also acquire the information about the case by having the surgeon or an assistant input the information from an input device (not shown).

[0029] The image acquisition unit 220 acquires an imaging signal from the imaging unit 150 of the endoscope 100 via an imaging signal cable 151. The image acquisition unit 220 performs imaging signal processing on the imaging signal acquired from the imaging unit 150 to sequentially acquire captured images D. The image acquisition unit 220 outputs the acquired captured images D to the image synthesis unit 290. The image acquisition unit 220 also outputs the acquired captured images D via the image recording unit 230 to the attention area detection unit 240, the insertion position detection unit 250, the relative position change detection unit 260, and the auxiliary information generation unit 270.

[0030] The image recording unit 230 is a part of the recording unit described above and is a non-volatile recording medium. The image recording unit 230 is a part of the memory described above and may be a volatile recording medium. The image recording unit 230 records the multiple captured images D that have been transferred.

[0031] The image recording unit 230 records a plurality of captured images D (image frames, time-series images) input in chronological order. When the recording capacity of the image recording unit 230 is insufficient, the oldest captured image D is deleted. The plurality of captured images D recorded in the image recording unit 230 may be captured images D of consecutive frames, or may be captured images D in which a plurality of frames have been thinned out from consecutive frames.

[0032] The region-of-interest detection unit 240 detects a region of interest (region of concern) from the captured image D. The region of interest may be an abnormal region such as a lesion (first region of interest), or the progression path of the insertion unit 110 of the endoscope 100 (second region of interest). The region-of-interest detection unit 240 detects the region of interest (region of concern) based on the captured image D.

[0033] The attention region detection unit 240 may identify an attention region (region of interest) included in the captured image D, for example, by image feature determination, pattern matching, etc. For example, the insertion position detection unit 250 compares the captured image D with a pre-recorded image of a lesion such as a polyp, and identifies an abnormal region (first attention region) included in the captured image D based on the similarity with the image features or pattern of the pre-recorded lesion.

[0034] The region-of-interest detection unit 240 detects an abnormal region (first region of interest) such as a lesion from the captured image D, for example, by using a machine learning model for lesion detection generated by machine learning using the learning captured image D. Furthermore, the region-of-interest detection unit 240 detects a path in the direction of progression within the lumen (progression path, second region of interest) from the captured image D, for example, by using a machine learning model for progression path detection generated by machine learning using the learning captured image D.

[0035] The attention area detection unit 240 transmits the detected attention area (area of ​​interest) to the auxiliary information generation unit 270 .

[0036] The insertion position detection unit 250 detects the insertion position within the lumen of the tip portion 120 from which the captured image D is being acquired. When the insertion unit 110 of the endoscope 100 is inserted into the large intestine, the insertion position detection unit 250 identifies the insertion position of the captured image D based on "divided structures (regions) within the large intestine" such as the cecum, ascending colon, transverse colon, descending colon, sigmoid colon, and rectosigmoid portion. When the insertion unit 110 of the endoscope 100 is inserted into the stomach, the insertion position detection unit 250 identifies the insertion position of the captured image D based on "divided structures (regions) within the stomach" such as the pharynx, esophagus, and inside of the stomach.

[0037] The insertion position detection unit 250 may (B1) detect the insertion position of the captured image D based on the captured image D, or (B2) detect the insertion position of the captured image D based on the output of the sensor 170. The detection methods (B1) to (B2) will be described below.

[0038] (B1) The insertion position detection unit 250 may identify the structure of a lumen contained in the captured image D by image feature determination, pattern matching, or the like. For example, the insertion position detection unit 250 compares pre-recorded images of each part with the captured image D and identifies the structure of a lumen contained in the captured image D based on the similarity to each pre-recorded part. Even a lumen that appears to be a series of similar structures may have different functions at different positions, and the characteristics may differ depending on the position. Deformations may appear due to the influence of adjacent organs or tissues, and the structure (part) may be identified from the characteristics. The insertion position detection unit 250 may identify the structure of a lumen by detecting such characteristics.

[0039] (B1) The insertion position detection unit 250 may infer and identify the structure (region) of the lumen included in the captured image D using an inference model. For example, the inference model is obtained by machine learning using pre-recorded images of each region and annotations of the regions included in the images as training data. In this method, the above-described intraluminal features are also used for inference.

[0040] (B2) The insertion position detection unit 250 may determine the insertion position from (the sum of) information obtained from the output of the sensor 170 (such as the acceleration, speed, direction, and posture of the distal end portion 120). Similarly, the insertion position detection unit 250 may determine the progress in the lumen based on the change over time in the captured image D, and may detect the insertion position by summing up the change over time.

[0041] The insertion position detection unit 250 may identify the insertion position by combining the detection methods (B1) and (B2).

[0042] The insertion position detection unit 250 transmits the insertion position of the distal end portion 120 in the lumen (the divided structure in the lumen) from which the captured image D is being acquired to the auxiliary information generation unit 270 .

[0043] The relative position change detection unit 260 detects a change in the relative position between the tip 120 of the endoscope 100 and the target part of the imaging target. The relative position change detection unit 260 can determine whether the change in relative position is mainly due to the movement of the endoscope 100, whether the change in relative position is mainly due to the movement of the living body, or whether the change in relative position is due to both movements.

[0044] The relative position change detection unit 260 may (C1) detect a change in relative position based on the captured image D, or (C2) detect a change in relative position based on the output of the sensor 170. The detection methods (C1) and (C2) will be described below.

[0045] (C1) The relative position change detection unit 260 may detect a change in the relative position by image recognition of the captured image D. Specifically, when the endoscope 100 (tip portion 120) itself is moving, image features such as the unevenness of the intraluminal tissue and blood vessels detected by the imaging unit 150 change in a generally regular direction (radially spreading from approximately the center of the screen to the outside of the screen or in the opposite direction during insertion or removal, and in the bending direction during bending), or features such as objects that were previously invisible from the other side of the moving lumen (the shadow of the lumen) becoming visible. Therefore, the relative position change detection unit 260 primarily determines that the endoscope 100 has moved when it determines that the image features of the intraluminal tissue have changed in a generally regular direction between image frames obtained in chronological order. The generally regular direction may be the same direction on the screen, up, down, left, or right, or may be the same radial direction accompanying insertion or removal. Note that a combination of these may occur. This image movement does not need to be assessed for all pixels or all image portions; assessment of representative pixels or image portions is sufficient. On the other hand, if the endoscope 100 (tip portion 120) itself remains nearly stationary but the luminal tissue moves, comparing the image frames acquired in chronological order will result in the overall image pattern, including the acquired image features, not moving in a regular direction. Therefore, if it is determined that the image features of the luminal tissue do not change in a regular direction, it is determined that the endoscope (tip portion) itself remains nearly stationary and that the luminal tissue has primarily moved. Furthermore, if the area beyond the lumen contracts when viewed in the direction of the lumen, the image features change in a way that tissue in a specific direction (a color in a specific direction) appears to be overlaying it. If a polyp or other object changes position vertically, and no deformation of the area where the polyp formed is confirmed, it can be determined that the imaging unit 150 at the tip of the endoscope is moving vertically. In such a detection method, the relative position change detection unit 260 may determine changes in the obtained images for each image frame obtained in chronological order, determine whether the endoscope is moving forward or backward along the lumen, determine the speed from the change and convert it into a position, or determine the position from image information inside the lumen.Furthermore, if the relative position change detection unit 260 similarly determines the orientation of the imaging unit 150 of the endoscope 100 from image information inside the lumen, it becomes possible to determine which position in which organ the endoscope tip (or the imaging unit 150) is facing.Furthermore, when something that was contracting expands, the image pattern changes such that tissue in a specific direction (a color in a specific direction) retreats from the periphery toward the periphery, and something (for example, the shadow of a hole in a tube) appears from beyond. When a polyp or other object changes position up or down, if deformation of the area where the polyp formed is simultaneously confirmed, it can be determined that the object itself is moving, rather than that the imaging unit at the tip of the endoscope is shaking. Furthermore, when both the endoscope 100 (tip portion 120) and the luminal tissue move relative to each other, these characteristics occur in combination, and the relative position change detection unit 260 simply needs to identify and detect these characteristics.

[0046] (C1) The relative position change detection unit 260 may infer a change in relative position using an inference model. For example, the inference model is obtained by machine learning using the results of annotation of the relative position changes for multiple image frames as training data.

[0047] (C2) The relative position change detection unit 260 may detect a change in relative position based on the output of the sensor 170 (such as the acceleration, speed, direction, and posture of the tip unit 120). When acceleration occurs in the tip unit 120, the relative position change detection unit 260 can calculate a change in speed or position using time information. The relative position change detection unit 260 can determine that a change in posture or direction is a change in the direction in which the tip unit 120 is facing.

[0048] The relative position change detection unit 260 may detect a change in the relative position between the tip portion 120 of the endoscope 100 and the target part to be imaged by combining the detection methods (C1) and (C2).

[0049] The relative position change detection unit 260 transmits the detected change in relative position to the auxiliary information generation unit 270 .

[0050] The auxiliary information generation unit 270 generates auxiliary information that guides the operation of the endoscope 100 in accordance with successive changes in the relative position between the tip 120 of the endoscope 100 and the target area to be imaged. Specifically, the auxiliary information generation unit 270 generates auxiliary information that guides the operation of the endoscope 100 based on the detection results of the attention area detection unit 240, the insertion position detection unit 250, and the relative position change detection unit 260. The auxiliary information generation unit 270 may generate auxiliary information on a rule basis, or may generate auxiliary information using an inference model 281 included in the inference unit 280.

[0051] Here, the "auxiliary information" includes the cause of the disappearance of the region of interest and a recovery measure for recovering from the disappearance in the multiple captured images D (image frames, time-series images) input in chronological order to the image recording unit 230. Note that the auxiliary information only needs to include at least one of the cause of the disappearance of the region of interest and the recovery measure for recovering from the disappearance.

[0052] FIG. 3 is a diagram showing an inference model 281. The inference model 281 is a model that is trained by identifying image frames before and after the disappearance of a region of interest from multiple image frames (learning images) of multiple cases, and using training data including annotations regarding the cause of the disappearance and recovery measures for image frames after the disappearance of the region of interest. The inference model 281 is, for example, a neural network, and is trained by deep learning. Note that the inference model 281 is not limited to a neural network, and may be another machine learning model that can output information for an input image.

[0053] The input of the inference model 281 is a plurality of captured images D (image frames, time-series images) input in chronological order. The input of the inference model 281 may include the output of the sensor 170. The output of the inference model 281 is auxiliary information.

[0054] FIG. 4 is a diagram illustrating training data. A plurality of image frames (a sequence of still images) obtained during endoscopic examinations of a plurality of cases are used as training data. The training data is a combination of a plurality of image frames (learning images) and annotations that identify image frames before and after a region of interest disappears from the plurality of image frames and that provide information about the cause of the disappearance and a remedial measure for the image frame after the region of interest disappears. The inference model 281 is a model trained using the training data to output a corresponding annotation for an input image frame (learning image).

[0055] The training data preferably includes training data in which the region of interest disappears due to movement of the object, training data in which the region of interest disappears due to movement of the endoscope 100, and training data in which the region of interest disappears due to changes in the object, such as water supply (irrigation) or air supply, as shown in FIG. 4 . An inference model 281 trained using such training data (image changes annotated with causes and recovery measures) can infer various causes and recovery measures for the disappearance of the region of interest. To create such training data, images may be recorded in the image recording unit 230 of FIG. 2 or in a recording unit (not shown). Information (semantics) such as the characteristics of the image and the annotations used for the image may also be recorded in a corresponding database. Such image semantics may correspond to medical knowledge and may be referred to as a knowledge database. The knowledge database may also include information such as the image characteristics of an image at a particular position in the lumen of a particular organ. 2, an inference model learning unit that uses this information to create an inference model 281 may be considered an important element of this embodiment. That is, this embodiment is characterized by providing a learning method for an inference model that detects a region of interest having a specific feature in an image region from consecutive images acquired by the image acquisition unit 220 of the endoscope 100 that observes the inside of a living body, identifies image frames before and after the disappearance of the specific feature from the images acquired by the image acquisition unit 220, annotates the frame after the disappearance with the cause of the disappearance and a recovery measure to generate training data, and, when an endoscopic image in which the region of interest having the specific feature has disappeared is input, infers and outputs the cause of the disappearance and a recovery measure.

[0056] The image synthesis unit 290 generates a synthetic image S including the captured image D and the auxiliary information E generated by the auxiliary information generation unit 270 (see FIG. 10 ). The image synthesis unit 290 sequentially creates synthetic images S corresponding to the captured images D sequentially generated by the image acquisition unit 220, and outputs the images to the display device 400. The display device 400 sequentially displays the received synthetic images S.

[0057] [Operation of Endoscopic System 500] Next, the operation of the endoscope system 500 (auxiliary information generation method) will be described. Specifically, the procedure for observing and treating the luminal wall of the large intestine using the endoscope system 500 will be described. The following description will be given following the flowchart showing the operation of the endoscope system 500 shown in FIG.

[0058] <Step S110> In step S110, the control unit 210 acquires information about the case in which the endoscopic system 500 is used (type of endoscope 100, patient information) from an in-hospital system, etc. The control unit 210 may acquire the information about the case by having the surgeon or an assistant input the information from an input device (not shown).

[0059] <Step S120> In step S120, the control unit 210 acquires information about the patient's condition (including posture information). The control unit 210 may acquire the information about the patient's condition by having the surgeon or an assistant input it from an input device (not shown). FIG. 6 shows an example of information about the case and information about the patient's condition. The information about the patient's condition is re-input each time the patient's condition is updated. The "left lateral position" illustrated in FIG. 6 is a position that makes it less likely for saliva to be aspirated compared to lying on one's back, and is characterized by the fact that water collects in the aorta of the stomach, making reflux from the stomach less likely to occur.

[0060] <Step S130> In step S130, the relative position change detection unit 260 detects whether there has been a change in the relative position between the tip portion 120 of the endoscope 100 and the target portion of the imaging target. If there has been a change in the relative position, the relative position change detection unit 260 determines whether the change in relative position is primarily due to the movement of the endoscope 100 or whether the change in relative position is primarily due to the movement of the living body. If the change in relative position is primarily due to the movement of the endoscope 100, the endoscopic system 500 then executes step S200. If the change in relative position is primarily due to the movement of the living body, the endoscopic system 500 then executes step S140.

[0061] In addition, if the change in relative position is based on both the movement of the endoscope and the movement of the living body, the endoscopic system 500 may perform both steps from step S140 onwards and steps from step S200 onwards.

[0062] <Step S140> In step S140, the region-of-interest detection unit 240 detects a region of interest (first region of interest) such as a lesion area, etc. The endoscope system 500 then executes step S150.

[0063] <Step S150> In step S150, the attention area detection unit 240 determines whether the attention area has disappeared. If the attention area has disappeared, the endoscope system 500 then executes step S160. If the attention area has not disappeared, the endoscope system 500 then executes step S290.

[0064] <Step S160> In step S160, the auxiliary information generator 270 determines whether auxiliary information (cause, recovery measures) for the lost region of interest can be inferred using the inference model 281. If the auxiliary information can be inferred, the endoscope system 500 then executes step S170. If the auxiliary information cannot be inferred, the endoscope system 500 then executes step S180.

[0065] 7 to 10 are diagrams showing an example of disappearance of a region of interest when a change in relative position is mainly due to movement of a living organism. The imaging unit 150 provided at the tip 120 of the endoscope 100 shown in FIG. 7 is positioned at a position where it can image a polyp P in the large intestine C. FIG. 8 shows a composite image S including an image D captured by the imaging unit 150 shown in FIG. 7. The polyp P can be observed in the image D shown in FIG. 8. As shown in FIG. 9, contraction (non-rigid body movement) of the large intestine C, which is a living organism, makes it impossible for the imaging unit 150 to observe the polyp P that it had been observing in the large intestine C. In this case, as shown in FIG. 10, the polyp P (region of interest) disappears in the image D. The auxiliary information generation unit 270 infers auxiliary information E as exemplified in FIG. 10 using an inference model 281.

[0066] <Step S170> In step S170, the image synthesis unit 290 generates a synthetic image S including the captured image D and the auxiliary information E generated by the auxiliary information generation unit 270, as shown in Fig. 10 , and outputs the synthetic image S to the display device 400. The display device 400 displays the received synthetic image S. The endoscope system 500 then executes step S290.

[0067] <Step S180> In step S180, the auxiliary information generator 270 generates and displays second auxiliary information. Details of the second auxiliary information generated in step S180 will be described later. The endoscope system 500 then executes step S290.

[0068] <Step S200> In step S200, the attention region detection unit 240 detects a path in the traveling direction within the lumen (travel path, second attention region). Note that in step S200, the attention region detection unit 240 may detect an attention region (first attention region) such as a lesion. The endoscope system 500 then executes step S210.

[0069] <Step S210> In step S210, the attention area detection unit 240 determines whether the attention area has disappeared. If the attention area has disappeared, the endoscope system 500 then executes step S220. If the attention area has not disappeared, the endoscope system 500 then executes step S290.

[0070] <Step S220> In step S220, the auxiliary information generator 270 determines whether auxiliary information (cause, recovery measures) for the lost region of interest can be inferred using the inference model 281. If the auxiliary information can be inferred, the endoscope system 500 then executes step S230. If the auxiliary information cannot be inferred, the endoscope system 500 then executes step S240.

[0071] 11 to 14 are diagrams illustrating an example of disappearance of the region of interest when the change in relative position is primarily due to the movement of the endoscope 100. As shown in FIG. 11 , the shape of a lumen, such as the large intestine C, is complex and varies from person to person, making it difficult for the surgeon to determine the path of travel (the path of travel). Furthermore, it is difficult to observe the up, down, left, and right directions of the captured image D in association with the up, down, left, and right directions of the lumen in the direction of travel. As shown in FIG. 11 , before the tip 120 of the insertion section 110 reaches a bent portion of the large intestine C, the path R of travel is easily observed, as in the captured image D shown in FIG. 12 . However, as shown in FIG. 13 , when the tip 120 of the insertion section 110 reaches a bent portion of the large intestine C, the path R of travel (the region of interest) disappears, as in the captured image D shown in FIG. 14 , due to reasons such as the tip 120 approaching the lumen wall. The auxiliary information generator 270 infers auxiliary information E, as illustrated in FIG. 14 , using the inference model 281.

[0072] 14 , in step S230, the image synthesis unit 290 generates a synthetic image S including the captured image D and the auxiliary information E generated by the auxiliary information generation unit 270, and outputs the generated synthetic image S to the display device 400. The display device 400 displays the received synthetic image S. The endoscope system 500 then executes step S290.

[0073] <Step S240> In step S240, the auxiliary information generator 270 generates and displays second auxiliary information. Details of the second auxiliary information generated in step S180 will be described later. The endoscope system 500 then executes step S290.

[0074] <Step S290> In step S290, the control unit 210 determines whether the procedure has ended. If the control unit 210 determines that the procedure has not ended, it executes step S120 and subsequent steps. If the control unit 210 determines that the procedure has ended, it executes step S300 and ends the control flow shown in FIG. 5.

[0075] Thus, if there is a region of interest detection function that detects a region of interest having a specific characteristic in an intraluminal image during intraluminal examination, changes in the image characteristics can be detected even when the endoscope 100 is inserted or bent. This change in image characteristics represents a change in the relative position between the image capture unit 150 and the object captured by the image capture unit 150. Specifically, by detecting a region of interest, the current situation (the position of the endoscope tip or the state of the lumen in the direction of travel) can be determined based on the image portion having the specific characteristic, or the operation of the endoscope 100 or the movement of the lumen can be determined based on successive changes in image characteristics, etc., based on the image portion having the specific characteristic. Therefore, by providing a relative position change detection function that detects this, endoscopic auxiliary information can be generated in response to successive changes in relative position. For example, if the center of the captured image D is dark and the insertion section 110 can be inserted in a straight line, and the bending section 130 is attempted to bend, auxiliary information (e.g., "Go straight, OK") can be generated to guide the user that the bending section 130 does not need to bend. Furthermore, when traveling straight would result in hitting the inner wall of the lumen, auxiliary information (such as "Don't go straight") can be generated to guide the user to prevent a collision by traveling straight. Furthermore, if an image feature of a hole in the lumen is detected on the right side of the screen, auxiliary information can be generated to guide the user to turn the bending section 130 to the right. Such information is auxiliary information that assists the user in determining the path of travel within the living body. The auxiliary information may utilize the inference results of the inference model 281 to which the captured image D is input.

[0076] [Generation of Second Auxiliary Information] The auxiliary information generation unit 270 generates second auxiliary information based on the rule base in steps S180 and S240, even if auxiliary information cannot be inferred using the inference model 281. Three types of second auxiliary information E2 will be described below.

[0077] <(1) Second Auxiliary Information: Use of Image Captured Before the Region of Attention Disappears> Figures 15 and 16 are diagrams showing an example in which image D captured before the region of attention disappears is used as second auxiliary information E2. The second auxiliary information E2 shown in Figure 15 displays image D captured before the region of attention disappeared as an image, and suggests to the surgeon that the surgeon return the imaging unit 150 of the endoscope 100 to the position where that image was captured as a recovery measure. The second auxiliary information E2 shown in Figure 16 further suggests the direction and distance to move the imaging unit 150 of the endoscope 100 as a recovery measure.

[0078] <(2) Second Auxiliary Information: Use of Information Regarding Patient Posture> Figures 17 and 18 are diagrams illustrating an example in which information regarding the patient's posture is used as the second auxiliary information E2. As is clear from Figure 17, the relative positions of organs and the direction of travel of the lumen of each organ (hereinafter also referred to as the "direction of travel") are determined to some extent by the anatomical characteristics of the human body. The direction of travel may be affected by gravity and may sag, but the effect of sagging may be corrected. Furthermore, when the endoscope tip is facing the direction of travel of the lumen, a black circular image feature representing the lumen hole is detected in the center of the image. The patient's posture allows us to determine the relationship between the direction of gravity (and the horizontal direction) and the direction of travel of the lumen at each site obtained according to the anatomical organ positioning characteristics. If the lumen hole image feature is located in the center of the captured image D obtained by the imaging unit 150, it can be determined that the endoscope tip is facing the direction of travel of the lumen. If the direction of gravity in the captured image D is already known and an acceleration sensor or the like is built into the distal end portion 120, this information can be supplemented to determine whether the distal end portion 120 is looking up or down in the direction of gravity. Such anatomical features (information regarding the positional relationships of organs in the human body and changes in orientation corresponding to the organ positions in the lumen direction) may be recorded in the recording unit 230 in Fig. 2 or in another recording unit not shown, and may be made available for reference as a knowledge database (a database in which medical knowledge is digitized, recorded, and made available for reference).

[0079] The direction in which the bending section 130 is bent when the insertion section 110 of the endoscope 100 is passed through a structure (part) of a lumen (for example, the large intestine) varies depending on the patient's posture. For example, when the patient is in the "left lateral position" as shown in Fig. 17 and the distal end section 120 is inserted into the rectum so that the up and down of the distal end section 120 (the up and down of the captured image D obtained from the imaging section 150) coincides with the up and down of the direction of gravity, the left side of the captured image D is the back side of the patient based on the traveling direction of the anatomical lumen.

[0080] When inserting the endoscope 100, auxiliary information that aids the surgeon in determining the path of travel can be helpful. The auxiliary information is generated based on information identifying the lumen (e.g., the large intestine), the patient's body position at the time of insertion (the direction in which the rectum faces relative to the direction of gravity) based on the vertical direction of the image capture screen D during endoscope insertion, and anatomical information regarding the direction of lumen bending that occurs during insertion of the specific lumen. For example, if the rectum is horizontal, then unless the endoscope 100 is twisted, the bending section 130 must be bent to the left (anatomically toward the back, but the relationship between up and down changes depending on the body position) and then bent downward to pass through the sigmoid colon.

[0081] On the other hand, when the patient is in the "right lateral position" and the tip 120 is inserted into the rectum so that the top and bottom of the tip 120 are aligned with the top and bottom of the direction of gravity, the right side of the captured image D is the back side of the patient based on the direction of progression of the anatomical lumen.

[0082] As the distal end portion 120 passes through the rectum and the sigmoid colon and is further inserted, the direction of advancement of the lumen changes due to the anatomical characteristics of the rectum and sigmoid colon, causing the insertion direction of the distal end of the endoscope to change. For example, if the rectum is aligned horizontally, then unless the endoscope 100 is twisted, the curved portion 130 must be bent to the right (anatomically toward the back, but the relationship between up and down changes depending on the body position) and then bent upward to pass through the sigmoid colon.

[0083] In this way, when displaying auxiliary information to assist in the path of travel of the endoscope 100 when the endoscope is inserted, the auxiliary information is generated based on information identifying which organ the lumen is, and anatomical information on the direction of lumen curvature that occurs with the insertion of the specific lumen, based on the up-down direction of the imaging screen D when the endoscope is inserted and the examination position of the subject.

[0084] Note that if the tip portion 120 is inserted into the lumen so that the up and down of the tip portion 120 (the up and down of the image capture screen D) coincides with the up and down of the direction of gravity, and the endoscope 100 simply moves straight without twisting, the up and down direction will not change. However, it should be noted that when the bending portion 130 bends, the relationship between the up and down of the image capture screen D and the up and down of the direction of gravity may change. However, if the direction in which the bending portion 130 bends is known in advance, the relationship between the up and down of the image capture screen D and the up and down of the direction of gravity after the bending portion 130 is bent can be determined. Furthermore, if there is twisting during insertion, this characteristic will appear in the image capture screen D during the twisting process, so auxiliary information corrected to take the twist into account may be generated. However, before bending the bending portion 130 and advancing the tip of the endoscope into a lumen extending in a different direction, it is not easy to bend and insert the tip of the endoscope toward the sigmoid colon via the rectum. The surgeon must understand the direction in which to bend the bending portion 130, taking into account the patient's posture. Therefore, in this embodiment, the surgeon is assisted by graphical guides such as arrows. As shown in Fig. 18 , it is desirable that the second auxiliary information E2 include the orientation of the patient's body. If the orientation of the patient's body in the captured image D is known, the surgeon can more easily grasp the general direction in which to bend the bending section 130 based on information about the lumen bending direction, which relies on the up-down relationship of the screen during insertion, or based on information about anatomical changes in the lumen direction and the direction of gravity. The second auxiliary information E2 illustrated in Fig. 18 presents the orientation of the patient's back in the captured image D using the direction of an arrow. Note that even if information about the patient's body orientation is not available, it is possible to automatically determine the orientation of the patient's body based on the characteristics of the captured image D during insertion, etc.

[0085] The reason why the surgeon needs to understand left and right in relation to the up and down of the captured image D and the up and down of the direction of gravity is because the operation unit 180 has an operation method of bending the tip of the endoscope up, down, left, and right. The essence of generating auxiliary information is to generate auxiliary information that allows the surgeon to correctly operate the operation unit 180 by looking at the up, down, left, and right arrow displays. Therefore, the auxiliary information may be an operation instruction that instructs the operation unit 180 on what operation to perform, in addition to an indication of up, down, left, and right. The operation instruction may be audio or vibration.

[0086] The direction in which a lumen bends changes as the region advances deeper into the lumen, for example, from the rectum to the sigmoid colon, or from the sigmoid colon to the descending colon, but is largely determined anatomically. Therefore, it is necessary to determine the position of the distal end portion 120 of the endoscope 100 within the lumen of a specific organ, and a situation in which the distal end portion 120 cannot advance deeper into the lumen unless it bends at that point. These can be determined from changes in the images and image features by sequentially acquiring multiple captured images D of the inside of the living body obtained by the imaging unit 150 of the endoscope 100.

[0087] For example, in areas where the tip of the endoscope only needs to move straight, the center of the screen appears dark because the light from the light source irradiating the light is not reflected back by the imaging unit 150. Because a lumen is generally cylindrical, a round, dark image is obtained. However, where the lumen bends, the front appears to be a wall. Thus, an attention area detection unit 240 is needed to detect an attention area with specific characteristics in an endoscopic image. This attention area in the image changes as the endoscope 100 is inserted or bent, and the image characteristics also change. This change in image characteristics (attention area change) represents a change in the relative position between the imaging unit 150 and the object imaged by the imaging unit 150. The reason for using an attention area is that the determination is based on an image portion with specific characteristics. Therefore, by providing a relative position change detection function to detect this, endoscopic auxiliary information can be generated in response to continuous changes in relative position. For example, if the center of the captured image D is dark and the insertion operation of the insertion section 100 can proceed straight, when an attempt is made to bend the bending section 130, auxiliary information (such as "Go straight") can be generated to guide the user not to bend the bending section 130. Furthermore, when proceeding straight would result in hitting the inner wall of the lumen, auxiliary information (such as "Don't proceed") can be generated to guide the user to prevent a collision by proceeding straight. Furthermore, if an image feature of a lumen hole is detected on the right side of the screen, auxiliary information can be generated to guide the user to bend the bending section 130 to the right. The auxiliary information may utilize the inference results of the inference model 281 to which the captured image D is input. Even if such image features that provide hints are not available, the auxiliary information generator can generate auxiliary information by combining them with information on the lumen bending direction based on the anatomical knowledge already described.

[0088] In this embodiment, the auxiliary information is information about a recovery measure for losing sight of the region of interest (for example, displaying the direction of travel of the lumen), but the auxiliary information is not limited to this. The auxiliary information may also be information that causes the region of interest to be lost. For example, depending on the position within the lumen, it is also possible to assist the surgeon with auxiliary information such as "The lumen is bent, so please look for a hole in the lumen."

[0089] 19 is a control flowchart for detecting the patient's body orientation. In step S410, the control unit 210 determines the patient's body posture based on input information about the patient's condition. In step S420, the auxiliary information generation unit 270 tentatively determines the patient's body orientation in the captured image D according to the patient's body posture. In step S430, the auxiliary information generation unit 270 determines whether the distal end portion 120 of the endoscope 100 has moved based on the captured image D or the output of the sensor 170. If the distal end portion 120 has moved, in step S440, the insertion position detection unit 250 detects the insertion position into which the distal end portion 120 of the endoscope 100 has been inserted. In step S450, the auxiliary information generation unit 270 determines the patient's body orientation in the captured image D according to the detected insertion position (site) of the distal end portion 120, taking into account the patient's body posture.

[0090] <(3) Second Auxiliary Information: Use of Information Regarding the Orientation of the Tip 120> Figure 20 is a diagram illustrating an example in which information regarding the orientation of the tip 120 is used as the second auxiliary information E2. Incorrect operation of the endoscope 100 may cause the tip 120 to point in an unintended direction, resulting in the disappearance of the region of interest from the captured image D. In this case, if there is a history of the orientation of the tip 120, the surgeon can easily return the orientation of the tip 120 to the orientation of the tip 120 before the region of interest disappeared. Therefore, as shown in Figure 20, it is desirable that the second auxiliary information E2 include a history of the orientation of the tip 120. The history of the orientation of the tip 120 shown in Figure 20 is a history of the orientation of the tip 120 in the up-down direction. The second auxiliary information E2 may also include a history of the orientation of the tip 120 in the left-right direction.

[0091] 21 is a control flowchart for displaying the history of the orientation of the tip portion 120. In step S510, the auxiliary information generation unit 270 records the relationship between the up-and-down direction of the captured image D and the output (gravitational acceleration) of the sensor 170. In step S520, the auxiliary information generation unit 270 determines whether the tip portion 120 of the endoscope 100 has moved based on the captured image D or the output of the sensor 170. In step S530, the auxiliary information generation unit 270 records the history of the orientation of the tip portion 120. In step S550, the image synthesis unit 290 generates a composite image S including the captured image D and the second auxiliary information E2 generated by the auxiliary information generation unit 270, and outputs the composite image S to the display device 400. The display device 400 displays the received composite image S.

[0092] In addition, even if the auxiliary information can be inferred using the inference model 281, the auxiliary information generation unit 270 may generate second auxiliary information and add it to the auxiliary information inferred using the inference model 281.

[0093] The endoscopic system 500 according to this embodiment can present auxiliary information (causes and recovery measures) for when a region of interest, such as a lesion (first region of interest) or a progression path (second region of interest), is lost. The endoscopic system 500 detects a change in the relative position between the distal end 120 of the endoscope 100 and the target region of the imaging target, and generates auxiliary information for when a region of interest is lost using different strategies depending on whether the change in relative position is primarily due to the movement of the endoscope 100 or whether the change in relative position is primarily due to the movement of the living body. The endoscopic system 500 can generate second auxiliary information even when auxiliary information cannot be inferred using the inference model 281.

[0094] Although one embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and design modifications and the like are also included within the scope of the present invention. Furthermore, the components shown in the above embodiment and the following modified examples can be appropriately combined to form a configuration.

[0095] In the above embodiment, the endoscopic auxiliary data generation device performs diagnostic support on images from a medical endoscope. However, the diagnostic target of the endoscopic auxiliary data generation device is not limited to images from a medical endoscope. The endoscopic auxiliary data generation device may perform diagnostic support on images acquired from other imaging devices, such as cameras, video cameras, industrial endoscopes, microscopes, robots with image acquisition capabilities, smartphones, mobile phones, smartwatches, tablet devices, laptop computers, and other mobile devices.

[0096] The present invention can be applied to an endoscope system and the like.

[0097] 500 Endoscope system 400 Display device 300 Light source device 100 Endoscope 110 Insertion section 120 Tip section 130 Bending section 140 Flexible tube section 150 Imaging section 160 Illumination section 170 Sensor 180 Operation section 200 Image processing processor device (endoscopic auxiliary information generating device) 210 Control section 220 Image acquisition section 230 Image recording section 240 Attention area detection section 250 Insertion position detection section 260 Relative position change detection section 270 Auxiliary information generation section 280 Inference section 281 Inference model 290 Image synthesis section D Captured image E Auxiliary information E2 Second auxiliary information S Synthesized image

Claims

1. An image acquisition unit sequentially acquires multiple images of the inside of a living body obtained by the imaging unit of an endoscope that images the inside of the lumen of a subject, A focus region detection unit detects a focus region having specific characteristics in the aforementioned image, A relative position change detection unit detects a change in the relative position between the imaging unit and the object being imaged by the imaging unit based on the aforementioned image, An auxiliary information generation unit generates auxiliary information to guide the operation of the endoscope in response to the continuous changes in the relative position, Equipped with, When the relative position change detection unit detects that the continuous change in the relative position is mainly due to the movement of the living organism, the auxiliary information generation unit generates the auxiliary information in accordance with the change in the area of ​​interest. The endoscope auxiliary information generation device described.

2. When the relative position change detection unit detects that the continuous change in the relative position is based on the movement of the endoscope, the auxiliary information generation unit generates the auxiliary information that assists in determining the path of the endoscope in the direction of movement within the living body. The endoscope auxiliary information generation device according to claim 1.

3. The auxiliary information that assists the aforementioned path in the direction of travel is, Information identifying the lumen, Based on the vertical direction of the screen during the insertion of the endoscope and the examination position of the subject, the anatomical information of the direction of lumen curvature that occurs with the insertion of the specific lumen, Characterized by being generated according to, The endoscope auxiliary information generation device according to claim 2.

4. The aforementioned auxiliary information includes either the cause of the loss of the area of ​​interest, or a measure to recover from the loss of the area of ​​interest. Endoscope auxiliary information generation device according to any one of claims 1 to 3.

5. The auxiliary information generation unit further includes an inference model, The inference model is a model trained using training data that identifies image frames before and after the disappearance of the region of interest from the plurality of images obtained by the image acquisition unit, and includes annotations regarding the cause of the disappearance for the image frame after the disappearance of the region of interest, and is trained to infer the cause of the disappearance when the image frame in which the region of interest has disappeared is input. The auxiliary information generation unit infers the cause of the disappearance of the area of ​​interest in the image as auxiliary information. The endoscope auxiliary information generation device according to claim 1.

6. The inference model is a model trained using training data that identifies image frames before and after the disappearance of the specific feature from the image obtained by the image acquisition unit, and further includes annotations relating to a recovery strategy for recovering the disappearance in the image frame after the disappearance of the region of interest, and is trained to infer the recovery strategy when the image frame in which the region of interest has disappeared is input. The auxiliary information generation unit further infers the recovery measures as auxiliary information for the image in which the area of ​​interest has disappeared. The endoscope auxiliary information generation device according to claim 5.

7. The auxiliary information includes the image before the area of ​​interest disappears. The endoscope auxiliary information generation device according to claim 1.

8. The aforementioned auxiliary information includes the orientation of the patient's body. The endoscope auxiliary information generation device according to claim 1.

9. The auxiliary information includes the history of the orientation of the imaging unit of the endoscope. The endoscope auxiliary information generation device according to claim 1.

10. When the auxiliary information generation unit cannot infer the auxiliary information using the inference model, The image before the region of interest disappears, The patient's body position and The history of the orientation of the imaging unit of the endoscope, One of the above is generated as the auxiliary information: The endoscope auxiliary information generation device according to claim 5 or claim 6.

11. Multiple images of the body obtained by the imaging unit of the endoscope are acquired sequentially. In the aforementioned image, a region of interest having specific characteristics is detected, Based on the aforementioned image, the system detects changes in the relative position between the imaging unit and the object being imaged by the imaging unit. In response to the continuous changes in the relative position, auxiliary information is generated to guide the operation of the endoscope. When it is detected that the continuous change in the relative position is primarily due to the movement of the living organism, the auxiliary information is generated in accordance with the change in the area of ​​interest. Method for generating endoscopic support information.

12. When it is detected that the continuous change in the relative position is primarily due to the movement of the living organism, the auxiliary information is generated in accordance with the change in the area of ​​interest. The method for generating endoscopic auxiliary information according to claim 11.

13. When it is detected that the continuous change in the relative position is mainly due to the movement of the endoscope, the auxiliary information is generated to assist the path of the endoscope in the direction of movement within the living body. The method for generating endoscopic auxiliary information according to claim 11.

14. On the computer, Multiple images of the body obtained by the imaging unit of the endoscope are acquired sequentially. In the aforementioned image, a region of interest having specific features is detected. Based on the aforementioned image, the imaging unit detects a change in the relative position between the imaging unit and the object it is imaging. In response to the continuous changes in the relative position, auxiliary information is generated to guide the operation of the endoscope. When it is detected that the continuous change in the relative position is primarily due to the movement of the living organism, the auxiliary information is generated in accordance with the change in the area of ​​interest. Endoscopy support information generation program.

15. From the continuous images obtained by the image acquisition unit of an endoscope used to observe the inside of a living body, a region of interest with specific characteristics within the image is detected. A method for training an inference model that identifies image frames before and after the disappearance of a specific feature from images obtained by the image acquisition unit, annotates the frames after the disappearance with the cause of the disappearance and countermeasures to recover it, and then, when an image in which a region of interest with a specific feature has disappeared from an endoscopic image is input, infers the cause of the disappearance and countermeasures to recover it and outputs it.

16. An insertion guide method for inserting an endoscope from the end of the lumen of a specific internal organ of a subject, The steps include sequentially acquiring image information from an imaging unit located at the tip of the endoscope, A display step in which the aforementioned image information is displayed sequentially, The steps include: inputting information about the subject's posture during the examination, The steps include: obtaining information on the direction of curvature within the lumen of the specified internal organ; The steps include generating endoscopic support information according to anatomical information of the direction of lumen curvature that occurs as a result of inserting the lumen of the specific internal organ, using the vertical direction of the screen during endoscope insertion as a reference, Equipped with, Method for generating endoscopic support information.

17. On the computer, The steps include sequentially acquiring image information from the imaging unit located at the tip of the endoscope, A display step in which the aforementioned image information is displayed sequentially, The steps include inputting the body position information of the subject undergoing the endoscopic examination, The steps include generating endoscopic support information according to anatomical information of the lumen curvature that occurs during the insertion of the specific organ, based on information of the lumen curvature direction within the lumen of the specific organ and the vertical direction of the screen during endoscope insertion, Equipped with, Endoscopy support information generation program.

18. An image acquisition unit that sequentially acquires image information from an imaging unit located at the tip of the endoscope, A display control unit for sequentially displaying the aforementioned image information, Information from an input unit that inputs the positional information of the subject during the endoscopic examination, information on the direction of curvature of the lumen within the lumen of a specific organ, and an auxiliary information generation unit that generates endoscopic auxiliary information according to the anatomical information of the direction of curvature that occurs as a result of inserting the lumen of the specific organ, based on the vertical direction of the screen during endoscopic insertion. Equipped with, Endoscope image processing device.

19. An image acquisition unit that sequentially acquires image information from an imaging unit located at the tip of the endoscope, A display control unit for sequentially displaying the aforementioned image information, An input unit for inputting the body position information of the subject of the endoscopic examination, An auxiliary information generation unit generates endoscopic auxiliary information according to anatomical information of the lumen curvature that occurs as a result of inserting the lumen of a specific organ, based on information of the direction of curvature within the lumen of a specific organ, the body position information, and the vertical direction of the screen during endoscope insertion. Equipped with, Endoscopic support system.