Endoscope diagnosis assistance device, endoscope diagnosis assistance method, and endoscope diagnosis assistance program

Abstract

This endoscope diagnosis assistance device comprises: an image acquisition unit that acquires lumen images from an endoscope; a region determination unit that identifies a transiting region in the lumen captured in the images; a characteristic determination unit that determines a characteristic of the transiting region; and a diagnosis assistance information generation unit that, on the basis of the characteristic of the transiting region, generates assistance information for accessing or diagnosing a region to be transited in the lumen.

Description

Endoscopic diagnosis support device, endoscopic diagnosis support method, and endoscopic diagnosis support program

[0001] The present invention relates to an endoscopic diagnosis support device, an endoscopic diagnosis support method, and an endoscopic diagnosis support program.

[0002] Computer-aided image diagnosis (CAD: Computer-Aided Detection / Diagnosis) has been developed to analyze endoscopic images using a computer and assist the surgeon in making a diagnosis. Because the lumens diagnosed by endoscopy vary from patient to patient, it is desirable to provide diagnostic support that takes into account the individual differences in the lumens of each patient. It is particularly desirable to provide diagnostic support that is suitable for surgeons with little diagnostic experience. Individual differences in the lumens of each patient should be considered not only in diagnostic support, but also in the process of accessing the diagnostic site leading up to a diagnosis, as well as in screening and treatment. Hereinafter, support for medical professionals under these circumstances will be collectively referred to as diagnostic support.

[0003] Patent Document 1 describes a diagnostic support system that provides information regarding endoscopic operation to support diagnosis. The diagnostic support system described in Patent Document 1 acquires three-dimensional medical images of the inside of the body using a device capable of capturing three-dimensional images of the inside of the body, such as an X-ray CT, an X-ray cone beam CT, an MRI-CT, or an ultrasonic diagnostic device, generates a virtual endoscopic image reconstructed from the acquired three-dimensional medical image, and outputs operational support information regarding endoscopic operation.

[0004] Japanese Patent Application Laid-Open No. 2021-153808

[0005] However, the diagnosis support system described in Patent Document 1 requires that virtual endoscopic images be acquired before surgery, which is difficult to do, particularly when emergency treatment is required for a patient.

[0006] The present invention has been made in consideration of the above circumstances, and aims to provide an endoscopic diagnosis support device, an endoscopic diagnosis support method, and an endoscopic diagnosis support program that can provide diagnostic support that takes into account individual differences in the lumens of each patient, without requiring any preliminary work before surgery.

[0007] In order to solve the above problems, the present invention proposes the following means: An endoscopic diagnosis support device according to a first aspect of the present invention includes an image acquisition unit that acquires an image of a lumen from an endoscope, a region determination unit that identifies a transit region of the lumen captured in the image, a feature determination unit that determines features of the transit region, and a diagnosis support information generation unit that generates support information for accessing or diagnosing the planned transit region in the lumen based on the features of the transit region.

[0008] The endoscopic diagnosis support device, endoscopic diagnosis support method, and endoscopic diagnosis support program of the present invention can provide diagnostic support that takes into account individual differences in the lumen of each patient, without requiring any preliminary work before surgery.

[0009] 1 is a diagram illustrating an endoscopic system according to an embodiment. It is a functional block diagram of the endoscopic system. It is a diagram illustrating diagnostic support information. It is a diagram illustrating an example of the diagnostic support information for a planned passage area. It is a diagram illustrating an example of the diagnostic support information for a planned passage area. It is a functional block diagram of a region determination unit. It is a diagram illustrating an inference model of a diagnostic support information generation unit. It is an example of diagnostic support information that takes into account individual differences in lumens for each class of patient. It is an example of diagnostic support information that takes into account individual differences in lumens for each class of patient. It is a diagram illustrating training data used for learning the inference model. It is a diagram illustrating an example of a composite image generated by a display control unit. It is a diagram illustrating an example of the composite image generated by the display control unit. It is a control flowchart of the endoscopic system.

[0010] An endoscope system 500 according to an embodiment of the present invention will be described with reference to FIGS. 1 to 14. FIG.

[0011] [Endoscope system 500] Fig. 1 is a diagram showing an endoscope system 500. The endoscope system 500 includes an endoscope 100, an image processing processor device 200, a light source device 300, and a display device 400. The image processing processor device 200 and the light source device 300 may be an integrated device (image control device).

[0012] The light source device 300 has a light source 310 such as an LED, and controls the light source to control the amount of illumination light transmitted to the endoscope 100 via the light guide 161 .

[0013] The display device 400 is a device that displays images generated by the image processing device 200 and various information related to the endoscope system 500. The display device 400 is, for example, a liquid crystal monitor or a head-mounted display.

[0014] [Endoscope 100] The endoscope 100 is a device for observing and treating the inside of the body of a patient lying on, for example, an operating table T. The endoscope 100 includes an elongated insertion section 110 that is inserted into the patient's body, an operation section 180 that is connected to the base end of the insertion section 110, and a universal cord 190 that extends from the operation section 180.

[0015] The insertion section 110 has a distal end section 120, a bendable bending section 130, and a long, flexible flexible tube section 140. The distal end section 120, the bending section 130, and the flexible tube section 140 are connected in this order from the distal end side. The flexible tube section 140 is connected to the operation section 180.

[0016] 2 is a functional block diagram of the endoscope system 500. The distal end portion 120 has an imaging unit 150, an illumination unit 160, and an information acquisition unit 170.

[0017] The imaging unit 150 has an optical system, an imaging element that converts optical signals into electrical signals, and an AD conversion circuit that converts analog signals output by the imaging element into digital signals. The imaging unit 150 captures an image of a subject and generates an imaging signal. The imaging signal is acquired by the image processing processor device 200 via an imaging signal cable 151.

[0018] The illumination unit 160 irradiates the subject with illumination light transmitted by the light guide 161. The light guide 161 is connected to the light source device 300 by inserting the insertion section 110, the operation section 180, and the universal cord 190. The illumination unit 160 may include a light source such as an LED, an optical element such as a phosphor having a wavelength conversion function, or the like.

[0019] The information acquisition unit 170 detects the position, speed, and direction of the tip unit 120. The information acquisition unit 170 is, for example, a six-axis sensor or a three-axis sensor. The output of the information acquisition unit 170 is acquired by the image processing processor device 200 via a signal cable 171. The information acquisition unit 170 detects the state and position of the tip unit 120, and can detect the direction of gravity from acceleration and the movement process of the tip unit 120 from changes in that direction. The information acquisition unit 170 may be any device that can achieve the same purpose, such as a magnetic sensor or a magnetically emitting component, or a device that detects movement or magnetism in cooperation with an external sensor or system. Furthermore, since the movement of the tip unit 120 can be detected from changes in the image output acquired by the imaging unit 150, the imaging unit 150 may serve as the information acquisition unit 170. Furthermore, the information acquisition unit 170 may be realized by coordinating the above-mentioned means.

[0020] The operation unit 180 accepts operations for the endoscope 100. The operation unit 180 has an ankle knob 181 that controls the bending section 130, an air / water supply button 182, a suction button 183, and a release button 184. Operations input to the air / water supply button 182, the suction button 183, and the release button 184 are acquired by the image processing processor 200. The release button 184 is a push button that is used to input an operation to save an image acquired from the imaging unit 150.

[0021] The universal cord 190 connects the endoscope 100 and the image processor device 200. The universal cord 190 is a cable through which the imaging signal cable 151, the light guide 161, the signal cable 171, etc. are inserted.

[0022] [Image Processing Processor Device 200] As shown in Figure 2, the image processing processor device 200 includes an information acquisition unit 210, an image acquisition unit 220, an image recording unit 230, an endoscopic diagnosis support unit 240, and a display control unit 290.

[0023] The image processor device 200 is a computer capable of executing programs and including a processor such as a CPU, a memory, a recording unit, etc. The functions of the image processor device 200 are realized by the processor executing the programs. At least some of the functions of the image processor device 200 may be realized by a dedicated logic circuit implemented in an ASIC or FPGA.

[0024] The image processing processor device 200 may further include components other than the processor, memory, and recording unit. For example, the image processing processor device 200 may further include an image calculation unit that performs part or all of the image processing and image recognition processing. By including the image calculation unit, the image processing processor device 200 can execute specific image processing and image recognition processing at high speed. The image calculation unit may be a calculation unit provided in a cloud server connected via the Internet.

[0025] The recording unit is a non-volatile recording medium that stores the above-mentioned program and data necessary for executing the program. The recording unit is composed of, for example, a writable non-volatile memory such as a flexible disk, a magneto-optical disk, a ROM, or a flash memory, a portable medium such as a CD-ROM, or a storage device such as a hard disk or SSD built into a computer system. The recording unit may also be a storage device provided in a cloud server connected via the Internet.

[0026] The above program may be provided by a "computer-readable recording medium" such as a flash memory. The program may be transmitted from a computer storing the program to a memory or recording unit via a transmission medium or by transmission waves in the transmission medium. A "transmission medium" that transmits the program is a medium that has the function of transmitting information. Media that have the function of transmitting information include networks (communication networks) such as the Internet and communication lines (communication lines) such as telephone lines. The above program may realize some of the functions described above. Furthermore, the above program may be a difference file (difference program). The above function may be realized by combining a program already recorded on the computer with a difference program.

[0027] The information acquisition unit 210 controls the entire image processing processor device 200. The information acquisition unit 210 also acquires information about the case in which the endoscopic system 500 is used (type of endoscope 100, patient information, surgeon information) from an in-hospital system or the like. The information acquisition unit 210 may also acquire information about the case by having the surgeon or an assistant input the information from an input device (not shown). The information input to the information acquisition unit 210 is acquired by the endoscopic diagnosis support unit 240.

[0028] The image acquisition unit 220 acquires an image signal from the imaging unit 150 of the endoscope 100 via an image signal cable 151. The image acquisition unit 220 performs image signal processing on the image signal acquired from the imaging unit 150 to sequentially acquire captured images D. The image acquisition unit 220 outputs the acquired captured images D to the display control unit 290. The image acquisition unit 220 also outputs the acquired captured images D to the endoscopic diagnosis support unit 240 via the image recording unit 230.

[0029] The image recording unit 230 is a part of the recording unit described above and is a non-volatile recording medium. The image recording unit 230 is a part of the memory described above and may be a volatile recording medium. The image recording unit 230 records the multiple captured images D that have been transferred.

[0030] The image recording unit 230 records multiple captured images D (image frames, time-series images) input in chronological order. When the image recording unit 230 does not have enough storage capacity, the oldest captured image D is deleted. The multiple captured images D recorded in the image recording unit 230 may be captured images D of consecutive frames, or captured images D obtained by thinning out multiple frames from consecutive frames. The image recording unit 230 temporarily records each frame (frame) obtained from the imaging unit 150 and compares the previous and next frames, thereby acquiring information similar to that of the information acquiring unit 170. By comparing the images of the previous and subsequent frames, if the image spreads from the center of the screen to the periphery, it can be determined that the distal end portion 120 is being inserted along the lumen. If the image of the periphery changes toward the center of the screen, it can be determined that the distal end portion 120 is being removed.

[0031] FIG. 3 is a diagram illustrating diagnostic support information. For example, as shown in FIG. 3 , the large intestine is made up of multiple connected cylindrical tubes, and the shape and size vary from patient (subject) P to patient (subject). Therefore, the time required to insert and remove the endoscope 100 forward in the direction of travel while bending the insertion section 110 of the endoscope 100 along the series of cylindrical tubes varies depending on the individual differences in the large intestine of each patient P and the skill of the surgeon S. Meanwhile, it is difficult to extend the examination time or treatment time due to factors such as the examination time and the effective time of anesthesia. Therefore, to support the surgeon S, the endoscopic diagnostic support unit 240 provides diagnostic support for the planned passage area AP that takes into account the individual differences in the large intestine of each patient P. Furthermore, it is desirable that the content and level of detail of the diagnostic support for the planned passage area AP be changed depending on the skill of the surgeon S.

[0032] The endoscopic diagnosis support unit 240 can also provide diagnostic support for the expected passage area AP in lumens other than the large intestine. For example, in order to support the surgeon S, the endoscopic diagnosis support unit 240 provides diagnostic support for the expected passage area AP taking into account individual differences in the stomach of each patient P.

[0033] Specifically, the endoscopic diagnosis support unit 240 determines the characteristics of the mid-pass region AC of the lumen captured in the captured image D. Next, based on the characteristics of the mid-pass region AC, the endoscopic diagnosis support unit 240 generates diagnostic support information for the expected pass region AP that takes into account individual differences in the lumen for each patient P. Here, since there is a correlation between each region of the lumen, it is possible to predict the characteristics of the expected pass region AP that take into account individual differences in the lumen for each patient P from the characteristics of the mid-pass region AC.

[0034] The endoscopic diagnosis support unit 240 generates diagnostic support information for the expected passage region AP, as needed, taking into account the patient information (gender, age, race, body type, examination results, etc.) acquired by the information acquisition unit 210. For example, it is said that "the internal diameter of the large intestine is positively correlated with height and weight and negatively correlated with age for both the transverse and sigmoid colons," that "transverse colon prolapse is correlated with gender, age, height, weight, obesity level, and the lengths of the transverse colon and large intestine," and that "sigmoid colon elevation is correlated with age and the lengths of the sigmoid colon and large intestine." Therefore, by taking into account the patient information acquired by the information acquisition unit 210, the endoscopic diagnosis support unit 240 can specify the correlations in each region of the lumen for each class (attribute) of patient P and more accurately predict the characteristics of the diagnostic support information for the expected passage region AP.

[0035] The expected passage region AP is a region located further forward in the direction of travel of the endoscope 100 than the passage region AC. As shown in Figure 3, when the insertion section 110 of the endoscope 100 is inserted into a lumen, the expected passage region AP is a region in the lumen that is deeper than the passage region AC. When the insertion section 110 of the endoscope 100 is removed from the lumen, the expected passage region AP is a region in the lumen that is closer to the natural orifice than the passage region AC.

[0036] Fig. 4 is a diagram showing an example of diagnostic support information for a planned passage region AP. As shown in Fig. 4, when the surgeon S inserts the insertion section 110 of the endoscope 100 into the large intestine and the current passage region AC is the "rectum," the endoscopic diagnostic support unit 240 generates diagnostic support information for the "sigmoid colon," which is the planned passage region AP. The diagnostic support information shown in Fig. 4 is a predicted time required for the surgeon S to examine the planned passage region AP.

[0037] 5 and 6 are diagrams showing examples of diagnostic support information for a planned passage region AP. As shown in Fig. 5, when examination results such as the location of pain in the patient P can be obtained in advance, the endoscopic diagnosis support unit 240 generates diagnostic support information for a candidate region where a lesion is located (also referred to as a "specific target region or specific site"). The diagnostic support information shown in Fig. 6 notifies the surgeon S that the candidate region where the lesion is located is the lower part of the descending colon or the right part of the sigmoid colon, and urges the surgeon S to observe carefully.

[0038] The main symptom a patient reports to a doctor is called a chief complaint. In the example shown in Figure 5, the chief complaint is a symptom of "pain here" while pointing to the patient's side. The endoscopic diagnosis support unit 240 can provide diagnostic support information that enables an appropriate diagnosis for such a chief complaint. For example, the patient may explain other concerns, but in this case, the patient clearly complains of abdominal pain, which is likely related to the endoscopic examination, and therefore is designated as the chief complaint related to the examination. Even if the patient or subject simply touches this area with their hand, the doctor can estimate the anatomical location of the area within the digestive tract during the endoscopic examination. The doctor may input information obtained from such a chief complaint (also referred to as "chief complaint information") as part of the patient information (S110 in Figure 14, described below). The endoscopic diagnosis support unit 240 may also acquire chief complaint information by reading the results of the patient's checkmarks written on a piece of paper with a shape resembling the human body, or by displaying a UI display resembling the human body on a terminal and reading the checkmarks entered there. The endoscopic diagnosis support unit 240 may be provided with a program or database that determines more specifically which part of which organ the part mentioned by the patient is, based on the input results.

[0039] As shown in FIG. 2 , the endoscopic diagnosis support unit 240 includes a region determination unit 250 , a feature determination unit 260 , and a diagnosis support information generation unit 270 .

[0040] The endoscopic diagnosis support unit 240 may be a device separate from the image processing device 200 (hereinafter, also referred to as an "endoscopic diagnosis support device"). The endoscopic diagnosis support device may be a computing device provided on a cloud server connected via the Internet. The computing device provided on the cloud server may have an image acquisition unit and may be equipped with a determination function for determining the location of the endoscope tip from the acquired image, or a determination function for determining the characteristics of the area through which the tip 120 of the endoscope is currently passing (characteristics of the area during tip passage) from information acquired by the tip 120. Depending on the system, an image recording unit may be provided in the computing device. Also, although images are acquired via the image recording unit 230 in FIG. 2, the image may not necessarily be acquired via the image recording unit 230. The area determination unit 250 and the feature determination unit 260 acquire intraluminal feature information obtained during the insertion process into the digestive lumen during endoscopic examination, and therefore may be referred to as an intraluminal feature information acquisition unit.

[0041] 7 is a functional block diagram of the region determination unit 250. The region determination unit 250 identifies a passing region AC of the lumen imaged in the captured image D. When the insertion section 110 of the endoscope 100 is inserted into the large intestine, the region determination unit 250 analyzes the obtained image and identifies whether the region of the large intestine included in the captured image D is one of the cecum, ascending colon, transverse colon, descending colon, sigmoid colon, rectosigmoid portion, etc.

[0042] The region determination unit 250 illustrated in Fig. 7 has multiple region determination units (a first region determination unit 251, a second region determination unit 252, and an n-th region determination unit 25n). Each region determination unit is a determination unit dedicated to a region of the lumen. For example, if the lumen is the large intestine, the first region determination unit 251 is a determination unit that determines that the mid-passage region AC of the lumen imaged in the captured image D is the "rectum." Furthermore, the second region determination unit 252 is a determination unit that determines that the mid-passage region AC of the lumen imaged in the captured image D is the "sigmoid colon."

[0043] Each region determination unit may use pattern matching to identify the passing region AC of the lumen included in the captured image D. For example, each region determination unit compares the captured image D with pre-recorded images of each part, and identifies the passing region AC of the lumen included in the captured image D based on the similarity to each pre-recorded part.

[0044] Each region determination unit may use a machine learning model to identify the passing region AC of the lumen included in the captured image D. For example, each region determination unit uses a machine learning model that has been trained in advance so as to be able to detect the region of the lumen included in the captured image D from the captured image D to identify the passing region AC of the lumen included in the captured image D.

[0045] Each region determination unit may identify a passing region AC of the lumen included in the captured image D based on the output of the information acquisition unit 170 (speed, direction, posture, etc. of the distal end portion 120).

[0046] The region determination unit 250 may identify the passing region AC of the lumen imaged in the multiple captured images D using only a region determination unit that can distinguish and identify multiple regions of the lumen, without using multiple region determination units.

[0047] The region determining section 250 may identify the passing region AC of the lumen included in the captured image D by combining the above-mentioned multiple techniques.

[0048] The region determining unit 250 transmits the identified region of the lumen included in the captured image D to the diagnosis assistance information generating unit 270 .

[0049] The feature determining unit 260 determines the features of the transit region AC from the obtained image signal. The features of the transit region AC determined by the feature determining unit 260 include, for example, the shape (intraluminal length, inner diameter, etc.) and the surface condition (wall condition, blood vessel condition, etc.). The intraluminal length is the length in the direction in which the lumen extends.

[0050] The feature determining section 260 may determine the features of the passing area AC by performing image processing on the captured image D.

[0051] The feature determining unit 260 may determine the feature of the passing region AC based on the output (such as the speed, direction, and posture of the distal end portion 120) of the information acquiring unit 170. For example, the feature determining unit 260 can determine the intraluminal length of the passing region AC based on the speed and direction of the distal end portion 120.

[0052] The characteristic determination unit 260 may determine the characteristics of the passing area AC based on the operation history input to the operation unit 180. For example, if multiple operations of supplying water to the passing area AC have been performed, the characteristic determination unit 260 can determine that the passing area AC is one of the areas where residue is likely to accumulate.

[0053] The feature determining unit 260 may determine the features of the passing area AC by combining the above-mentioned methods.

[0054] The feature determining unit 260 transmits the determined region of the lumen included in the captured image D to the diagnosis assistance information generating unit 270 .

[0055] The diagnostic assistance information generating unit 270 generates diagnostic assistance information for the planned passage region AP based on the captured image D and the identification result and characteristics of the currently passing region AC. The diagnostic assistance information generating unit 270 may generate the diagnostic assistance information on a rule basis, or may generate the diagnostic assistance information using an inference model 281 included in the inference unit 280. The diagnostic assistance information generating unit 270 can also be expressed as outputting guide information for accessing a specific region prior to diagnosis, and may therefore be rephrased as a guide information output unit that outputs guide information for when the distal end portion 120 of the endoscope 100 is inserted further from the pass region in accordance with the intraluminal characteristic information to access a specific target region.

[0056] The inference unit 280 may use conventional general-purpose arithmetic processing circuits such as a CPU or FPGA (Field Programmable Gate Array), but because much of the processing in neural networks involves matrix multiplication, it may also use a GPU (Graphic Processing Unit) or a Tensor Processing Unit (TPU), which are specialized for matrix calculations. In recent years, such dedicated artificial intelligence (AI) hardware, called "neural network processing units (NPUs)," have been designed to be integrated and embeddable with other circuits such as CPUs, and may even become part of the processing circuit.

[0057] FIG. 8 is a diagram showing an inference model 281 and an image of how it is trained. The inference model 281 is a model trained using training data including annotations related to diagnostic support information for a planned passage area AP for multiple image frames (learning captured images) in multiple cases. The inference model 281 is, for example, a neural network, and is trained by deep learning. Note that the inference model 281 is not limited to a neural network and may be another machine learning model that can output information for an input image.

[0058] The inputs of the inference model 281 are a plurality of captured images D (image frames, time-series images) input in chronological order, and the identification results and features of the passing area AC. The output of the inference model 281 is diagnostic support information for the planned passing area AP.

[0059] The input to the inference model 281 may include patient information and surgeon information acquired by the information acquisition unit 210. For example, by inputting patient information of patient P (gender, age, race, body type, examination results, etc.) to the inference model 281, the inference model 281 can more easily output diagnostic support information that takes into account individual differences in lumens for each class (attribute) of patient P. For example, by inputting the skill (proficiency) of surgeon S, which is surgeon information, to the inference model 281, the inference model 281 can more easily output diagnostic support information that corresponds to the skill of surgeon S.

[0060] In this way, by training the inference model 281 using training data in which feature information of a second section inserted later than the first section is annotated for an image of a first section obtained during the process of inserting the endoscope into the digestive lumen during endoscopic examination, a guide control method for an endoscope can be provided, which includes a display control unit 290 that infers and displays feature information of the second section inferred by the inference model 281 when an image corresponding to the first section of an intraluminal image acquired from the imaging unit 150 of the endoscope 100 is input to the inference model 281. Furthermore, when the diagnostic assistance information generation unit 270 is referred to as a "guide information output unit," it can also be expressed as having an inference model trained using training data in which feature information of a second section, which is reached by inserting the distal end 120 of the endoscope 100 further into the first section, is annotated for an image of the first section obtained during the passage of the distal end of the endoscope. When inserting the endoscope 100, it is impossible to know what lies beyond the insertion point, so a mechanism for predicting this is important. Therefore, although the first section has been described as being close to the insertion position and the second section as being further inside, the reverse is also possible. This concept can also be used when removing the endoscope 100, because the lumen is soft and may assume a different shape when inserted and removed.

[0061] Furthermore, when a subject during an endoscopic examination has a chief complaint, such as a concern, there is a need to closely observe that area. Therefore, a doctor can be mentally prepared if some kind of guidance is provided prior to the observation of the specific area before the imaging unit 150 of the distal end portion 120 of the endoscope 100 moves through that area. That is, if endoscopic images obtained during the process of intraluminal insertion of the digestive tract during a gastrointestinal endoscopic examination of the subject are acquired, and the specific area is determined based on the chief complaint, an inference model 281 can be obtained that is trained using training data in which information about the specific area is annotated for endoscopic images of the insertion process acquired at a timing prior to the specific area. Therefore, by using an endoscopic guide control method including a display control step of inferring and displaying information about the specific area inferred by the inference model 281 when an intraluminal image acquired from the imaging unit 150 of the endoscope 100 is input, an accurate examination can be performed without overlooking any concerns the patient may have.

[0062] FIG. 9 is an example of diagnostic support information that takes into account individual differences in the lumen for each class of patient P. When the surgeon S inserts the insertion section 110 of the endoscope 100 into the large intestine and the current passage area AC is the "rectum," diagnostic support information for the "sigmoid colon," "descending colon," and "transverse colon," which are the expected passage areas AP, is output for each class of patient P. The diagnostic support information illustrated in FIG. 9 is the predicted time required for an average surgeon to inspect (pass through) the expected passage area AP. The predicted time required for the inspection is calculated, for example, based on the intraluminal length (estimated value) of the expected passage area AP. The diagnostic support information may also be the difficulty level for each expected passage area AP.

[0063] FIG. 10 is an example of diagnostic support information that takes into account individual differences in the lumen for each class of patient P. When the surgeon S inserts the insertion section 110 of the endoscope 100 into the large intestine and the current transit area AC is the "sigmoid colon," diagnostic support information for the "descending colon," "transverse colon," and "ascending colon," which are the expected transit areas AP, is output for each class of patient P. The diagnostic support information illustrated in FIG. 10 is the predicted time required for an average surgeon to inspect (pass through) the expected transit area AP. The predicted time required for the inspection is calculated, for example, based on the intraluminal length (estimated value) of the expected transit area AP. The diagnostic support information may also be the difficulty level for each expected transit area AP.

[0064] The input of the inference model 281 may include the identification result and characteristics of the passed area AD. The passed area AD is an area located behind the passing area AC in the direction of travel of the endoscope 100, and is an area through which the tip 120 of the endoscope 100 has already passed. For example, when the surgeon S inserts the insertion section 110 of the endoscope 100 into the large intestine, and the passing area AC is the "sigmoid colon," one of the passed areas AD is the "rectum." By inputting the identification result and characteristics of the passed area AD to the inference model 281, the inference model 281 can more accurately output diagnostic support information for the planned passing area AP. As the examination and diagnosis progress, the passed area AD increases, and the diagnostic support information for the planned passing area AP becomes more accurate.

[0065] 11 is a diagram illustrating training data. A plurality of image frames (a sequence of still images) obtained during endoscopic examinations of a plurality of cases are used as training data. The training data is a combination of a plurality of image frames (learning captured images) and annotations related to diagnostic support information for the planned passage area AP. The inference model 281 is a model trained using the training data to output corresponding annotations for input image frames (learning captured images).

[0066] The annotations of the training data may include a recovery plan. For example, the recovery plan may be, "The planned passage area AP is very difficult, so we should seek support from veterans or experts." If the annotations of the training data include a recovery plan, the inference model 281 can further output the recovery plan.

[0067] The diagnostic assistance information generator 270 may generate diagnostic assistance information for the expected passage region AP based only on the captured image D. In this case, the inference model 281 becomes a model that outputs diagnostic assistance information for the expected passage region AP from only the passing region AC of the lumen imaged in the captured image D.

[0068] The diagnostic assistance information generating unit 270 may generate diagnostic assistance information for the planned passing region AP based only on the identification result and characteristics of the currently passing region AC.

[0069] 12 and 13 are diagrams showing an example of the composite image S. The display control unit (image composition unit) 290 generates the composite image S including the captured image D and the diagnostic support information E.

[0070] The composite image S shown in Fig. 12 was generated when the surgeon S inserted the insertion section 110 of the endoscope 100 into the large intestine (advance / retract direction: insertion). The diagnostic support information E shown in Fig. 12 includes the class (attributes) of the patient P, the overall difficulty level of the planned passage area AP, presentation of the "transverse colon" which is a particularly difficult area, and a recovery plan.

[0071] The composite image S shown in Fig. 13 was generated when the surgeon S was performing an examination while withdrawing the insertion section 110 of the endoscope 100 from the large intestine (advance / retract direction: withdrawal). The diagnostic support information E shown in Fig. 13 includes a presentation of the class (attributes) of the patient P and a specific target region (for example, a region corresponding to a location where the patient P complains of pain, which is a candidate region for a lesion).

[0072] [Operation of Endoscopic System 500] Next, the operation of the endoscope system 500 (diagnosis support method) will be described. Specifically, the procedure of observing and treating the luminal wall of the large intestine using the endoscope system 500 will be described. Note that the endoscope system 500 can also provide diagnostic support for organs other than the large intestine (such as the stomach, bronchi, or urinary organs). The following description will be given following the control flowchart of the endoscope system 500 shown in FIG. 14.

[0073] <Step S110> In step S110, the information acquisition unit 210 acquires information about the case (type of endoscope 100, patient information, surgeon information). Chief complaint information may also be input in step S110. The information acquisition unit 210 may determine, from the input chief complaint information, which parts should be closely examined in the upcoming examination by referring to a specific program or database. The information acquisition unit 210 may acquire information about the patient's condition by having the surgeon or assistant input the information from an input device (not shown). The endoscope system 500 then executes step S120.

[0074] <Step S120> In step S120, the region determination unit 250 detects the insertion direction of the endoscope 100. Specifically, the region determination unit 250 acquires the output of the information acquisition unit 170 and detects the advancement / retraction direction (insertion direction, withdrawal direction) of the tip portion 120 of the endoscope 100 based on the output of the information acquisition unit 170. Note that the region determination unit 250 may identify the region of the lumen imaged in the captured image D from the captured image D and detect the advancement / retraction direction of the endoscope 100 from the history of the identified region. If the insertion direction of the endoscope 100 is the insertion direction, the endoscopic system 500 next executes step S130. If the insertion direction of the endoscope is the withdrawal direction, the endoscopic system 500 next executes step S200.

[0075] <Step S130> In step S130, the region determination unit 250 identifies the passing region AC of the lumen imaged in the captured image D. Next, the endoscope system 500 executes step S140.

[0076] <Step S140> In step S140, the region determination unit 250 determines whether the distal end portion 120 of the endoscope 100 has been inserted to the edge of a specific region such as the cecum. If the distal end portion 120 of the endoscope 100 has been inserted to the edge of the specific region, the endoscopic system 500 executes step S120. If the distal end portion 120 of the endoscope 100 has not been inserted to the edge of the specific region, the endoscopic system 500 executes step S150.

[0077] <Step S150> In step S150, the feature determination unit 260 determines the features of the passing region AC. Based on the image information obtained at this time, the inference model 281 or other databases capture image data information for outputting information on the region to be examined at a later time. Next, the endoscope system 500 executes step S160.

[0078] <Step S160> In step S160, the diagnostic assistance information generator 270 generates diagnostic assistance information for the planned passage area AP based on the captured image D and the identification results and characteristics of the currently passing area AC. The diagnostic assistance information inferred by the inference model 281 when an intraluminal image acquired from the imaging unit 150 of the endoscope 100 is input may be inferred. (This does not necessarily include the diagnosis itself, but may include various information to enable the diagnosis. It may also be auxiliary information for access. Alternatively, it may include information on various operations such as water injection, air supply, and suction, or switching of the light source, switching of image processing, and spraying of reagents.) This provides guide information that predicts what will happen next when an intraluminal image acquired from the imaging unit 150 of the endoscope 100 is input. For example, an endoscope guide control method can be provided that includes a display control step that uses the inference model 281 to infer and display information on a specific area inferred when an endoscopic image is input (information related to ease of access, relationship to the patient's chief complaint, etc., as well as observation and diagnostic know-how). The information about the specific region within the digestive lumen here may be determined from information obtained from the subject's chief complaint, or may be determined according to information about the region currently being observed (passing through) during this digestive lumen examination, as indicated by "Identification result of transiting region AC" in Figure 8. Next, the endoscope system 500 executes step S170.

[0079] <Step S170> In step S170, the display control unit 290 generates a composite image S including the captured image D and the diagnostic support information E, and outputs the composite image S to the display device 400. The display device 400 displays the composite image S. Next, the endoscope system 500 executes step S120. Because the doctor can grasp the information before reaching the examination site, the doctor can mentally prepare and access and examine the patient accurately and precisely without having to hesitate to make a decision in the event that insertion suddenly becomes difficult or overlooking a point of concern to the patient.

[0080] The display control unit 290 may promptly present and notify the user of the diagnostic assistance information according to the content of the generated diagnostic assistance information. For example, if the expected passage area AP is very difficult in consideration of the skill of the surgeon S, the display control unit 290 promptly presents and notifies the user of the diagnostic assistance information.

[0081] <Step S200> In step S200, the region determination unit 250 identifies a passing region AC of the lumen imaged in the captured image D. Next, the endoscope system 500 executes step S210. <Step S210> In step S210, the region determination unit 250 determines whether the passing region AC is a specific target region (for example, a region corresponding to an area where the patient P complains of pain, and which is a candidate region for a lesion). If the passing region AC is a specific target region, the endoscope system 500 then executes step S220. If the passing region AC is not a specific target region, the endoscope system 500 then executes step S230.

[0082] <Step S220> In step S220, the diagnostic assistance information generation unit 270 displays detailed diagnostic assistance information for the examination. Specifically, as shown in Fig. 6 , the diagnostic assistance information generation unit 270 notifies the operator S that the currently passing region AC or the planned passing region AP is a candidate region for a lesion. Furthermore, the diagnostic assistance information generation unit 270 executes the same processes as steps S150 to S170 to generate a composite image S including the captured image D and the diagnostic assistance information E, and outputs the composite image S to the display device 400. Next, the endoscope system 500 executes step S240.

[0083] <Step S230> In step S230, the diagnostic assistance information generator 270 displays normal diagnostic assistance information for the examination. Specifically, the diagnostic assistance information generator 270 executes the same processes as steps S150 to S170 to generate a composite image S including the captured image D and the diagnostic assistance information E, and outputs the composite image S to the display device 400. Next, the endoscope system 500 executes step S240.

[0084] <Step S240> In step S240, the endoscopic diagnosis support unit 240 determines whether the procedure has ended. If the endoscopic diagnosis support unit 240 determines that the procedure has not ended, it executes step S120 and subsequent steps. If the endoscopic diagnosis support unit 240 determines that the procedure has ended, it executes step S300 and ends the control flow shown in FIG. 14.

[0085] According to the endoscopic system 500 of this embodiment, the endoscopic diagnosis support unit 240 (endoscopic diagnosis support device) can provide diagnostic support that takes into account the individual differences in the lumen of each patient P without performing any preliminary work before surgery.

[0086] Although one embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and design modifications and the like are also included within the scope of the present invention. Furthermore, the components shown in the above embodiment and the following modified examples can be appropriately combined to form a configuration.

[0087] 2 is composed of multiple devices, namely, the endoscope 100, the image processing processor 200, and the light source device 300. However, the endoscope 100 may be configured so that the image processing processor 200 and the light source device 300 are built in. Furthermore, the image processing processor 200 does not necessarily need to include the information acquisition unit 210, the endoscopic diagnosis support unit 240, the display control unit 290, and the image recording unit 230. From the viewpoint of system expandability and commonality, the image processing processor 200 may work in conjunction with another processor to achieve similar functions. In other words, the image acquisition unit that acquires an image of the lumen from the endoscope 100, the region determination unit that identifies the currently passing region of the lumen, the feature determination unit that determines the characteristics of the currently passing region, and the diagnostic support information generation unit that generates support information for passage or diagnosis for the expected passage region in the lumen based on the characteristics of the currently passing region do not necessarily need to be included in the same device. The devices may work in conjunction with each other as needed to configure an endoscopic diagnosis support system.

[0088] In the above embodiment, the endoscopic diagnosis support unit (endoscopic diagnosis support device) performs diagnostic support on images taken with a medical endoscope. However, the diagnostic target of the endoscopic diagnosis support unit (endoscopic diagnosis support device) is not limited to images taken with a medical endoscope. The endoscopic diagnosis support unit (endoscopic diagnosis support device) may perform diagnostic support on images taken with other imaging devices, such as cameras, video cameras, industrial endoscopes, microscopes, robots with image acquisition functions, smartphones, mobile phones, smartwatches, tablet terminals, notebook PCs, and other mobile devices.

[0089] The present invention can be applied to an endoscope system and the like.

[0090] 500 Endoscope system 400 Display device 300 Light source device 100 Endoscope 110 Insertion section 120 Tip section 200 Image processing processor device 210 Information acquisition section 220 Image acquisition section 230 Image recording section 240 Endoscopic diagnosis support section 250 Area determination section 260 Feature determination section 270 Diagnosis support information generation section 280 Inference section 281 Inference model 290 Display control section (image synthesis section) AC Passing area AD Passed area AP Planned passage area D Captured image E Diagnosis support information

Claims

1. An endoscopic diagnosis support device comprising: an image acquisition unit that acquires an image of a lumen from an endoscope; an area determination unit that identifies a passing area of ​​the lumen captured in the image; a characteristic determination unit that determines characteristics of the passing area; and a diagnosis support information generation unit that generates support information for accessing or diagnosing the planned passing area in the lumen based on the characteristics of the passing area.

2. The endoscopic diagnosis support device according to claim 1, wherein the characteristics include an intraluminal length in the direction in which the lumen extends.

3. The endoscopic diagnosis support device according to claim 1, wherein the diagnosis support information generation unit generates the support information based on characteristics of a passed region in the lumen that is located behind the passing region in the direction of travel of the endoscope.

4. The endoscopic diagnosis support device according to claim 1, wherein the diagnosis support information generation unit generates more detailed support information when the passing region is a specific target region.

5. The endoscopic diagnosis support device according to claim 4, wherein the specific target area is an area corresponding to a location where the patient is in pain.

6. (Diagnostic support information: predicted time required for examination) The endoscopic diagnostic support device of claim 2, wherein the support information is a predicted time required for examination of the planned passage area, calculated based on the intraluminal length of the planned passage area.

7. The endoscopic diagnostic support device of claim 1, wherein the diagnostic support information generation unit generates the support information for the expected passage area from the characteristics of the currently passing area based on an inference model that has learned about the relationship between the characteristics of the currently passing area in the lumen and the characteristics of the expected passage area.

8. The endoscopic diagnostic support device described in claim 7, wherein the inference model is a model that learns about the relationship between characteristics of a passed area in the lumen that is located behind the passing area in the direction of travel of the endoscope and characteristics of the planned passage area, and the diagnostic support information generation unit generates the support information for the planned passage area from the characteristics of the passed area based on the inference model.

9. The endoscopic diagnosis support device according to claim 1, wherein the diagnosis support information generation unit notifies a user of the support information early on depending on the content of the generated support information.

10. An endoscopic diagnosis support device having an inference model that learns by annotating the characteristics of a planned passage region that is located in front of the currently passing region in the direction of travel of the endoscope in the lumen for an image of the currently passing region obtained during the process of inserting an endoscope into a lumen, and converting the annotated data into training data, wherein the inference model infers the characteristics of the planned passage region when an image of the currently passing region obtained from the endoscope is input.

11. An endoscopic diagnosis support method, comprising: acquiring an image of a lumen from an endoscope; identifying a passing region of the lumen captured in the image; determining characteristics of the passing region; and generating support information for accessing or diagnosing a planned passing region in the lumen based on the characteristics of the passing region.

12. An endoscopic diagnosis support program which causes a computer to obtain an image of a lumen from an endoscope, identify a passing region of the lumen captured in the image, determine characteristics of the passing region, and generate support information for accessing or diagnosing the planned passing region in the lumen based on the characteristics of the passing region.

13. An endoscopic diagnosis support system comprising: an image acquisition unit that acquires an image of a lumen from an endoscope; an area determination unit that identifies a passing area of ​​the lumen captured in the image; a characteristic determination unit that determines characteristics of the passing area; and a diagnosis support information generation unit that generates passage or diagnosis support information for the planned passing area in the lumen based on the characteristics of the passing area.

14. An endoscope guide control method comprising: a step of acquiring an endoscopic image obtained during the process of insertion or removal into a digestive lumen during an endoscopic examination; and a display control step of having an inference model trained using teacher data in which an endoscopic image of a first section during the insertion or removal process is annotated with characteristic information of a second section that passes through at a timing after the first section, and inferring and displaying characteristic information of the second section inferred by the inference model when an image corresponding to the first section of an intraluminal image acquired from an imaging unit of an endoscope is input.

15. The endoscope guide control method according to claim 14, wherein the display control step further comprises: a step of inputting an image obtained from the endoscope into an inference model trained using training data in which the image of the first section is annotated with characteristic information of a second section to which the tip of the endoscope moves after changing position from the passband during the insertion or removal process; and a step of adopting the output result of the inference model.

16. An endoscopic examination guide device comprising: an intraluminal characteristic information acquisition unit that acquires characteristic information of the inside of the lumen through which the tip of the endoscope passes, obtained during the process of insertion into the digestive lumen during endoscopic examination; and a guide information output unit that outputs guide information when the tip of the endoscope is further inserted from the pass zone to access a specific target site in accordance with the intraluminal characteristic information.

17. The endoscopic examination guide device according to claim 16, wherein the guide information output unit is provided with an inference model trained using teacher data in which images of a first section obtained during the passage of the tip of the endoscope are annotated with feature information of a second section reached by the tip of the endoscope being inserted further from the passband.

18. An endoscope guide control method comprising: a step of acquiring information about a specific site within a subject's digestive lumen; a step of acquiring an endoscopic image obtained during the process of insertion into the digestive lumen during a digestive endoscopic examination of the subject; and a display control step of inferring and displaying information about the specific site inferred by an inference model trained using teacher data in which information about the specific site is annotated for an endoscopic image of a section acquired at a timing preceding the specific site for the endoscopic image during the insertion process, when an intraluminal image acquired from an imaging unit of an endoscope is input.

19. The endoscope guide control method according to claim 18, further comprising the steps of: inputting a chief complaint of the subject; and determining information relating to a specific site within the digestive lumen according to the chief complaint information.

20. The endoscope guide control method according to claim 18, further comprising a step of determining information regarding the specific site within the digestive organ lumen according to information regarding a site currently being observed during an examination of the digestive organ lumen.

Images