Endoscopic diagnosis assistance device, endoscopic diagnosis assistance method, and endoscopic diagnosis assistance program

Abstract

This endoscopic diagnosis assistance device is provided with: an image acquisition unit that acquires an image of a lumen from an endoscope; a region determination unit that specifies a passing region of the lumen captured in the image; a feature determination unit that determines a feature of the passing region; and a diagnosis assistance information generation unit that generates assistance information for access or diagnosis of a predetermined passing region in the lumen on the basis of the characteristics of the passing region.

Description

Technical Field

[0001] This invention relates to endoscopic diagnostic aids, endoscopic diagnostic aids, and endoscopic diagnostic aids procedures. Background Technology

[0002] Previously, computer-aided image diagnosis (CAD) was developed to assist surgeons in diagnosis by analyzing endoscopic images. Since individual differences exist among patients in the luminal lumens diagnosed endoscopically, it is desirable to provide diagnostic assistance that takes into account these individual differences. Appropriate diagnostic assistance is particularly desirable for surgeons with limited diagnostic experience. These individual differences in each patient's lumen should be considered not only in diagnostic assistance but also in the process of accessing, screening, and managing the diagnostic site leading to diagnosis. Hereinafter, support for healthcare professionals in this context will be collectively referred to as diagnostic assistance.

[0003] Patent Document 1 describes a diagnostic assistance system that provides information related to the operation of an endoscope for diagnostic aid. The diagnostic assistance system described in Patent Document 1 uses devices such as X-ray CT, cone-beam X-ray CT, MRI-CT, and ultrasound diagnostic devices to capture three-dimensional images of a unit within the body, obtains three-dimensional medical images of the body, generates a virtual endoscopic image reconstructed from the acquired three-dimensional medical images, and outputs operational assistance information related to the operation of the endoscope.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2021-153808 Summary of the Invention

[0007] The problem that the invention aims to solve

[0008] However, the diagnostic aid system described in Patent Document 1 requires obtaining virtual endoscopic images before surgery. This is particularly problematic in situations requiring emergency treatment of a patient, where obtaining virtual endoscopic images before surgery is difficult.

[0009] The present invention was made in view of the following circumstances, and its object is to provide an endoscopic diagnostic aid device, endoscopic diagnostic aid method, and endoscopic diagnostic aid procedure that can provide diagnostic aids that take into account the individual differences in the lumen of each patient without the need for preoperative preparation.

[0010] Methods for solving problems

[0011] To address the aforementioned issues, the present invention proposes the following solutions.

[0012] The endoscopic diagnostic aid device of the first aspect of the present invention includes: an image acquisition unit that acquires an image of a lumen from an endoscope; a region determination unit that determines a passage region of the lumen captured in the image; a feature determination unit that determines the features of the passage region; and a diagnostic aid information generation unit that generates aid information for accessing or diagnosing a predetermined passage region in the lumen based on the features of the passage region.

[0013] Invention Effects

[0014] The endoscopic diagnostic aid device, endoscopic diagnostic aid method, and endoscopic diagnostic aid procedure of the present invention can provide diagnostic aids that take into account the individual differences in the lumen of each patient without the need for preoperative preparation. Attached Figure Description

[0015] Figure 1 This is a diagram illustrating one implementation of an endoscope system.

[0016] Figure 2 This is a functional block diagram of the endoscope system.

[0017] Figure 3 It is a diagram illustrating diagnostic auxiliary information.

[0018] Figure 4 This is a diagram illustrating an example of diagnostic support information for a predetermined area.

[0019] Figure 5 This is a diagram illustrating an example of diagnostic support information for a predetermined area.

[0020] Figure 6 This is a diagram illustrating an example of diagnostic support information for a predetermined area.

[0021] Figure 7 This is a functional block diagram of the region determination unit.

[0022] Figure 8 This is a diagram representing the reasoning model of the diagnostic auxiliary information generation unit.

[0023] Figure 9 This is an example of diagnostic aids that takes into account individual differences in each type of lumen in patients.

[0024] Figure 10 This is an example of diagnostic aids that takes into account individual differences in each type of lumen in patients.

[0025] Figure 11 This is a diagram illustrating the training data used in the learning of this inference model.

[0026] Figure 12 This is a diagram showing an example of a composite image generated by the display control unit.

[0027] Figure 13 This is a diagram illustrating an example of the composite image generated by the display control unit.

[0028] Figure 14 This is the control flowchart of the endoscope system. Detailed Implementation

[0029] Reference Figures 1 to 14 An endoscope system 500 according to one embodiment of the present invention will be described.

[0030] [Endoscopic System 500]

[0031] Figure 1 This is a diagram representing the endoscopic system 500.

[0032] The endoscope system 500 includes an endoscope 100, an image processing unit 200, a light source unit 300, and a display unit 400. The image processing unit 200 and the light source unit 300 can also be integrated into one device (image control device).

[0033] The light source device 300 has a light source 310 such as an LED, and controls the light source to control the amount of illumination light transmitted to the endoscope 100 via the light guide 161.

[0034] Display device 400 is a device that displays images generated by image processing processor device 200, various information related to endoscope system 500, etc. Display device 400 is, for example, a liquid crystal monitor or a head-mounted display.

[0035] [Endoscope 100]

[0036] Endoscope 100 is, for example, a device for observing and manipulating the body of a patient lying on an operating table T. Endoscope 100 includes an elongated insertion part 110 that is inserted into the patient's body, an operating part 180 connected to the base of the insertion part 110, and a universal cable 190 extending from the operating part 180.

[0037] The insertion part 110 has a front end portion 120, a flexible bending portion 130, and a long, flexible tube portion 140. The front end portion 120, the bending portion 130, and the flexible tube portion 140 are connected sequentially from the front end side. The flexible tube portion 140 is connected to the operation part 180.

[0038] Figure 2 This is a functional block diagram of the endoscopic system 500.

[0039] The front end 120 has a camera unit 150, a lighting unit 160, and an information acquisition unit 170.

[0040] The camera unit 150 includes an optical system, an imaging element that converts light signals into electrical signals, and an analog-to-digital converter circuit that converts the analog signals output by the imaging element into digital signals. The camera unit 150 captures images of a subject and generates an image signal. The image signal is acquired by the image processing processor device 200 via the image signal cable 151.

[0041] The illumination unit 160 illuminates the subject with illumination light transmitted through the light guide 161. The light guide 161 passes through the insertion unit 110, the operation unit 180, and the universal cable 190 and is connected to the light source device 300. In addition, the illumination unit 160 may also have light sources such as LEDs, optical elements such as phosphors with wavelength conversion functions, etc.

[0042] The information acquisition unit 170 detects the position, speed, and direction of the front end portion 120. The information acquisition unit 170 is, for example, a 6-axis sensor or a 3-axis sensor. The output of the information acquisition unit 170 is acquired by the image processing processor device 200 via the signal cable 171. Furthermore, the information acquisition unit 170 detects the state and position of the front end portion 120, and can detect the direction of gravity based on acceleration, or detect the movement of the front end portion 120 based on its changes. The information acquisition unit 170 can be any unit that achieves the same purpose; it can be a magnetic sensor, a component that generates magnetism, or a component that detects motion and magnetism in cooperation with external sensors or systems. Furthermore, since the movement of the front end portion 120 can be detected by changes in the image output acquired by the imaging unit 150, the imaging unit 150 can be used instead of the information acquisition unit 170. Alternatively, the information acquisition unit 170 can be implemented by the cooperation of the aforementioned units.

[0043] The operation unit 180 handles operations on the endoscope 100. The operation unit 180 includes an angle control knob 181 for the bending section 130, an air / water supply button 182, a suction button 183, and a release button 184. Operations input to the air / water supply button 182, suction button 183, and release button 184 are received by the image processing unit 200. The release button 184 is used to input operations for saving the image captured by the camera unit 150.

[0044] The universal cable 190 connects the endoscope 100 and the image processing processor device 200. The universal cable 190 is a cable through which the camera signal cable 151, the optical fiber 161, and the signal cable 171 are inserted.

[0045] [Image processing processor device 200]

[0046] like Figure 2As shown, the image processing processor device 200 includes an information acquisition unit 210, an image acquisition unit 220, an image recording unit 230, an endoscopy diagnostic assistance unit 240, and a display control unit 290.

[0047] The image processing processor device 200 is a computer equipped with a processor such as a CPU, memory, and a recording unit, capable of executing programs. The functions of the image processing processor device 200 are implemented by the processor executing programs. At least some of the functions of the image processing processor device 200 can also be implemented using dedicated logic circuits mounted on an ASIC or FPGA.

[0048] The image processing processor device 200 may also have structures other than a processor, memory, and recording unit. For example, the image processing processor device 200 may also have part or all of an image processing unit that performs image processing and image recognition processing. By also having an image processing unit, the image processing processor device 200 can perform specific image processing and image recognition processing at high speed. The image processing unit may also be a processor installed on a cloud server connected via the Internet.

[0049] The recording unit is a non-volatile recording medium that stores the aforementioned program and the data required to execute the program. The recording unit may be composed of, for example, writable non-volatile memory such as floppy disks, optical disks, ROM, and flash memory; removable media such as CD-ROMs; or storage devices such as hard disks and SSDs built into a computer system. The recording unit may also be a storage device located on a cloud server connected via the Internet.

[0050] The aforementioned program can also be provided, for example, by a "computer-readable recording medium" such as flash memory. The program can also be transferred from the computer holding the program to the memory or recording unit via a transmission medium or by transmission waves within the transmission medium. The "transmission medium" for transmitting the program is a medium capable of transmitting information. Media capable of transmitting information include networks (communication networks) such as the Internet and communication lines (communication lines) such as telephone lines. The aforementioned program can also implement a portion of the aforementioned functions. Furthermore, the aforementioned program can also be a differential file (differential program). The aforementioned functions can also be implemented through a combination of a program already recorded on the computer and a differential program.

[0051] The information acquisition unit 210 controls the overall image processing processor device 200. Furthermore, the information acquisition unit 210 acquires information about cases using the endoscope system 500 (type of endoscope 100, patient information, and surgeon information) from hospital systems or the like. Additionally, the information acquisition unit 210 can acquire case information by having the operator or assistant input it through an input device (not shown). The information input to the information acquisition unit 210 is acquired by the endoscopy diagnostic assistance unit 240.

[0052] The image acquisition unit 220 acquires a camera signal from the camera unit 150 of the endoscope 100 via the camera signal cable 151. The image acquisition unit 220 performs camera signal processing on the camera signal acquired from the camera unit 150, and sequentially acquires camera images D. The image acquisition unit 220 outputs the acquired camera image D to the display control unit 290. Furthermore, the image acquisition unit 220 outputs the acquired camera image D to the endoscope diagnostic assistance unit 240 via the image recording unit 230.

[0053] The image recording unit 230 is part of the aforementioned recording unit and is a non-volatile recording medium. The image recording unit 230 is also part of the aforementioned memory and can be a volatile recording medium. The image recording unit 230 records multiple transmitted camera images D.

[0054] Multiple video images D (image frames, time-series images) input sequentially in a time series are recorded in the image recording unit 230. When the recording capacity of the image recording unit 230 is insufficient, the oldest video image D is deleted. The multiple video images D recorded in the image recording unit 230 can be video images D of consecutive frames, or video images D obtained by removing multiple frames at intervals from consecutive frames. By temporarily recording each frame (frame) obtained from the imaging unit 150 in the image recording unit 230 and comparing the previous and subsequent frames, the same information as the information acquisition unit 170 can be obtained. By comparing the images of the previous and subsequent frames, if the image expands from the center of the screen to the periphery, it can be determined that the front end 120 has been inserted along the lumen; if there is a change in the image from the periphery toward the center of the screen, it can be determined that the front end 120 has been pulled out.

[0055] Figure 3 It is a diagram illustrating diagnostic auxiliary information.

[0056] For example, such as Figure 3 As shown, the large intestine is composed of multiple interconnected tubular tubes, and the shape and size of each patient (subject) P vary individually. Therefore, the time required to insert / withdraw the endoscope 100 forward in the direction of travel while bending the insertion portion 110 along the chain of tubular tubes varies depending on the individual differences in the large intestine of each patient P and the skill of the surgeon S. On the other hand, due to factors such as examination time and effective anesthesia time, it is difficult to prolong the examination and treatment time. Therefore, the endoscopic diagnostic assistance unit 240 provides diagnostic assistance for the predetermined passage area AP, taking into account the individual differences in the large intestine of each patient P, to assist the surgeon S. Furthermore, regarding the diagnostic assistance for the predetermined passage area AP, it is preferable to modify the content and level of detail according to the skill of the surgeon S.

[0057] The endoscopic diagnostic assistance unit 240 can also provide diagnostic assistance for the predetermined passage area AP in lumens outside the large intestine. For example, the endoscopic diagnostic assistance unit 240 provides diagnostic assistance for the predetermined passage area AP that takes into account the individual differences of each patient P's stomach, in order to assist the surgeon S.

[0058] Specifically, the endoscopic diagnostic assistance unit 240 determines the characteristics of the passage region AC of the lumen captured in the camera image D. Then, based on the characteristics of the passage region AC, the endoscopic diagnostic assistance unit 240 generates diagnostic assistance information for a predetermined passage region AP, taking into account the individual differences in the lumen of each patient P. Here, since there are correlations between different regions of the lumen, it is possible to predict the characteristics of the predetermined passage region AP based on the characteristics of the passage region AC, taking into account the individual differences in the lumen of each patient P.

[0059] The endoscopic diagnostic assistance unit 240, as needed, considers the patient information (gender, age, race, body type, examination results, etc.) obtained by the information acquisition unit 210 and generates diagnostic assistance information for the predetermined passage area AP. For example, "relative to the diameter of the large intestine, the transverse colon and sigmoid colon are positively correlated with height and weight, and negatively correlated with age," "transverse colon ptosis is correlated with gender, age, height, weight, obesity, and the length of the transverse colon and large intestine," and "sigmoid colon elevation is correlated with age, the length of the sigmoid colon, and the length of the large intestine." Therefore, by considering the patient information obtained by the information acquisition unit 210, the endoscopic diagnostic assistance unit 240 can specify the correlations in each region of the lumen for each category (attribute) of the patient P, and can more accurately predict the characteristics of the diagnostic assistance information for the predetermined passage area AP.

[0060] The planned passage area AP is the area located ahead of the endoscope 100 in the direction of travel, compared to the passage area AC. For example... Figure 3 As shown, when the insertion portion 110 of the endoscope 100 is inserted into the lumen, the predetermined passage area AP is a region in the lumen that is further inward than the intermediate passage area AC. When the insertion portion 110 of the endoscope 100 is withdrawn from the lumen, the predetermined passage area AP is a region in the lumen that is further inward than the intermediate passage area AC towards the natural opening.

[0061] Figure 4 This is a diagram illustrating an example of diagnostic support information for a pre-defined area AP.

[0062] like Figure 4 As shown, when the surgeon S inserts the insertion part 110 of the endoscope 100 into the large intestine, if the passage area AC is the "rectum", for example, the endoscope diagnostic assistance unit 240 generates diagnostic assistance information for the "sigmoid colon" which is the predetermined passage area AP. Figure 4The illustrated diagnostic aid information is the predicted time required for surgeon S to pass through the area AP as scheduled.

[0063] Figure 5 and Figure 6 This is a diagram illustrating an example of diagnostic support information for a pre-defined area AP.

[0064] like Figure 5 As shown, after obtaining the examination results such as the location of the pain of patient P in advance, the endoscopic diagnostic assistance unit 240 generates diagnostic assistance information for candidate areas with lesions (also known as "specific object areas, specific locations"). Figure 6 The illustrated diagnostic aids inform the surgeon S that the candidate area for the lesion is the lower part of the descending colon or the right part of the sigmoid colon, prompting the surgeon S to observe carefully.

[0065] The primary symptom that a patient reports to a doctor is called the chief complaint. Figure 5 In the example shown, the chief complaint refers to the symptom of "pain in this area" in the flank. The endoscopic diagnostic aid unit 240 can provide diagnostic aid information that enables an appropriate diagnosis of such a chief complaint. For example, it might also indicate that the patient is concerned about this area, but here, the patient explicitly states abdominal pain, which is related to the endoscopy, and therefore serves as the chief complaint related to the examination. Even if the patient or examinee only touches this area with their hand, the physician can anatomically infer that it is located within the lumen of the digestive organ during the endoscopy. The physician can input such information obtained from the chief complaint (also called "chief complaint information") as part of the patient information (described later). Figure 14 (S110).

[0066] Additionally, the endoscopic diagnostic assistance unit 240 can also read the results written by the patient with checkmarks on a piece of paper to obtain the chief complaint information. This paper has a display shaped like a human body. The endoscopic diagnostic assistance unit 240 can also display a UI display shaped like a human body on a terminal, read the checkmarks input thereon, and obtain the chief complaint information. The endoscopic diagnostic assistance unit 240 may also have a program and a database that determine, more specifically, which organ and part of the body the patient is referring to based on the input results.

[0067] like Figure 2 As shown, the endoscopic diagnostic assistance unit 240 includes a region determination unit 250, a feature determination unit 260, and a diagnostic assistance information generation unit 270.

[0068] The endoscopic diagnostic aid unit 240 may be a device separate from the image processing processor device 200 (hereinafter also referred to as "endoscopic diagnostic aid device"). The endoscopic diagnostic aid device may also be a computing device installed on a cloud server connected via the Internet. The computing device installed on the cloud server may also have an image acquisition unit, equipped with a function to determine the position of the endoscope tip based on the acquired image, and a function to determine the characteristics of the area currently traversed by the endoscope tip 120 (characteristics of the area traversed by the endoscope tip) based on information obtained from the tip 120. Depending on the system, the image recording unit may also be installed on the computing device. Furthermore, in... Figure 2 Images are acquired via the image recording unit 230, but sometimes it is not necessary to acquire images via the image recording unit 230. The region determination unit 250 and the feature determination unit 260 acquire intraluminal feature information obtained during insertion into the lumen of the digestive organ during endoscopic examination, and therefore can also be called intraluminal feature information acquisition units.

[0069] Figure 7 This is a functional block diagram of the region determination unit 250.

[0070] The region determination unit 250 determines the region AC of the lumen captured in the camera image D. When the insertion part 110 of the endoscope 100 is inserted into the large intestine, the region determination unit 250 analyzes the obtained image to determine which of the following is the region of the large intestine contained in the camera image D: cecum, ascending colon, transverse colon, descending colon, sigmoid colon, rectosigmoid part, etc.

[0071] Figure 7 The illustrated region determination unit 250 has multiple region determination units (first region determination unit 251, second region determination unit 252, and nth region determination unit 25n). Each region determination unit is dedicated to a specific region of the lumen. For example, in the case where the lumen is the large intestine, the first region determination unit 251 determines that the region AC passing through the lumen in the photographed image D is the "rectum". Similarly, the second region determination unit 252 determines that the region AC passing through the lumen in the photographed image D is the "sigmoid colon".

[0072] Each region determination unit can also determine the passage region AC of the lumen contained in the camera image D through pattern matching. For example, each region determination unit compares the pre-recorded images of each part with the camera image D, and determines the passage region AC of the lumen contained in the camera image D based on the pre-recorded similarity for each part.

[0073] Each region determination unit can also determine the passing region AC of the lumen contained in the camera image D using a machine learning model. For example, each region determination unit determines the passing region AC of the lumen contained in the camera image D using a machine learning model that has been pre-learned to detect regions of the lumen contained in the camera image D.

[0074] Each region determination unit can determine the passage region AC of the lumen contained in the camera image D based on the output of the information acquisition unit 170 (speed, direction, attitude, etc. of the front end 120).

[0075] The region determination unit 250 may also determine the multiple through regions AC of the lumen captured in the camera image D by using only a region determination unit that can distinguish and determine multiple regions of the lumen, instead of using multiple region determination units.

[0076] The region determination unit 250 can also combine the above methods to determine the passage region AC of the lumen contained in the camera image D.

[0077] The region determination unit 250 sends the region of the lumen contained in the determined camera image D to the diagnostic auxiliary information generation unit 270.

[0078] The feature determination unit 260 determines the features of the intermediate region AC based on the obtained image signal. The features determined by the feature determination unit 260 are, for example, shape (intraluminal length, inner diameter, etc.) and surface condition (wall condition, blood vessel condition, etc.). Intraluminal length refers to the length in the direction of the lumen's extension.

[0079] The feature determination unit 260 can also determine the features of the central region AC by image processing of the camera image D.

[0080] The feature determination unit 260 can also determine the features of the passage through the middle region AC based on the output of the information acquisition unit 170 (speed, direction, attitude, etc. of the front end 120). For example, the feature determination unit 260 can determine the length of the tube passing through the middle region AC based on the speed and direction of the front end 120.

[0081] The feature determination unit 260 can also determine the characteristics of the intermediate region AC based on the operation history input to the operation unit 180. For example, if multiple water delivery operations have been performed on the intermediate region AC, the feature determination unit 260 can determine that the intermediate region AC is any region among the regions where residue is prone to accumulate.

[0082] The feature determination unit 260 can also combine the above-mentioned methods to determine the features of the middle region AC.

[0083] The feature determination unit 260 sends the region of the lumen contained in the determined camera image D to the diagnostic auxiliary information generation unit 270.

[0084] The diagnostic assistance information generation unit 270 generates diagnostic assistance information for a predetermined passage area AP based on the camera image D, the determination result of the passage area AC, and its features. The diagnostic assistance information generation unit 270 can generate diagnostic assistance information based on rules or through the reasoning model 281 provided by the reasoning unit 280. The diagnostic assistance information generation unit 270 can also function as a guidance information unit for accessing a specific site prior to diagnosis; therefore, it can also be referred to as a guidance information output unit. This guidance information output unit outputs guidance information for further insertion of the endoscope 100's tip 120 into the passage area to access a specific target site based on the aforementioned intraluminal feature information.

[0085] The inference unit 280 can also use conventional general-purpose computing circuits such as CPUs and FPGAs (Field Programmable Gate Arrays). However, since neural network processing mostly involves matrix multiplication, it sometimes utilizes devices specifically designed for matrix computation, known as GPUs (Graphics Processing Units) or TPUs (Tensor Processing Units). In recent years, such artificial intelligence (AI) dedicated hardware, the "Neural Network Processing Unit (NPU)," has been designed to be integrated and assembled with other circuits such as CPUs, sometimes becoming part of the processing circuitry.

[0086] Figure 8 It is a diagram representing the inference model 281 and the images during its learning process.

[0087] Inference model 281 is a model learned using annotated training data, which relates the annotations to diagnostic aid information for a predetermined region AP across multiple image frames (learning camera images) for multiple cases. Inference model 281 is, for example, a neural network, learned through deep learning. Furthermore, inference model 281 is not limited to neural networks, but can also be other machine learning models capable of outputting information from input images.

[0088] The input to inference model 281 consists of multiple camera images D (image frames, time-series images) input in a time-series order, as well as the determination results and features of the intermediate region AC. The output of inference model 281 is diagnostic auxiliary information for the predetermined region AP.

[0089] The input to the inference model 281 may include patient information and surgeon information acquired by the information acquisition unit 210. For example, by using patient information of patient P (gender, age, race, body type, examination results, etc.) as input to the inference model 281, the inference model 281 can easily output diagnostic auxiliary information that takes into account the individual differences in the lumen for each category (attribute) of patient P. For example, by using the surgeon's skill (proficiency) as input to the inference model 281 as surgeon information, the inference model 281 can easily output diagnostic auxiliary information corresponding to the surgeon's skill.

[0090] Thus, by learning the inference model 281 using training data, an endoscope guidance control method with a display control unit 290 can be provided. This training data is obtained by annotating the image of a first interval obtained during insertion into the lumen of a digestive organ under endoscopy with feature information of a second interval inserted at a time after the first interval. The display control unit 290 displays the feature information of the second interval inferred by the inference model 281 when an image corresponding to the first interval of the lumen image obtained from the camera unit 150 of the endoscope 100 is input to the inference model 281. Furthermore, when the diagnostic assistance information generation unit 270 is presented as a "guidance information output unit," it can also be presented as having an inference model obtained by learning using training data. This training data is obtained by annotating the image of a first interval obtained during the passage of the endoscope's tip with feature information of the second interval reached by the endoscope's tip 120 further inserted from the passage area. Since it is unknown what lies ahead of the endoscope 100 during insertion, a structure for predicting this is important. Therefore, although it is described that the first section is closer to the insertion position and the second section is further inside, it can also be the other way around. This approach can also be used when withdrawing the endoscope 100. This is because the lumen is flexible and sometimes takes on different shapes during insertion and withdrawal.

[0091] Furthermore, when there is a patient's subjective complaint regarding a point of interest during an endoscopy, there is a desire to observe that area well. Therefore, if the doctor can be given some guidance before the camera unit 150 at the tip 120 of the endoscope 100 moves to that area at a certain time before observing that specific area, the doctor can be mentally prepared. That is, if endoscopic images are obtained during the insertion of the endoscope into the lumen of the digestive organs during the patient's digestive organ endoscopy, and if a specific area is determined based on the aforementioned subjective complaint, a reasoning model 281 can be obtained by learning using training data. This training data is obtained by annotating endoscopic images of the insertion process with information related to the specific area in the interval of endoscopic images obtained at a time prior to the specific area. Therefore, by using an endoscopic guidance control method with a display control step, an accurate examination can be performed without missing points of interest to the patient. This display control step displays information related to the specific location inferred by the inference model 281 when an intraluminal image obtained from the camera unit 150 of the endoscope 100 is input.

[0092] Figure 9 This is an example of diagnostic aid information that takes into account individual differences in the lumen of each category of patient P. When the surgeon S inserts the insertion part 110 of the endoscope 100 into the large intestine, when the middle region AC is "rectum", diagnostic aid information is output for "sigmoid colon", "descending colon" and "transverse colon" as the predetermined passage region AP, according to each category of patient P. Figure 9 The diagnostic aids illustrated are the average predicted time required for the surgeon to examine (through) the pre-planned area of ​​access (AP). The predicted time required for examination is calculated, for example, based on the estimated intraluminal length of the pre-planned AP. Diagnostic aids can also be the difficulty level of each pre-planned AP.

[0093] Figure 10 This is an example of diagnostic aid information that takes into account individual differences in the lumen of each category of patient P. When the surgeon S inserts the insertion part 110 of the endoscope 100 into the large intestine, when passing through the middle region AC as "sigmoid colon", diagnostic aid information is output for "descending colon", "transverse colon" and "ascending colon" as the predetermined passage region AP, according to each category of patient P. Figure 10 The diagnostic aids illustrated are the average predicted time required for the surgeon to examine (through) the pre-planned area of ​​access (AP). The predicted time required for examination is calculated, for example, based on the estimated intraluminal length of the pre-planned AP. Diagnostic aids can also be the difficulty level of each pre-planned AP.

[0094] The input to inference model 281 can also include the determination results and characteristics of the traversed regions AD. The traversed regions AD are areas located posterior to the endoscopic region AC in the direction of travel of the endoscope 100, and are areas already traversed by the tip 120 of the endoscope 100. For example, when the surgeon S inserts the insertion portion 110 of the endoscope 100 into the large intestine, if the traversed region AC is "sigmoid colon," one of the traversed regions AD is "rectum." By using the determination results and characteristics of the traversed regions AD as input to inference model 281, inference model 281 can more easily and accurately output diagnostic auxiliary information for the predetermined traversed region AP. As the examination and diagnosis proceed, the number of traversed regions AD increases, thus the diagnostic auxiliary information for the predetermined traversed region AP becomes more accurate.

[0095] Figure 11 This is a graph illustrating the training data.

[0096] As training data, multiple image frames (a sequence of static images) obtained from endoscopic examinations of multiple cases are used. The training data is composed of multiple image frames (learning images) and annotations associated with diagnostic auxiliary information for a predetermined region AP. The inference model 281 is a model that learns to output corresponding annotations for the input image frames (learning images) using the training data.

[0097] The annotations for the training data can also include rescue strategies. A rescue strategy could be something like, "Since the difficulty of passing the regional AP is very high, we should request support from experienced or expert personnel." When rescue strategies are included in the annotations for the training data, the inference model 281 can further output rescue strategies.

[0098] Furthermore, the diagnostic assistance information generation unit 270 can also generate diagnostic assistance information for the predetermined passage region AP based solely on the camera image D. In this case, the inference model 281 becomes a model that outputs diagnostic assistance information for the predetermined passage region AP based solely on the passage region AC of the lumen captured in the camera image D.

[0099] Furthermore, the diagnostic assistance information generation unit 270 can also generate diagnostic assistance information for a predetermined passage area AP based solely on the determination results and characteristics of the passage area AC.

[0100] Figure 12 and Figure 13 This is a diagram representing an example of a composite image S.

[0101] The display control unit (image synthesis unit) 290 generates a composite image S including a camera image D and diagnostic auxiliary information E.

[0102] Figure 12The composite image S shown was generated when the surgeon S inserted the insertion part 110 of the endoscope 100 into the large intestine (inward and outward direction: insertion). Figure 12 The illustrated diagnostic aid information E includes the patient P's category (attribute), the overall difficulty of the area AP to be passed, especially the "transverse colon" as a high-difficulty area, and rescue strategies.

[0103] Figure 13 The composite image S shown was generated when the surgeon S was performing an examination while pulling the insertion part 110 of the endoscope 100 out of the large intestine (inward and outward direction: pulling out). Figure 13 The illustrated diagnostic aid information E includes the patient P’s category (attribute) and a suggestion of a specific object region (e.g., a candidate region with lesions that corresponds to the location where the patient P complains of pain).

[0104] [Operation of Endoscopic System 500]

[0105] Next, the operation (diagnostic assistance method) of the endoscope system 500 will be explained. Specifically, the surgical procedure of observing and manipulating the wall of the large intestine using the endoscope system 500 will be explained. In addition, the endoscope system 500 can also provide diagnostic assistance for organs other than the large intestine (stomach, bronchi, urinary tract, etc.). The following will be conducted according to... Figure 14 The control flow diagram of the endoscope system 500 shown is used for explanation.

[0106] <Step S110>

[0107] In step S110, the information acquisition unit 210 acquires information about the case (type of endoscope 100, patient information, surgeon information). Chief complaint information can be input in step S110. The information acquisition unit 210 can also, based on the input chief complaint information, determine which area should be carefully examined during subsequent examinations by referring to a specific procedure and database. The information acquisition unit 210 can acquire information about the patient's condition by having the operator or assistant input it from an input device (not shown). The endoscope system 500 then executes step S120.

[0108] <Step S120>

[0109] In step S120, the region determination unit 250 detects the insertion direction of the endoscope 100. Specifically, the region determination unit 250 acquires the output of the information acquisition unit 170 and detects the forward and backward movement (insertion direction, withdrawal direction) of the tip 120 of the endoscope 100 based on the output of the information acquisition unit 170. Alternatively, the region determination unit 250 can also determine the region of the lumen captured in the image D based on the image D, and detect the forward and backward movement of the endoscope 100 based on the history of the determined region. If the insertion direction of the endoscope 100 is the insertion direction, the endoscope system 500 then executes step S130. If the insertion direction of the endoscope is the withdrawal direction, the endoscope system 500 then executes step S200.

[0110] <Step S130>

[0111] In step S130, the region determination unit 250 determines the passage region AC of the lumen captured in the camera image D. Next, the endoscope system 500 executes step S140.

[0112] <Step S140>

[0113] In step S140, the region determination unit 250 determines whether the tip 120 of the endoscope 100 has been inserted into a specific region such as the cecum. If the tip 120 of the endoscope 100 has been inserted into the specific region, the endoscope system 500 executes step S120. If the tip 120 of the endoscope 100 has not been inserted into the specific region, the endoscope system 500 executes step S150.

[0114] <Step S150>

[0115] In step S150, the feature determination unit 260 determines the features passing through the central region AC. Based on the image information obtained at this time, image data information is retrieved through the inference model 281, other databases, etc., and this image data information is used to output information about the area to be examined periodically in front of it. Next, the endoscope system 500 executes step S160.

[0116] <Step S160>

[0117] In step S160, the diagnostic assistance information generation unit 270 generates diagnostic assistance information for a predetermined passage region AP based on the image D and the determination results and features of the passage region AC. When an intraluminal image obtained from the camera unit 150 of the endoscope 100 is input, diagnostic assistance information inferred by the inference model 281 can be deduced (this information may not be the diagnosis itself, but includes various information to enable diagnosis. It may be assistance information for access, or it may include information on various operations such as water injection, air delivery, suction, or switching of light sources, switching of image processing, and dispensing of reagents). Thus, this becomes guidance information for predicting what will happen next when an intraluminal image obtained from the camera unit 150 of the endoscope 100 is input. For example, an endoscopy guidance control method could be provided, which includes a display control step in which an inference model 281 is used to infer and display information about a specific location inferred when an endoscopic image is input to it (in addition to information related to ease of access, relationship to the patient's complaints, and observational and diagnostic know-how). The information about the specific location within the lumen of the digestive organ can be determined based on information obtained from the patient's complaints, but it can also be... Figure 8 As recorded in the document, "Determination result of passing through the central region AC," the determination is made based on information about the location (passing through) during the current observation in the endoscopic examination of the digestive organ lumen. Next, the endoscope system 500 executes step S170.

[0118] <Step S170>

[0119] In step S170, the display control unit 290 generates a composite image S including the camera image D and diagnostic auxiliary information E, and outputs the composite image S to the display device 400. The display device 400 displays the composite image S. Next, the endoscope system 500 executes step S120. The doctor is able to obtain information before reaching the examination site, thus avoiding difficulties in predicting unforeseen situations due to sudden insertion, or missing points of concern to the patient, allowing for psychological preparation and reliable and accurate access and examination.

[0120] Furthermore, the display control unit 290 can also notify the user of diagnostic assistance information in advance, based on the content of the generated diagnostic assistance information. For example, if the difficulty of passing the regional AP is expected to be very high considering the surgeon S's skill level, the diagnostic assistance information can be notified to the surgeon S in advance.

[0121] <Step S200>

[0122] In step S200, the region determination unit 250 determines the passage region AC of the lumen captured in the camera image D. Next, the endoscope system 500 executes step S210.

[0123] <Step S210>

[0124] In step S210, the region determination unit 250 determines whether the region AC is a specific target region (e.g., a candidate region with a lesion corresponding to the area where patient P reports pain). If the region AC is a specific target region, the endoscope system 500 then proceeds to step S220. If the region AC is not a specific target region, the endoscope system 500 then proceeds to step S230.

[0125] <Step S220>

[0126] In step S220, the diagnostic assistance information generation unit 270 displays detailed diagnostic assistance information from the examination. Specifically, such as... Figure 6 As shown, the surgeon S is notified that the area AC or the predetermined area AP is a candidate area where a lesion exists. Furthermore, the diagnostic auxiliary information generation unit 270 performs the same processing as steps S150 to S170 to generate a composite image S including the camera image D and diagnostic auxiliary information E, and outputs the composite image S to the display device 400. Next, the endoscope system 500 performs step S240.

[0127] <Step S230>

[0128] In step S230, the diagnostic assistance information generation unit 270 displays typical diagnostic assistance information during the examination. Specifically, the diagnostic assistance information generation unit 270 performs the same processing as steps S150 to S170, generates a composite image S including the camera image D and diagnostic assistance information E, and outputs the composite image S to the display device 400. Next, the endoscope system 500 executes step S240.

[0129] <Step S240>

[0130] In step S240, the endoscopic diagnostic assistance unit 240 determines whether the surgery has ended. If it determines that the surgery has not ended, the endoscopic diagnostic assistance unit 240 executes step S120 and subsequent steps. If it determines that the surgery has ended, the endoscopic diagnostic assistance unit 240 executes step S300 to terminate the procedure. Figure 14 The control flow is shown.

[0131] According to the endoscope system 500 of this embodiment, the endoscope diagnostic assistance unit 240 (endoscope diagnostic assistance device) can provide diagnostic assistance that takes into account the individual differences in the lumen of each patient P without the need for preoperative preparation.

[0132] The present invention has been described in detail above with reference to the accompanying drawings, but the specific structure is not limited to this embodiment, and design changes are also included without departing from the spirit of the invention. Furthermore, the constituent elements shown in the above embodiment and the variations shown below can be appropriately combined to form a configuration.

[0133] in particular Figure 2 The endoscope system 500 shown comprises multiple devices such as an endoscope 100, an image processing unit 200, and a light source unit 300. However, it is also possible to have a structure where the image processing unit 200 and the light source unit 300 are integrated into the endoscope 100. Furthermore, the image processing unit 200 does not necessarily need to include an information acquisition unit 210, an endoscope diagnostic assistance unit 240, a display control unit 290, and an image recording unit 230. From the viewpoint of system scalability and versatility, it can cooperate with other processor devices to achieve the same function. That is, the image acquisition unit that acquires an image of the lumen from the endoscope 100, the region determination unit that determines the passage region of the lumen, the feature determination unit that determines the characteristics of the passage region, and the diagnostic assistance information generation unit that generates assistance information for passage or diagnosis of a predetermined passage region in the lumen based on the characteristics of the passage region do not necessarily need to be included in the same device. As long as the devices cooperate appropriately to form an endoscope diagnostic assistance system, it is acceptable.

[0134] In the above embodiments, the endoscopic diagnostic assistance unit (endoscopic diagnostic assistance device) provides diagnostic assistance for images from medical endoscopes. However, the diagnostic objects of the endoscopic diagnostic assistance unit (endoscopic diagnostic assistance device) are not limited to images from medical endoscopes. The endoscopic diagnostic assistance unit (endoscopic diagnostic assistance device) can also provide diagnostic assistance for images acquired from other imaging devices such as cameras, video cameras, industrial endoscopes, microscopes, robots with image acquisition capabilities, smartphones, mobile phones, smartwatches, tablet terminals, and laptop PCs.

[0135] Industrial utilization potential

[0136] This invention can be applied to endoscope systems, etc.

[0137] Explanation of reference numerals in the attached figures

[0138] 500 Endoscopic System

[0139] 400 display device

[0140] 300 light source device

[0141] 100 Endoscopes

[0142] 110 Insertion Section

[0143] 120 front end

[0144] 200 image processing processor device

[0145] 210 Information Acquisition Department

[0146] 220 Image Acquisition Unit

[0147] 230 Image Recording Department

[0148] 240 Endoscopic Diagnostic Auxiliary Department

[0149] 250 Area Determination Department

[0150] 260 Feature Determination Unit

[0151] 270 Diagnostic Auxiliary Information Generation Department

[0152] 280 Reasoning Department

[0153] 281 Inference Model

[0154] 290 Display Control Unit (Image Composition Unit)

[0155] AC passes through the middle area

[0156] AD has passed the region

[0157] AP scheduled through the area

[0158] D camera images

[0159] E-diagnostic auxiliary information

Claims

1. An endoscopic diagnostic auxiliary device, comprising: The image acquisition unit acquires images of the lumen from the endoscope; A region determination unit that determines the region through which the lumen is captured in the image; The feature determination unit determines the features of the region passed through. as well as A diagnostic assistance information generation unit generates assistance information for accessing or diagnosing a predetermined passage area in the lumen based on the characteristics of the passage area.

2. The endoscopic diagnostic auxiliary device according to claim 1, wherein, The feature includes the intraluminal length of the lumen in the direction of its extension.

3. The endoscopic diagnostic auxiliary device according to claim 1, wherein, The diagnostic auxiliary information generation unit generates the auxiliary information based on the characteristics of the already traversed area in the lumen, which is located behind the endoscopic travel direction of the passage area compared to the traversed area.

4. The endoscopic diagnostic auxiliary device according to claim 1, wherein, When the area being traversed is a specific object area, the diagnostic assistance information generation unit generates more detailed assistance information.

5. The endoscopic diagnostic auxiliary device according to claim 4, wherein, The specific target area is the area corresponding to the location of the patient's pain.

6. (Diagnostic auxiliary information: predicted time required for examination) According to claim 2, the endoscopic diagnostic auxiliary device, wherein, The auxiliary information is the predicted time required for inspection of the predetermined passage area, calculated based on the length of the lumen within the predetermined passage area.

7. The endoscopic diagnostic auxiliary device according to claim 1, wherein, The diagnostic auxiliary information generation unit generates auxiliary information for the predetermined passage region based on a reasoning model obtained by learning the relationship between the features of the passing region and the features of the predetermined passage region in the lumen.

8. The endoscopic diagnostic auxiliary device according to claim 7, wherein, The inference model is obtained by learning the relationship between the features of the already traversed region and the features of the predetermined traversed region, which is located in the lumen behind the endoscopic travel direction of the endoscope compared to the traversed region. The diagnostic auxiliary information generation unit generates auxiliary information for the predetermined passing area based on the inference model and the characteristics of the already passed area.

9. The endoscopic diagnostic auxiliary device according to claim 1, wherein, The diagnostic auxiliary information generation unit notifies the user in advance of the auxiliary information based on the content of the generated auxiliary information.

10. An endoscopic diagnostic auxiliary device, wherein, This endoscopic diagnostic aid device has a reasoning model, which is learned by training data on features of a predetermined passage region obtained during the insertion of the endoscope into the lumen, using image annotations of the passage region. The predetermined passage region is located in the lumen ahead of the endoscope's direction of travel, compared to the passage region. The inference model infers the features of the predetermined passage area when it is given an image of the passage area obtained from the endoscope.

11. An endoscopic diagnostic aid method, wherein, Images of the lumen obtained from an endoscope Determine the passage region of the lumen captured in the image. Determine the characteristics of the region through which the passage passes. Based on the characteristics of the passage area, auxiliary information for accessing or diagnosing predetermined passage areas in the lumen is generated.

12. An endoscopic diagnostic aid program that enables a computer to perform the following processing: Images of the lumen obtained from an endoscope Determine the passage region of the lumen captured in the image. Determine the characteristics of the region through which the passage passes. Based on the characteristics of the passage area, auxiliary information for accessing or diagnosing predetermined passage areas in the lumen is generated.

13. An endoscopic diagnostic auxiliary system, comprising: The image acquisition unit acquires images of the lumen from the endoscope; A region determination unit that determines the region through which the lumen is captured in the image; The feature determination unit determines the features of the region passed through. as well as The diagnostic auxiliary information generation unit generates auxiliary information for passage or diagnosis of a predetermined passage area in the lumen based on the characteristics of the passage area.

14. An endoscope-guided control method comprising the following steps: The steps of obtaining endoscopic images during insertion or withdrawal of an endoscope into the lumen of a digestive organ; and The display control steps involve displaying feature information of the second interval inferred by the inference model when an image corresponding to the first interval of the intraluminal image obtained from the endoscope's camera is input. The inference model is obtained by learning using training data, which is obtained by annotating the endoscope image of the first interval during the insertion or removal process with feature information of the second interval that passes through at a later time than the first interval.

15. The endoscopic guidance and control method according to claim 14, wherein, The display control step also includes the following steps: An image obtained from the endoscope is input into a reasoning model learned using training data, wherein the training data is obtained by annotating the first interval with feature information of the second interval reached by the endoscope's anterior end portion as it changes position through the region during insertion or removal; and The output results of the inference model are used.

16. An endoscopic examination guidance device, comprising: The endoscopic feature information acquisition unit acquires endoscopic feature information of the lumen during insertion of the endoscope into the digestive organ lumen for endoscopic examination; and The guidance information output unit outputs guidance information when the tip of the endoscope is further inserted from the passage area to access a specific object site, based on the characteristic information within the lumen.

17. The endoscopic examination guiding device according to claim 16, wherein, The guidance information output unit has a reasoning model, which is learned using training data. The training data is obtained by annotating the image of the first interval obtained during the passage of the endoscope's front end with feature information of the second interval reached by the endoscope's front end from the passage area.

18. An endoscope-guided control method comprising the following steps: The steps to obtain information related to a specific location within the lumen of the subject's digestive organs; The steps for obtaining endoscopic images during the insertion of the digestive tract into the lumen of the digestive tract during an endoscopic examination of the subject's digestive organs; and The display control steps show information related to the specific location inferred by the inference model when an intraluminal image obtained from the camera section of the endoscope is input into the inference model. The inference model is learned using training data, which is obtained by annotating information related to the specific location in endoscopic images of timed intervals before the specific location in the endoscopic images during the insertion process.

19. The endoscopic guidance and control method according to claim 18, comprising the following steps: Enter the subject's chief complaint; and Based on the chief complaint, determine the information related to a specific location within the lumen of the digestive organ.

20. The endoscopic guidance and control method according to claim 18, comprising the following steps: Based on the information of the currently observed location during the examination within the lumen of the digestive organ, information related to a specific location within the lumen of the digestive organ is determined.

Images