Image classification device for endoscope, image classification method, image classification program, and search method

Abstract

This image classification device for an endoscope comprises: a site determination unit that determines anatomical sites in a human body in accordance with captured images sequentially obtained in a process of inserting or extracting the endoscope into or from the body; a feature determination unit that determines, regarding at least one determination item, site distinctive features which are features of the determined sites, and obtains site distinctive feature information; and an association unit that records, in a manner associated with the captured images, the site distinctive feature information for the respective sites.

Description

Endoscopic image classification device, image classification method, image classification program, and search method

[0001] The present invention relates to an endoscopic image classification device, an image classification method, an image classification program, and a search method for classifying and organizing images obtained in the process from the insertion to the removal of an endoscope for use in examination, diagnosis, etc.

[0002] An endoscope is a device that is inserted into the body and enables observation of diseased areas that cannot be seen from the outside. An endoscope has an insertion section that is inserted into a body cavity, and an imaging device is provided, for example, at the tip of the insertion section. During an examination using an endoscope, a doctor sequentially displays images acquired by imaging the tip of the insertion section, and adjusts the position of the tip of the insertion section while checking the displayed images to diagnose the patient's health or disease state.

[0003] The imaging device continues to take images throughout the process from insertion to removal of the insertion part, and in endoscopic examinations, in addition to images used for diagnosis, a huge amount of information is obtained, including images of each part (organ) of the human body that has passed through.

[0004] A technique for managing acquired image information is proposed in Japanese Patent Laid-Open Publication No. 2002-253539 (hereinafter referred to as Patent Document 1). This proposal discloses a technique for identifying medical images and adding imaging attribute information of the most matching category as management information.

[0005] Japanese Patent Application Laid-Open No. 2002-253539

[0006] The technology proposed in Patent Document 1 automatically adds imaging attribute information, such as imaging device, imaging region, and imaging direction, to medical images as management information. The device in Patent Document 1 makes it easy to distinguish which device, such as an X-ray device or an MRI device, was used to image which region from which direction, for example, whether the head or chest was imaged. The technology in Patent Document 1 makes it possible to determine which region an image belongs to, facilitating retrieval of past cases of a specific subject. However, while the acquired management information proposed in Patent Document 1 is useful for diagnosing the subject himself, complex search operations are required to use it as reference information for diagnosing others. The present invention aims to provide an endoscopic image classification device, image classification method, image classification program, and search method that accumulate medical information acquired during the process from insertion to removal of an endoscope and classify each region according to its characteristics based on the images. This allows for obtaining information that is extremely useful for diagnosis and treatment.

[0007] An endoscopic image classification device according to one aspect of the present invention comprises a part determination unit that determines anatomical parts of the human body according to captured images obtained sequentially during the process of inserting an endoscope into or removing it from the body, a feature determination unit that determines part-specific features that are characteristics of the determined parts for one or more determination items to obtain part-specific feature information, and an association unit that records the part-specific feature information for each part in association with the captured image.

[0008] An image classification method according to one aspect of the present invention determines anatomical parts of the human body based on captured images obtained sequentially during the process of inserting an endoscope into or removing it from the body, determines part-specific features that are characteristics of the determined parts for one or more determination items to obtain part-specific feature information, and records the part-specific feature information for each part in association with the captured image.

[0009] An image classification program according to one aspect of the present invention causes a computer to execute the following steps: determine anatomical parts of the human body based on captured images, etc., obtained sequentially during the process of inserting an endoscope into or removing it from the body; determine the part-specific features that are characteristic of the determined parts for one or more determination items to obtain part-specific feature information; and record the part-specific feature information for each part in association with the captured image.

[0010] Another aspect of the image classification method of the present invention involves determining a plurality of anatomical regions of the human body according to captured images obtained sequentially during the process of inserting or removing an endoscope into or from the body, and in order to obtain region-specific features that are characteristic of each of the plurality of regions, referring to a database that organizes determination items that indicate the features to be obtained for each of the regions, determining the region-specific features for each of the determined regions, and obtaining region-specific feature information based on the region-specific features so that the determined region-specific features can be recorded in association with the captured images.

[0011] A search method according to one aspect of the present invention searches for endoscopic images acquired in a second case using region-specific characteristic information indicating the characteristics of each anatomical region through which an endoscope passes during insertion into or removal from the body in a first case.

[0012] According to the present invention, by accumulating medical information acquired during the process from insertion to removal of the endoscope and classifying the information according to the characteristics of each part based on images of each part, it is possible to obtain information that is extremely useful for diagnosis and treatment.

[0013] FIG. 10 is a block diagram showing an endoscopic image classification device according to an embodiment of the present invention. FIG. 11 is an explanatory diagram showing an example of an input image. FIG. 12 is an explanatory diagram for explaining an inference model that enables body part determination. FIG. 13 is an explanatory diagram showing an example in which the correspondence between each body part and each determination item is summarized in a table format. FIG. 5 is an explanatory diagram showing an example of the shape of the large intestine. FIG. 5 is a flowchart for explaining the operation of the embodiment. FIG. 5 is a flowchart for explaining the operation of the embodiment. FIG. 6 is an explanatory diagram for explaining an example of an endoscopic examination. FIG. 7 is an explanatory diagram for explaining images acquired in the example of FIG.

[0014] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0015] (Embodiment) FIG. 1 is a block diagram showing an endoscopic image classification device according to one embodiment of the present invention. Video obtained through endoscopic examinations contains images of various parts of the body, from the outside to various organs such as the throat, esophagus, stomach, and duodenum, providing extremely rich information, but this information has not been fully utilized. In this embodiment, based on the images of each part, each part is classified by multiple features, the classification results are accumulated, and the accumulated classification results are made available as information useful for diagnosis. For example, by classifying the digestive system by features and accumulating the classification results, it is possible to determine the digestive features of people who are prone to digestive diseases, which can be useful for diagnosis and prevention.

[0016] A region is an anatomical part that can identify specific anatomical features. A region unit may be an organ unit with a specific function. Large organs such as the large intestine and stomach are further subdivided medically (for the stomach, names include the cardia, fundus, body, antrum, and pylorus; for the large intestine, names include the ascending colon, transverse colon, descending colon, sigmoid colon, and rectum). Therefore, a region may be a part of a subdivided organ, or a region may be an organ subdivided using a method different from the medical method.

[0017] In FIG. 1 , medical images, such as endoscopic images acquired by an endoscope (not shown), are input to the endoscopic image classification device 10. The endoscope has an insertion section inserted into a body cavity, and an imaging device is attached to the tip of the insertion section. This imaging device includes an imaging element, such as a CCD or CMOS sensor, and photoelectrically converts an optical image from a subject to obtain an imaging signal. This imaging signal may be supplied to a video processor (not shown) for signal processing and then supplied to the endoscopic image classification device 10. Furthermore, a channel (a communication path connecting the inside and outside of the body cavity) is provided inside the endoscope tube for passing instruments such as treatment tools, and ultrasonic sensors can be used through this channel. Various sensors and transmitters can also be attached to the tip of the endoscope. By linking these sensors or other devices, information other than images can be acquired as the endoscope is inserted into the body. The linked device may be an in-hospital system, and the doctor's voice, comments, and terminal operations during the endoscopic examination may also be utilized. Since endoscopic video information is time-series information, useful information can be obtained by adjusting the timing of the video. These may be recorded in association with the images as "accompanying information" in Fig. 1. The endoscopic classification device 10 is described here as a special device that classifies images obtained from many endoscopes, but it may also be directly connected to endoscopes to classify the results of endoscopic examinations in real time, or it may be linked to the display control unit 16 to display a guide that applies its functions during endoscopic examinations.

[0018] The medical images input to the endoscopic image classification device 10 are images of the inside of the body acquired by an endoscopic examination or the like. These input images, for example, endoscopic images, may be supplied directly from a video processor, or may be images read out and supplied from a recording device. While FIG. 1 shows an example of a video as the image input to the endoscopic image classification device 10 (input image), in addition to a series of continuously acquired images, still images may also be acquired and input as needed by the operation of a medical professional or under specific conditions.

[0019] Figure 2 is an explanatory diagram showing an example of an input image. In Figure 2, each frame f1 to f6 of a series of images Pe that are acquired consecutively is indicated by a square frame. Frame f4 includes an image portion of the lesion, indicated by an oval. In a typical endoscopic examination, the image of frame f4 is used for diagnosis, and the other frames f1 to f3, f5, and f6 are generally not used for purposes other than inserting the endoscope.

[0020] 1 shows an example in which many images including moving images of three persons A to C are input. For persons A to C, the images acquired in 2020 are input images P2020A to P2020C, respectively, and the images acquired in 2021 are input images P2021A to P2021C, respectively.

[0021] The input image may be accompanied by not only a video but also accompanying information, including operation information, sensor information, setting information for the endoscopic device, information on the behavior of the doctor performing the endoscopic examination and his / her assistant during the endoscopic examination, information on the treatment tools used, and other diagnostic results, pathological results, and interview results.

[0022] It may include information about the hospital and operating room, patient information such as the date and time, age, race, and gender, information about the doctor in charge and medical staff who participated in the examination, etc. It may also include information about other equipment used during the examination, and for example, recently, conversations between doctors and medical staff can be recorded using a microphone or the like, so it may be possible to record such information so that it can be associated with the image frames.

[0023] Similar to this, it may also include the various sensor information mentioned above. For example, various switches are provided on the operation unit of the endoscope (not shown), and operation information such as the operation of these switches may be added to the image data of the endoscopic image as accompanying information and input. Also, operation information based on the operation of the air and water supply device may be added as accompanying information. If information indicating which timing of the video frame these are from is included, this can also be used for the endoscopic image classification of the present application.

[0024] Furthermore, because special light observation images have different characteristics from normal images, it may be better to treat special light observation images and normal light observation images separately for various determinations. Therefore, information for distinguishing whether each frame of the captured image is a special light observation image or a normal light observation image may be added to the image information as accompanying information. Furthermore, information acquired by the endoscope insertion shape observation device may be added to the image information as accompanying information.

[0025] The endoscopic image classification device 10 includes a body part determination unit 12, a body part feature determination unit 13, an active operation determination unit 14, an association unit 15, a display control unit 16, a search unit 17, and a recording control unit 18. Each component of the endoscopic image classification device 10 may be configured by a processor using a CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), GPU (Graphics Processing Unit), or the like, may operate according to a program stored in a memory (not shown) to control each unit, or may realize some or all of its functions using a hardware electronic circuit.

[0026] In this embodiment, the part determination unit 12 of the endoscopic image classification device 10 sequentially determines which part of the body is being imaged in each input image. The part determination unit 12 determines each part of the human body from endoscopic images sequentially acquired by the endoscope during the insertion and removal process of the endoscope, i.e., from the start of insertion of the endoscope to the completion of removal. The part determination unit 12 can not only directly identify the anatomical features of each part from the image of each part, but can also determine the part when the anatomical features of each part can be inferred from information on other parts whose anatomical features have been identified.

[0027] For example, the part determination unit 12 determines the part by image analysis or inference processing of the endoscopic image. For example, the part determination unit 12 can identify a specific part such as the nasal cavity or stomach by determining the shape of the lumen from image features. Furthermore, for example, the part determination unit 12 can determine a specific part such as the large intestine by determining changes in image features (image history) that indicate that the insertion part is progressing while bending, and can also determine whether it is the descending colon or the transverse colon.

[0028] The part determination unit 12 may also determine the part by referring to a database (not shown) based on the characteristic color or shape of the part, or the pattern of blood vessels or other visible blood vessels on the surface. Another method involves using an external receiver to receive the transmission results of a magnetic or other transmitter attached to the tip or insertion section of the endoscope, determining the position of the tip of the endoscope on the body, and then determining which part the image was taken of. Another method involves determining the part based on the insertion depth and twist of the endoscope. Representative images of each part (e.g., images with adjusted image quality, as in papers) and data indicating the image characteristics of each part may be recorded on a recording medium (not shown), and the part determination unit 12 may determine each part by referring to the information recorded on the recording medium.

[0029] Furthermore, the part determination unit 12 may determine the part by AI (artificial intelligence) processing.

[0030] FIG. 3 is an explanatory diagram illustrating an inference model that enables body part determination. When determining body parts using AI processing, the body part determination unit 12 employs a network corresponding to each body part. The example of FIG. 3 shows two networks N1 and N2 for inferring first and second body parts. A large number of images corresponding to the input and output are provided as training data to the networks N1 and N2, respectively. In the example of FIG. 3, multiple images (rectangular frames) within the first body part are input as training data to the network N1, and multiple images (rectangular frames) within the second body part are input as training data to the network N2. The input images are annotated with, for example, the body part name corresponding to each image.

[0031] By learning using a large amount of training data, the network designs of networks N1 and N2 are determined so that they can obtain outputs corresponding to their respective inputs. That is, in the example of Figure 3, when an image of a first body part is input, network N1 outputs information on the body part name of the first body part, and when an image of a second body part is input, network N2 outputs information on the body part name of the second body part. That is, network N1 constructs an inference model that detects the body part name of the first body part from the input image, and network N2 constructs an inference model that detects the body part name of the second body part from the input image.

[0032] The correctness determining unit 12a determines whether the inference results from the networks N1 and N2 are correct, and if they are incorrect, performs learning control to reconstruct the network design of the networks N1 and N2.

[0033] Deep learning is a multilayered version of the machine learning process using neural networks. A typical example is a forward propagation neural network, which sends information from front to back and makes a judgment. In its simplest form, it requires three layers: an input layer consisting of m1 neurons, a hidden layer consisting of m2 neurons determined by parameters, and an output layer consisting of m3 neurons corresponding to the number of classes to be discriminated. The neurons in the input and hidden layers, and those in the hidden and output layers, are connected by connection weights, and a bias value is added between the hidden and output layers, making it easy to form logic gates. While three layers are sufficient for simple discrimination, increasing the number of hidden layers makes it possible to learn how to combine multiple features during the machine learning process. In recent years, neural networks with 9 to 152 layers have become practical due to their training time, judgment accuracy, and energy consumption.

[0034] The network N1 used for machine learning may be any of a variety of well-known networks. For example, R-CNN (Regions with CNN features) or FCN (Fully Convolutional Networks) using CNN (Convolution Neural Network) may be used. This involves a process called "convolution" that compresses image features, operates with minimal processing, and is strong in pattern recognition. Furthermore, a "recurrent neural network" (fully connected recurrent neural network) that can handle more complex information and allows information analysis whose meaning changes depending on the order or sequence of information may be used, allowing information to flow bidirectionally.

[0035] To realize these technologies, conventional general-purpose arithmetic processing circuits such as CPUs and FPGAs can be used, but because much of the processing in neural networks involves matrix multiplication, GPUs and Tensor Processing Units (TPUs), which are specialized for matrix calculations, may also be used.In recent years, such dedicated artificial intelligence (AI) hardware, called "neural network processing units (NPUs)," have been designed to be integrated and embeddable with CPUs and other circuits, and may even become part of the processing circuit.

[0036] Furthermore, inference models may be obtained by employing various well-known machine learning techniques, not limited to deep learning. For example, techniques such as support vector machines and support vector regression are available. Here, learning involves calculating the weights, filter coefficients, and offsets of a classifier; other techniques include using logistic regression processing. When a machine is to make a judgment, a human must teach the machine how to make the judgment. In this embodiment, a method for deriving an image judgment using machine learning is employed. However, a rule-based method for applying rules acquired by humans through experience or heuristics to make a specific judgment may also be used.

[0037] In this embodiment, the characteristics of each site (hereinafter referred to as site-specific characteristics) are determined for each site determined by the site determination unit 12, and the site is classified according to the determination results. The determination items for determining site-specific characteristics include the presence or absence of an affected area, passing time, number of frames acquired, number of times the lumen direction was lost, number of times the insertion / removal direction was redone, color characteristics, deformation during air supply, degree of residue, length, thickness, color, wrinkles, movement, elasticity, twisting, deflection, etc.

[0038] 4 is an explanatory diagram showing an example of the correspondence between each part and each evaluation item in a classified table format, in which the anus, rectum, sigmoid colon, and descending colon are shown as examples of parts.

[0039] The "presence or absence of an affected area" assessment item includes size, location, number, color, etc. The "passing time," "number of frames acquired," "number of times the lumen was lost," and "number of times the insertion / removal direction was retried" indicate the state of the endoscope insertion section during insertion and removal, respectively indicating the time required to pass through the area, the number of frames acquired at the area, the number of times the lumen (deep area) was lost while imaging the area, and the number of times the insertion section was inserted and removed while imaging the area. The "color characteristics" indicate the characteristics of the color and color change of the area, and the "air supply deformation" indicates the characteristics of the state in which the shape of the area changes due to air supply to the area. The state of deformation due to the swelling of the luminal tissue during air supply can also indicate characteristics such as whether the tissue is elastic, hard, or soft. The same applies to water supply, and characteristics such as the effectiveness of water supply may be recorded. In other words, the endoscopic image classification device of the present application is characterized in that a judgment item database (DB) 40 for referencing judgment items indicating characteristics to be judged for the above-mentioned regions includes features (judgment items) for judging region-specific characteristics of image changes following active treatment. Active treatment refers to not only ordinary screening or diagnostic observation, but also active intervention on the object that differs from ordinary observation (such as water supply, air supply, irradiation with special light (and image judgment at that time), staining (and image judgment at that time), and intervention with a treatment tool).

[0040] Additionally, the "level of residue" indicates the amount of residue from feces, etc. Of course, if the color, size, location, and other characteristics of the residue are specifically recorded, it may be possible to use this information to identify the characteristics of the internal structure of the digestive tract, digestive capacity, and identify illnesses.

[0041] In addition to air insufflation, it is also possible to determine whether water-based cleaning is effective. In other words, these can be said to classify the reactions of measures that actively act on the lumen (active manipulation) as characteristics. Since such active manipulation may or may not be performed depending on the case, and the applicable area varies, it may be impossible to compare the sensations of different cases. Therefore, it is possible to enter an examination item in the judgment item DB 40, such as insufflating air at this area to check the softness of the lumen, and to display a guide during the examination that encourages the examiner to perform this operation in every case, thereby making it a standardized examination item.

[0042] When an endoscopic examination is performed by connecting an endoscope directly to the endoscopic image classification device 10, the judgment item DB 40 may record information on the active operation recommended for each of the above-mentioned anatomical parts, and may display a guide to the endoscope operator in accordance with the information, in conjunction with the display device that displays the endoscopic image. The display control unit 16 is provided with such a composite display function.

[0043] The evaluation items "length," "thickness," "color," "wrinkles," "movement," "elasticity," "torsion," and "flexibility" indicate the characteristics of the part's length, thickness, color, wrinkles, movement, elasticity, torsion, and flexion. One method for determining length, etc., is to determine the insertion and removal time or fine structures such as blood vessels as patterns, and then determine how long they last from images. Another method is to infer lumen length from endoscopic images using an inference model trained on images of the endoscope moving inside the lumen and the lumen length obtained from X-rays, etc., as training data. Other characteristics such as the number, size, and spacing of intestinal folds may also be recorded.

[0044] The region feature determination unit 13 determines region features, which are characteristics of the region determined by the region determination unit 12, for one or more determination items. That is, the region feature determination unit 13 as a feature determination unit determines region features for various determination items for each region to obtain region feature information. The region feature determination unit 13 may obtain region feature information that classifies and indicates the region feature determination results in text format or table format. For example, the region feature determination unit 13 may determine each determination item by image analysis or AI processing of the input image. The region feature determination unit 13 may determine region features based on image changes in each frame of sequentially obtained images. As shown in FIG. 4 , common determination items may be set for all regions, but the determination items may differ for each region. Alternatively, table-format information may be stored for each region with common determination items. The determination item DB 40 is a database that stores information on determination items for each region. The region-specific feature determination unit 13 may be configured to read out the determination items stored for each region in the determination item DB 40 and make a determination for the read-out determination items. In other words, the endoscopic image classification device of the present application has a database for referencing features (determination items) for determining region-specific features for each of multiple regions through which the endoscope passes.

[0045] The active operation determination unit 14 obtains active operation information by determining signals based on active operations performed by the surgeon when inserting the endoscope into the body. Possible active operations include, for example, operating the control unit of the endoscope to capture still images, or operating devices for supplying air or water into the body. The active operation determination unit 14 receives operation signals based on operations on these devices, thereby obtaining active operation information as a determination result of the active operation. The active operation determination unit 14 may also obtain a determination result of the active operation by image analysis of the input image. It may also be possible to determine whether water has been supplied (e.g., an image of water splashing is obtained and then subsides) or air has been supplied (e.g., tissue has expanded and returned to its original state) based on changes in the image over time.

[0046] The associating unit 15 associates the part-specific features of each part with the input image and records them. That is, the associating unit 15 associates the determination results of the part determination unit 12, the part-specific feature determination unit 13, and the active operation determination unit 14 with the input image. The associating unit 15 controls the recording control unit 18 to record the associated results in the recording device 30. For example, the associating unit 15 may associate and record these determination results as metadata of a still image or video file. As a result, part information indicating which part was captured, part-specific feature information about that part in, for example, a table or text format, and active operation information performed at the time of capturing that frame are associated and recorded in the still image file or the image information of each frame of the video file. Note that accompanying information associated with the input image is also associated and recorded as metadata in the image file. Furthermore, when the part determination unit 12 determines a part using AI processing, information on the inference model used for the determination may also be associated and recorded in association with the image.

[0047] The recording control unit 18 controls the recording of data in accordance with the associating unit 15. The recording device 30 has a recording medium (not shown), and records the associated image files under the control of the recording control unit 18. In this way, images of many people who have undergone endoscopic examinations are recorded in association with each frame of body part information, body part characteristic information, and active operation information, in addition to the accompanying information added at the time of imaging.

[0048] The display control unit 16 controls the display of the display device 20. The display device 20 is a display device such as an LCD (liquid crystal display), and performs various displays under the control of the display control unit 16. The display control unit 16 can display the search results of the search unit 17, which will be described later, on the display screen.

[0049] The search unit 17 searches for images based on user operations on an input device (not shown). For example, the search unit 17 may search for images that have common body part information and common body part characteristic information. That is, the search unit 17 may perform a search process based on the body part characteristic information for captured images of the body part determined by the body part determination unit 12 to extract captured images of similar cases. The search unit 17 may also search for images of the same body part of the same person captured at different times. The search unit 17 provides the search results to the display control unit 16, which causes the display unit 20 to display the search results on the display screen.

[0050] FIG. 1 shows an example of a display on the display screen of the display device 20. In the example of FIG. 1, an image Pe1 of a certain region currently (2023) acquired for subject α is displayed. An image Pe1a of a lesion such as a polyp is displayed within image Pe1. The search unit 17 searches for an image having region information and region characteristic information that match the region information and region characteristic information associated with image Pe1. For example, it is assumed that the region of image Pe1 is determined to be the large intestine by the region determination unit 12.

[0051] FIG. 5 is an explanatory diagram showing examples of the shape of the large intestine. Example E1 shows an example of a large intestine with little twisting or bending, Example 2 shows an example of a large intestine with significant twisting, and Example 3 shows an example of a large intestine with significant bending. The region-specific characteristic information also includes information on the shape characteristics of such regions. Even if organs of the same region have different sizes, shapes, etc., symptoms may differ even if similar lesions are present. Therefore, in this embodiment, images of the same region but with the same region-specific characteristics are searched for.

[0052] The search unit 17 searches for images having region information and region-specific characteristic information corresponding to image Pe1. That is, images of the same region, the large intestine, that have similar shapes, sizes, elasticities, etc., and similar affected areas, i.e., images with the same region-specific characteristics, are obtained as search results. Note that, depending on how the region-specific characteristics are classified, the search unit 17 may be configured to obtain, as search results, not only images with the same region-specific characteristics, but also images with similar region-specific characteristics.

[0053] The example in FIG. 1 shows that image Pe2 has been searched for and displayed as an image having site information and site-specific characteristic information corresponding to image Pe1. Image Pe2 includes image Pe2a of a lesion similar to image Pe1a of image Pe1. Note that image Pe2 was captured in 2020. Furthermore, the search unit 17 searches for progress information regarding the treatment details and the condition of the affected area for the person whose image Pe2 was captured, for example, whether there is an image obtained by capturing the same site in 2021, after the time image Pe2 was captured, and displays the search results. Image Pe3 shows an image resulting from this search, and image Pe3 includes image Pe3a, which shows that the lesion shown in image Pe2a has grown in size.

[0054] In other words, the search unit 17 can search for endoscopic images acquired in the second case using the region-specific characteristic information obtained for each anatomical region through which the endoscope passes during insertion into or removal from the body in the first case.

[0055] 1 , for example, a doctor can determine that there is a possibility that the lesion in subject α will also expand, since other people who have a lesion similar to the lesion in the colon of subject α and the same site-specific characteristics have subsequently expanded the lesion. Furthermore, the site-specific characteristic information can also be used to identify patients. Therefore, even if patient information such as the patient's name is unavailable, the site-specific characteristic information can be used to identify the patient and record the case.

[0056] (Operation) Next, the operation of the embodiment configured as above will be described with reference to Fig. 6 to Fig. 10. Fig. 6, Fig. 7 and Fig. 10 are flowcharts for explaining the operation of the embodiment. Fig. 8 is an explanatory diagram for explaining an example of endoscopic examination, and Fig. 9 is an explanatory diagram for explaining images acquired in the example of Fig. 8.

[0057] Figure 8 shows the oral cavity, larynx, and pharynx of a human body B. Figure 8 shows an upper gastrointestinal endoscopy or bronchial examination using oral endoscopy, in which an insertion section 51 is inserted through the mouth. As shown in Figure 8, in an examination or treatment using an endoscope 50, a doctor operates the endoscope 50 to insert the insertion section 51 of the endoscope 50 into the human body B. A bending section (not shown) is provided at the tip of the insertion section 51, and the doctor operates a bending knob 50a or the like provided on the endoscope 50 to bend the bending section or move the insertion section 51 forward or backward, thereby inserting the tip of the insertion section 51 to the site to be observed.

[0058] Fig. 9 shows an example of images acquired during the endoscopic examination of Fig. 8. The rectangular frames in Fig. 8 indicate frames to be acquired, and when the endoscope is inserted, many images of the tongue, trachea, esophagus, etc. are acquired in addition to the region to be observed. In this way, the insertion section 51 passes through various regions before reaching the esophagus, and images of various regions are acquired.

[0059] During the insertion of this endoscope, images taken up to the specific observation site may not be used for diagnosis, etc. However, as the endoscope passes through various sites on the way to the observation site, images of sites other than the observation site are also acquired during insertion or removal of the endoscope. That is, in addition to the endoscopic images used for diagnosis, many endoscopic images are acquired during the insertion or removal of the endoscope. These endoscopic images may contain information that is extremely useful for the next examination or for treating others with similar symptoms.

[0060] However, simply recording these endoscopic images makes it extremely difficult to extract useful information from the vast number of images. Therefore, in this embodiment, the characteristics of each region are determined and the determination results are recorded in association with the images as metadata, making it easier to extract useful information.

[0061] Figure 6 shows an example in which such determination of regional features is performed when the endoscope is inserted, but it may also be performed during screening or observation.Furthermore, after the endoscope is removed, endoscopic images recorded in a recording device (not shown) may be provided to the endoscopic image classification device 10 to determine regional features.

[0062] In the example of FIG. 6 , the endoscopic image classification device 10 determines in S1 whether an endoscope is being inserted. During insertion (YES in S1), the site determination unit 12 performs site determination and determines whether site determination has been completed (S2). Once site determination has been completed, in the next step S3, special light observation images are treated separately, and the site feature determination unit 13 detects site features. In special light observation, organs are imaged under special light, such as infrared light. When determining site features, it is not possible to simply compare images of organs illuminated under normal light (white light) with images captured under special light. Therefore, special light observation images are treated separately. Specifically, site features of special light observation images are determined within a group of special light observation images, and are used for searching within that group. The endoscopic image classification device 10 may determine whether an image is a special light observation image by image analysis. Alternatively, if accompanying information indicating whether an image is a special light observation image is attached to the image, the accompanying information may be used to determine whether the image is a special light observation image.

[0063] The site determination unit 12 and the site feature determination unit 13 may perform site determination and site feature determination not only by image analysis of the input image or AI processing, but also by using accompanying information and active operation information. For example, with regard to deformation during air supply, which is one of the site feature determination items, information on the air supply operation is obtained from active operation information, and site features are determined by image analysis of the deformation of the organ during the air supply operation. That is, the active operation information in this case is related to the timing of the captured image, and effective determination results can be obtained by determining site features according to changes in the image corresponding to the active operation. Site determination may also be performed using accompanying information based on the output of the insertion shape observation device.

[0064] When the region feature determination unit 13 detects region features (YES in S4), the association unit 15 organizes the region features by region and records them in association with the endoscopic image (S5). For example, the association unit 15 may record region information and region feature information as text, associated with each frame as image metadata. When recording is complete, the process returns to S1. Note that if the determinations in S2 and S4 are NO, the process also returns to S1. Note that when recording in S5, not only the region information and region feature information are recorded, but also various information such as accompanying information and active operation information included in the input image, and the type of logical model used to determine the region, is recorded.

[0065] In this way, the vast amount of image information obtained from endoscopic examinations is organized by regional characteristics, making it possible to search for endoscopic images of other cases using regional characteristic information for each anatomical region that the endoscope passes through during insertion into or removal from the body in a particular case.

[0066] If the endoscopic image classification device 10 determines in S1 that it is not the time of insertion (NO in S1), it determines in S11 whether it is the time of observation. If the endoscopic image classification device 10 determines in S11 that it is not the time of observation (NO in S11), it performs screening in S12. Note that screening refers to the act of imaging a relatively wide area to search for a lesion, etc. Furthermore, the act of imaging a specific area in detail and checking it, for example by imaging close up or zooming in on the specific area, is referred to as "observation."

[0067] The endoscopic image classification device 10 can determine insertion, screening, and observation times through image analysis. For example, assume that the insertion section 51 is advancing through a lumen. In this case, the image portion of the lumen deep inside (deep in the direction of the lumen length) where illumination light from the endoscope tip does not reach has a low-brightness lumen cross-sectional shape (often approximately circular). As the insertion section 51 advances through the lumen, this image portion is located approximately at the center of the endoscopic image, and successive images are obtained in which the lumen wall pattern moves toward the periphery of the image. Furthermore, if the endoscope tip is bent toward the target wall from a state in which the low-brightness lumen cross-sectional shape portion representing the deep lumen is located at the center of the image, the low-brightness portion that was in the center moves to the periphery of the image, and this state can be determined through image analysis. Furthermore, when observing a specific patterned area for a long period of time by moving closer and further away or changing the viewing direction, similar patterns such as blood vessels and lesion irregularities are captured in successive images, and the observation state can be determined based on changes in these patterns.

[0068] In S13, the results of the screening are displayed on the display device 20 and recorded in the recording device 30. Furthermore, during observation (YES in S11), the endoscopic image classification device 10 determines and differentiates lesions using known image processing, AI (artificial intelligence) processing, etc. (S14), and displays the results on the display device 20 and records them in the recording device 30 (S15). After the processing of S13 and S15, the process returns to S1.

[0069] In this way, image files containing endoscopic images of many people who have undergone endoscopic examinations, with information such as site information, site-specific characteristic information, accompanying information, and active operation information associated with the images, are stored in the recording device 30. The image information recorded in the recording device 30 can be classified by the information associated with the images, and images can be searched based on the information associated with the images.

[0070] (Video) The flow of Fig. 7 corresponds to the flow of Fig. 6 and is an example of detecting region-specific features using video. In Fig. 7, the same steps as in the flow of Fig. 6 are given the same reference numerals and their explanations are omitted. Note that the flow of Fig. 7 also shows an example in which region determination and region-specific feature determination are performed when the endoscope is inserted, but region determination and region-specific feature determination may also be performed when the endoscope is removed, during endoscopic observation, during screening, or after the endoscopic examination is completed.

[0071] In S21, imaging by the endoscope is started, determination of the region and region-specific features (feature determination) is started, display of the captured image is started, and recording of the captured image is started. In S22, the endoscopic image classification device 10 determines whether the endoscope is being inserted. When it is being inserted (YES in S22), a variable N indicating the region is initialized to 1, and it is determined whether determination of the Nth region has begun (YES in S24). When determination of the Nth region has begun (YES in S24), in S25, the endoscopic image classification device 10 matches the imaging conditions, such as whether special light observation or normal observation is being performed, and acquires the Nth region image. Note that accompanying information and active operation information are also acquired simultaneously with the image.

[0072] The part determination unit 12 determines each part using the acquired images, accompanying information, and active operation information. Furthermore, in S26, the part feature determination unit 13 determines part features for each part using the acquired images, accompanying information, and active operation information. In S26, the endoscopic image classification device 10 determines whether the determination process for the Nth part has been completed. If the determination of the Nth part has not been completed (NO in S26), the process returns to S25, and determination of parts and part features continues.

[0073] When the determination of the Nth part is completed (NO in S26), in S27, the endoscopic image classification device 10 acquires timing information and track information. The timing information is information about the imaging time for the Nth part, and the track information is information that indicates the position of the recording unit (track) in the image information of the Nth part.

[0074] The associating unit 15 organizes the region-specific characteristic information for the Nth region and temporarily records it in table format or text format as metadata in the image information (S28). This region-specific characteristic information is recorded in association with the endoscopic image in correspondence with timing information and track information. By using the timing information and track information, the region-specific characteristic information for the Nth region can be easily extracted from the recorded image information. When recording metadata in text format, since the patterns of the recorded text can be counted in advance, a code corresponding to each text pattern may be assigned and the code may be recorded as metadata. In this case, the correspondence information between the text pattern and the code is shared appropriately with the system that references the code.

[0075] In S29, the endoscopic image classification device 10 increments the variable N indicating the region, and in S30, determines whether the examination has ended. If the examination has not ended (NO in S30), the process returns to S22 and repeats. If images of all intended regions have been captured and the examination has ended (YES in S30), the associating unit 15 controls the recording control unit 18 in S31 to record region-specific characteristic information for all regions in table format or text format in the recording device 30 as metadata in an image file (image file acquired during examination), thereby ending the examination. As a result, the recording control unit 18 records tabular information as region-specific characteristic information describing the characteristics of each region for each of the multiple regions passed through during the insertion or removal process.

[0076] If the determination in S22 is NO, the operation is the same as that shown in FIG.

[0077] In this way, image files containing endoscopic images of many people who have undergone endoscopic examinations, with information such as site information, site-specific characteristic information, accompanying information, and active operation information associated with the images, are stored in the recording device 30. The image information recorded in the recording device 30 can be classified by the information associated with the images, and images can be searched based on the information associated with the images.

[0078] (Use of Classified and Accumulated Information) FIG. 10 shows a method for using image information recorded in the recording device 30 (data confirmation flow).

[0079] The endoscopic image classification device 10 receives an examination specification for specifying the image information to be used via an input device (not shown). In S41, the endoscopic image classification device 10 is in a standby state for the specification of an examination for a certain subject. For example, a user can specify the target image information by specifying the person's name, the examination date and time, the type of examination (e.g., upper endoscopy), etc. When an examination specification is received (YES in S41), the search unit 17 searches for an image file corresponding to the specified examination (S42). The search unit 17 provides the searched image file to the display control unit 16, which then displays the image on the display device 20.

[0080] The search unit 17 determines whether the user has specified a region of the image they wish to view (region specification) (S43). If no region has been specified, the search unit 17 controls the display control unit 16 to start video playback of the displayed image (S46), and then determines whether a playback stop operation has been performed (S47). In S47, the search unit 17 continues video playback until a stop operation is performed by the user. If a stop operation is performed, the process proceeds to S44. If no stop operation is performed, the process proceeds to S51. In S51, the search unit 17 determines whether the process has ended, and if not, the process returns to S43.

[0081] If a stop operation is performed during playback of the moving image (YES in S47), the search unit 17 displays a still image of the part displayed at the time of the stop operation (S44). Also, if the user inputs a part such as the stomach or large intestine using an input device (not shown) in S43 (YES in S43), the search unit 17 displays a still image of the part corresponding to the input designation (S44).

[0082] In this way, an image of the specified subject and the part of the body that the doctor or the like wants to check is displayed. In S45, the search unit 17 determines whether or not an operation for checking similar cases has been performed. If no such operation has been performed (NO in S45), the process proceeds to S51. If a confirmation operation has been performed, the process proceeds to S48.

[0083] In S48, the search unit 17 searches for videos with similar characteristics. That is, the search unit 17 searches the vast amount of image information stored in the recording device 30 for image portions of images that are based on the region information and have region-specific characteristics based on the region-specific characteristic information (S48), reads out the image portions based on timing information and track information that specify the positions of the image portions, and displays them as still images on the display device 20 (S49) (see image Pe2 in Figure 1). The search unit 17 may also use accompanying information and active operation information in this search process.

[0084] Furthermore, if there is progress information for the subject whose similar image was found in S48, the search unit 17 searches for the progress information and displays the found information (see image Pe3 in FIG. 1).

[0085] In this embodiment, each part is classified by its regional features based on the image of each part, and the classification results are stored. Searching for images based on the regional features facilitates the search for images with similar regional features. This makes it easier to extract not only one's own examination images with similar regional features, but also other people's examination images similar to one's own, which can be useful for diagnosis and prevention.

[0086] It is also possible to search for similar images, including the procedure, using information about the movements of the surgeon (doctor) and the movements of the assistant during an endoscopic examination, treatment, or surgery using an endoscope. Similarly, it is also possible to search for images using setting information about the treatment tools and devices used in the examination, treatment, or surgery.

[0087] In this embodiment, an example of a gastrointestinal endoscopy has been described, but the present invention can also be applied to endoscopes other than those inserted into the gastrointestinal tract, such as those used in laparoscopic surgery and thoracoscopic surgery (in the fields of surgery, otorhinolaryngology, gynecology, urology, etc.) Not only in the case of a single connected tube such as the intestinal tract, but also in the case of a region such as the bronchi, where the tubes are branched, it is preferable to add information on which tube has been selected.

[0088] Furthermore, because endoscopes move back and forth within the body during insertion and removal, their characteristics can be classified separately for insertion and removal. In this case, two tables like those in Figure 4 are created for each case. Characteristics can change due to friction, such as the push and pull of the lumen, so the table can be organized as average values.

[0089] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some of the components shown in the embodiments may be omitted. Furthermore, components from different embodiments may be appropriately combined.

[0090] Furthermore, among the technologies described herein, many of the controls and functions, mainly those described in the flowcharts, can be set by a program, and the above-mentioned controls and functions can be realized by a computer reading and executing the program. The program can be recorded or stored, in whole or in part, as a computer program product on a portable medium such as a flexible disk, CD-ROM, or nonvolatile memory, or on a storage medium such as a hard disk or volatile memory, and can be distributed or provided at the time of product shipment, via a portable medium, or via a communication line. A user can easily realize the endoscopic image classification device of this embodiment by downloading the program via a communication network and installing it on a computer, or by installing it on a computer from a recording medium.

Claims

1. A site determination unit that determines an anatomical site in a human body according to captured images sequentially obtained during the process of inserting or removing an endoscope into or from the body; a feature determination unit that determines feature information specific to a site by determining, for one or more determination items, features specific to the determined site; and an association unit that associates and records the feature information specific to each site with the captured image, wherein the endoscope image classification device is characterized by comprising these components.

2. The endoscope image classification device according to claim 1, further comprising a recording control unit that records, for each of a plurality of sites passed through during the insertion or removal process, tabular information as the feature information specific to each site, which describes the features of each site.

3. The endoscope image classification device according to claim 1, further comprising a database for referring to the determination items for determining the features specific to each site for each of a plurality of sites passed through during the insertion or removal process.

4. The endoscope image classification device according to claim 1, wherein the database includes the determination items for determining the features specific to the site of image change according to an active treatment.

5. The endoscope image classification device according to claim 3, wherein the database includes information on recommended active operations at the anatomical site, and further comprises a display control unit that displays a guide to an endoscope operator according to the information on the active operation.

6. The endoscope image classification device according to claim 1, further comprising a search unit that searches for endoscope images obtained in a second case using the feature information specific to each anatomical site obtained for each anatomical site passed through by the endoscope during the process of inserting or removing the endoscope into or from the body in a first case.

7. The endoscope image classification device according to claim 1, further comprising an active operation determination unit that determines a signal based on an operator's operation when the endoscope is inserted into the body to obtain active operation information, and the association unit associates and records site information indicating the site, the feature information specific to the site, and the active operation information with the captured image.

8. The endoscope image classification device according to claim 7, wherein accompanying information is added to the captured image, and the association unit associates and records the site information, the feature information specific to the site, the active operation information, and the accompanying information with the captured image.

9. The endoscopic image classification device according to claim 1, wherein the feature determination unit obtains part-specific feature information that classifies and shows the determination results of the part-specific features in text format, symbols, or table format.

10. The endoscopic image classification device according to claim 9, wherein the association unit adds and records the part-specific feature information in the table format as metadata to the captured image when recording the file of the captured image.

11. The endoscopic image classification device according to claim 9, wherein the association unit records the part-specific feature information in association with the captured image in correspondence with the timing information of the captured image.

12. The endoscopic image classification device according to claim 1, wherein the feature determination unit determines the part-specific features based on the image changes of each frame of the sequentially obtained captured images.

13. The endoscopic image classification device according to claim 1, wherein the active operation information related to the timing of the captured image is acquired, and the part-specific features are determined according to the image changes corresponding to the active operation.

14. The endoscopic image classification device according to claim 1, further comprising a search unit that performs a search process based on the part-specific feature information on the captured image of the part determined by the part determination unit to extract captured images of similar cases.

15. The endoscopic image classification device according to claim 9, further comprising a display control unit for displaying the search results of the search unit on a display device.

16. The endoscopic image classification device according to claim 15, wherein when there is progress information of the subject in which the captured image of the similar case is captured, the search unit searches for the progress information, and the display control unit displays the progress information.

17. An image classification method, comprising: determining an anatomical part in the human body according to the captured images sequentially obtained during the process of inserting the endoscope into the body or removing it from the body; determining part-specific features, which are the features of the determined part, for one or more determination items to obtain part-specific feature information; and recording the part-specific feature information for each part in association with the captured image.

18. An image classification program for causing a computer to execute a procedure of determining an anatomical site in a human body according to captured images sequentially obtained during the process of inserting an endoscope into the body or removing it from the body, determining site-specific features, which are features of the determined site, for one or more determination items to obtain site-specific feature information, and recording the site-specific feature information for each site in association with the captured image.

19. An image classification method characterized by determining a plurality of anatomical sites in a human body according to captured images sequentially obtained during the process of inserting an endoscope into the body or removing it from the body, referring to a database in which determination items indicating features to be obtained for each site are arranged in order to obtain site-specific features, which are features of the site, for each of the plurality of sites, determining the site-specific features for each of the determined sites, and obtaining site-specific feature information based on the site-specific features so that the determined site-specific features can be recorded in association with the captured image.

20. A search method characterized by searching for an endoscope image obtained in a second case by using site-specific feature information indicating features for each anatomical site through which the endoscope passes during the process of inserting the endoscope into the body or removing it from the body in a first case.

Images