Image processing device, image processing method, and image processing program
The image processing device improves AI accuracy in endoscopic images by selectively applying AI models to each region within the image, addressing the challenge of multiple body parts, and enhancing lesion detection precision.
Patent Information
- Application Number
- PCT/JP2023/019534
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-25
- Publication Date
- 2024-11-28
AI Technical Summary
In endoscopic image processing, the accuracy of AI-based lesion detection deteriorates when images of multiple human body parts are included in the same image, as existing technologies fail to selectively apply the appropriate AI models to each region, leading to partial incompatibility and reduced inference accuracy.
An image processing device and method that determines the regions within an image and selectively applies the most suitable AI models and parameters to each area, distinguishing between applicable and non-applicable targets, thereby improving AI processing accuracy by optimizing model application for each organ or region.
This approach enhances the accuracy of AI processing by ensuring the appropriate AI model is applied to each region, reducing processor load and improving the reliability of lesion detection, even in images with multiple body parts, thereby facilitating more precise diagnostics.
Smart Images

Figure JP2023019534_28112024_PF_FP_ABST
Abstract
Description
Image processing device, image processing method, and image processing program
[0001] The present invention relates to an image processing device, an image processing method, and an image processing program that improve the accuracy of inspection using endoscopic images.
[0002] In recent years, technologies that utilize AI (artificial intelligence) based on image data to assist in judgments that were previously made visually by humans have been developed in various fields. For example, in the medical field, computer-aided detection (CADe) and computer-aided diagnosis (CADx) are being used to detect lesion areas using multiple inference models generated from endoscopic images of various parts (regions) within the body.
[0003] The trained model (inference model) that identifies lesions within the body is generated by deep learning using a multi-layer neural network, using endoscopic images of each area as training data.
[0004] For example, Patent Document 1 (International Publication No. 2022 / 181748) discloses a technology for performing inspection by switching AI (inference model) according to the object of observation.
[0005] International Publication No. 2022 / 181748
[0006] However, in images in which images of multiple body parts are included in the same image, such as images of the boundaries between organs, an AI appropriate for the object of observation is not necessarily selected, which can result in a deterioration in the accuracy of AI processing such as lesion detection.The present invention aims to provide an image processing device, an image processing method, and an image processing program that can improve the accuracy of AI processing by making it possible to change the settings of the AI to be applied for each part, even when images of multiple parts are included in one image.
[0007] An image processing device according to one aspect of the present invention includes a processor consisting of hardware, which determines whether an image includes a first AI application target and also includes a second AI application target or an AI non-application target that is different from the first AI application target, identifies the area in the image to which the first AI application target is included, selects an AI to apply to the image, and if the image does not include the second AI application target or the AI non-application target, applies a first AI that is compatible with the first AI application target to the image, and if the image includes the second AI application target or the AI non-application target, has the area identification unit identify at least the area to which the first AI application target is included, and applies the first AI that is compatible with the first AI application target only to the area of the image to which the first AI application target is included.
[0008] An image processing method according to one aspect of the present invention has a control unit determine whether an image includes a first AI application target and also includes a second AI application target or an AI non-application target that is different from the first AI application target, identify areas in the image to which the first AI application target and the second AI application target are included, and if the image does not include an application target for the second AI or an application target for the AI non-application target, apply a first AI that is compatible with the first AI application target to the image, and if the image includes an application target for the second AI or an application target for the AI non-application target, identify at least the areas to which the first AI application target is included, and apply a first AI that is compatible with the first AI application target only to the areas of the image to which the first AI application target is included.
[0009] An image processing program according to one aspect of the present invention causes a computer to execute a procedure of determining whether an image contains a first AI application target and also a second AI application target or an AI non-application target that is different from the first AI application target, identifying the area in the image to which the first AI application target is to be applied, selecting an AI to apply to the image, and if the image does not contain the second AI application target or the AI non-application target, applying a first AI that is compatible with the first AI application target to the image, and if the image contains the second AI application target or the AI non-application target, having the area identification unit identify at least the area to which the first AI application target is to be applied, while applying a first AI that is compatible with the first AI application target only to the area of the image to which the first AI application target is to be applied.
[0010] According to the present invention, even when one image contains images of multiple parts, the AI processing accuracy can be improved by making it possible to change the settings of the AI to be applied to each part.
[0011] FIG. 1 is a block diagram showing an endoscope device including an image processing device according to a first embodiment of the present invention. FIG. 2 is a flowchart for explaining the operation of the first embodiment. FIG. 3 is an explanatory diagram showing an object to be inspected. FIG. 4 is an explanatory diagram showing the state of inspection. FIG. 5 is an explanatory diagram for explaining an image obtained by inspection. FIG. 6 is a flowchart for explaining another example of operation of the first embodiment. FIG. 7 is an explanatory diagram for explaining an image obtained by inspection. FIG. 8 is an explanatory diagram showing an example of display. FIG. 9 is an explanatory diagram showing another example of display. FIG. 10 is a flowchart showing an operation flow adopted in a second embodiment. FIG. 11 is a flowchart showing another operation flow.
[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[0013] 1 is a block diagram showing an endoscope device including an image processing device according to a first embodiment of the present invention. In this embodiment, in cases where an image to which AI (inference model) is applied includes images of multiple types of targets as targets to which AI is applied (hereinafter referred to as AI target), or where images include not only AI target targets but also targets to which AI is not applied (hereinafter referred to as AI non-target targets), the accuracy of AI processing is improved by making it possible to use or not use AI for each target image, and to switch the AI to be used.
[0014] This embodiment describes an example of controlling the setting changes of AI used for diagnosing lesions in endoscopic examinations of the upper gastrointestinal tract, but the target of AI application is not limited to the upper gastrointestinal tract and may be any part of the human body. Furthermore, this embodiment is applicable not only to the human body but also to various image processing devices that apply AI to each part of an image. Note that AI is not limited to AI for detecting lesions but also includes various AIs, such as AI for super-resolution processing, for improving image quality.
[0015] In an upper gastrointestinal tract endoscopy, an endoscope is inserted through the mouth and advanced through the oral cavity, pharynx, upper esophagus, lower esophagus, and stomach for observation, and then withdrawn and folded back toward the mouth for observation. In a lower gastrointestinal tract endoscopy, multiple areas of the large intestine (e.g., ascending colon, sigmoid colon) are examined. Furthermore, each endoscopy examines each area using a light source that switches between normal light and special light such as Narrow Band Imaging (NBI), or image enhancement techniques such as Texture and Color Enhancement Imaging (TXI). In addition to the upper or lower gastrointestinal tract, each endoscopy may also examine the small intestine.
[0016] Since examinations are performed on multiple organs or multiple regions, it may be difficult to detect lesions if inference is performed using the same model and the same parameters.
[0017] (Endoscopy Examination Method in Comparative Example) Therefore, in general, in order to improve inference accuracy (AI processing accuracy), it is necessary to switch the AI (hereinafter also referred to as model) applied to each organ or each region. For example, a model or parameters for the oral cavity must be applied to the oral cavity, and a model or parameters for the pharynx must be applied to the pharynx. Furthermore, models or parameters may be prepared for each region of an organ, such as the pylorus and the vestibule. Therefore, in endoscopic examinations using AI, there is a possibility that the effort of switching models during the examination may be required. Therefore, in the image processing device of the comparative example, a method is considered in which an organ is identified based on an examination image and the model is automatically switched. However, for example, a single image near the boundary between organs may contain multiple organs, regions, or tissues, and simply automatically switching models has the disadvantage of causing partial incompatibility.
[0018] (Control in the embodiment) Therefore, in the present embodiment, for example, by using area information of the part and information other than the part (scope, dark area, etc.), a state in which multiple parts are observed in the same image, such as the boundary between organs, is determined, and the optimal model is applied to each area of each part, thereby increasing the accuracy of inference and improving the accuracy of lesion detection.
[0019] 1, the endoscopic device includes an endoscope 10, an image processing device 20, and a display device 40. The endoscope 10 includes an image sensor 11. The endoscope 10 has an elongated and flexible insertion portion (not shown) that is inserted into a body cavity, and the image sensor 11 is provided, for example, at the tip of the insertion portion.
[0020] The endoscope 10 has an optical system (not shown) that guides an optical image of a subject to an imaging surface of the imaging element 11. The imaging element 11 is configured with a CCD, CMOS sensor, or the like, and photoelectrically converts the optical image of the subject from the optical system to obtain an image of the subject (image signal). The optical system may also include lenses and apertures (not shown) for zooming and focusing, and may also include a zoom (magnification change) mechanism, focusing mechanism, and aperture mechanism (not shown) that drive these lenses.
[0021] The endoscope 10 may also be provided with a forceps port (not shown). In this case, the surgeon can insert a treatment tool through the forceps port and protrude the treatment tool from the distal end opening of the insertion portion to perform treatment.
[0022] The endoscope 10 and the image processing device 20 are electrically connected. The image processing device 20 is provided with an imaging control unit 22, which drives the imaging element 11. The imaging element 11 captures an image of a subject under the control of the imaging control unit 22 and outputs an imaging signal to the image processing device 20. The insertion section of the endoscope 10 is provided with a bending section (not shown), which is configured to be actively bent in the up, down, left, and right directions by user operation. The imaging range of the imaging element 11 changes depending on the orientation of the tip of the insertion section.
[0023] The image processing device 20 may include, for example, a control unit 21, an imaging control unit 22, an image acquisition unit 23, an image generation unit 24, an area determination unit 25, an AI application target determination unit 26, an AI application unit 27, a parameter setting unit 28, an inspection result acquisition unit 29, a model memory unit 30, and a display control unit 31.
[0024] The control unit 21 of the image processing device 20 comprehensively controls each unit within the image processing device 20. The control unit 21 and each unit constituting the image processing device 20 may be configured by a processor using a CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), or the like, may operate according to a program stored in a memory (not shown) to control each unit, or may realize some or all of the functions by a hardware electronic circuit.
[0025] The image acquisition unit 23 captures captured images (moving images and still images) from the imaging element 11. The image generation unit 24 performs predetermined signal processing on the captured captured images, such as color adjustment processing, matrix conversion processing, noise removal processing, and various other signal processing. The display control unit 31 provides the image (endoscopic image) from the image generation unit 24 to the display device 40 for display. The display device 40 is, for example, a display device having a display screen such as an LCD (liquid crystal display device). The number of display devices 40 is not limited and may be multiple, or one display device 40 may have multiple display sections.
[0026] The display control unit 31 displays not only the endoscopic image acquired by the imaging element 11 but also the examination results using AI, which will be described later. That is, the display control unit 31 is provided with the AI processing results (inference results), which will be described later, and causes the display unit 40 to display these on the display device 40. For example, the display control unit 31 can also display the inference results indicating the position of the lesion on the image (observation image) from the imaging element 11, and the results of differentiation of the lesion.
[0027] The region determination unit 25, which serves as a region identification unit, determines one or more regions (hereinafter referred to as target regions) included in the image to which AI is to be applied and acquires region information for the target regions. The region determination unit 25 determines the region of each target included in the image by performing image analysis processing, such as AI processing, on the image acquired by the image acquisition unit 23. For example, the region determination unit 25 determines regions of human body parts (hereinafter referred to as region regions) and regions of insertion parts and dark parts (hereinafter referred to as non-region regions) in the image. The region determination unit 25 also determines region regions for each organ, such as the pharynx, esophagus, stomach, and duodenum, from the image. The region determination unit 25 also determines region regions for each part of an organ, such as the upper esophagus, middle esophagus, and lower esophagus, from the image. The region determination unit 25 also determines non-region regions, such as the insertion part of the endoscope 10, treatment tools, lumen, bubbles, residue, dark parts, reflected light, and blur, from the image. Furthermore, the area determination unit 25 determines the area where a pigment such as indigo carmine has been sprayed.
[0028] The model storage unit 30 stores a plurality of types of models that can be applied to organs, parts, tissues, dark areas, residues, bubbles, insertion parts, etc. As described above, these models include various models such as models for lesion detection and diagnosis, models for super-resolution processing, etc.
[0029] The AI application target determination unit 26 selects a model that matches each target region from the model storage unit 30. Furthermore, using the determination result of the region determination unit 25, the AI application target determination unit 26 determines AI application targets to which a model stored in the model storage unit 30 is applied in the image acquired by the image acquisition unit 23, and also determines AI non-application targets to which no model is applied. For example, the AI application target determination unit 26 determines AI application targets and AI non-application targets for organs, parts, tissues, dark areas, residue, bubbles, inserted parts, etc. included in the image. The AI application target determination unit 26 also determines whether two or more AI application targets are included in the image.
[0030] The AI application unit 27 reads out from the model storage unit 30 and applies the model (hereinafter referred to as the target application model) selected by the AI application target determination unit 26 for each target region obtained from the region information from the region determination unit 25. That is, the AI application unit 27 can execute inference processing using each target application model for the image of each target region of the image.
[0031] In this embodiment, when there are multiple target regions in an image, the AI application unit 27 may be configured to apply only the target application model that is compatible with only one target region to that target region.
[0032] The AI application unit 27 improves inference accuracy by switching the target application model for each target region, but the same target application model with different parameters may be used for multiple target regions. In this case, the parameter setting unit 28 is employed.
[0033] The parameter setting unit 28 adjusts and applies the parameters of the object application model for each target region obtained from the region information from the region determination unit 25. That is, the AI application unit 27 executes inference processing using the object application model whose parameters have been optimally adjusted for each image of each target region of the image.
[0034] (Reduction of processing) The area determination unit 25 and the AI application target determination unit 26 can always determine areas in the image, such as organs, parts, tissues, dark areas, residues, bubbles, and inserted parts, and set a target application model suitable for each area. However, processing that assumes that multiple parts are always shown in the image increases the processing load of the processor.
[0035] Therefore, the area determination unit 25 first determines whether the image contains only the first AI application target, or whether the image contains both the first AI application target and an object other than the first AI application target (S2 in FIG. 2). In the latter case, the increase in the processor processing load can be suppressed by detecting the boundary between the first AI application target and the object other than the first AI application target. Suppressing the increase in the processor processing load has the advantage of being able to increase the number of frames to which the detection result display can be applied.
[0036] For example, if the first AI application target is the stomach, examples of the first AI non-application target include the esophagus, residue, bubbles, the endoscope insertion portion, treatment tools, etc. If the first AI non-application target is an organ or a part of an organ, that part may be treated as the second AI application target and the second AI may be applied to it. For example, in the above example, if the esophagus is shown as the first AI non-application target, the esophagus AI may be applied to the esophagus part.
[0037] In other words, when the boundary portion is determined by the region determination unit 25, the AI application unit 27 and the parameter setting unit 28 execute AI processing (inference) for portions other than the boundary portion using a single target application model corresponding to each organ. Also, for the boundary portion, the AI application unit 27 and the parameter setting unit 28 execute AI processing for the boundary portion using a target application model in which parameters based on the determinations of the region determination unit 25 and the AI application target determination unit 26 are set.
[0038] The examination result acquisition unit 29 acquires the examination results based on the processing results of the inference processing by the AI application unit 27 or the inference processing by the examination result acquisition unit 29. The examination results, such as the location of the lesion and the type of the lesion, are acquired by the examination result acquisition unit 29. As described above, the examination results are supplied to the display control unit 31, and the endoscopic image and the examination results are displayed on the display screen of the display device 40.
[0039] Next, the operation of the embodiment configured as above will be described with reference to Figs. 2 to 8. Fig. 2 is a flowchart for explaining the operation of the first embodiment. Fig. 3 is an explanatory diagram showing an inspection object, and Fig. 4 is an explanatory diagram showing the inspection process. Fig. 5 is an explanatory diagram for explaining images obtained by the inspection. Fig. 6 is a flowchart for explaining another example of the operation of the first embodiment. Fig. 7 is an explanatory diagram for explaining images obtained by the inspection. Figs. 8 and 9 are explanatory diagrams showing display examples.
[0040] First, the inspection object and inspection method will be described with reference to FIGS.
[0041] The examination targets are the pharynx P1, upper esophagus P2, middle esophagus P3, lower esophagus P4, stomach P5, and duodenum P6 shown in Figure 3. The insertion section 10a of the endoscope 10 is inserted through the mouth or other opening (not shown) and advanced while capturing images with the imaging element 11. The imaging element 11 located at the tip of the insertion section 10a sequentially captures images of the pharynx P1, upper esophagus P2, middle esophagus P3, lower esophagus P4, stomach P5, and duodenum P6. The imaging direction (hereinafter referred to as the observation direction) of the imaging element 11 at the tip of the insertion section 10a (the direction of the arrow pointing downward in Figure 4) when the imaging direction is oriented in the insertion direction of the insertion section 10a is referred to as the insertion direction, and observation in which the observation direction is the insertion direction is referred to as insertion direction observation.
[0042] It should be noted that the upper esophagus P2, the middle esophagus P3, and the lower esophagus P4 may not be strictly distinguishable, and in the following description, the entire esophagus (mainly the lower esophagus) will be referred to as the esophagus P4.
[0043] 4 shows that the insertion section 10a is bent at the stomach P5, and the esophagus P4 is imaged from the stomach P5 side. The observation direction in this case, i.e., the observation direction when the imaging direction of the image sensor 11 is oriented in the opposite direction to the insertion direction of the insertion section 10a, is referred to as the withdrawal direction (the direction of the arrow pointing upward on the paper surface of FIG. 4), and observation in the withdrawal direction is referred to as withdrawal direction observation. Furthermore, while the insertion section 10a is being withdrawn, the duodenum P6, stomach P5, esophagus P4, middle esophagus P3, upper esophagus P2, and pharynx P1 are sequentially imaged by the image sensor 11, thereby obtaining endoscopic images of each part.
[0044] 2, the image acquisition unit 23 acquires an image from the image sensor 11. The area determination unit 25 and the AI application target determination unit 26 determine an AI application target from the inspection image acquired by the image acquisition unit 23. The area determination unit 25 and the AI application target determination unit 26 also determine whether a single inspection image includes a first AI application target, and also includes a second AI application target different from the first AI application target, or an AI non-application target (S2).
[0045] Assume now that the upper esophagus P2 is being imaged by the imaging element 11. In this case, the examination image mainly includes the upper esophagus P2, which is the target of application of the first AI (NO in S2), and the AI application unit 27 or the parameter setting unit 28 performs inference by applying the first AI, which is the AI for the upper esophagus, to the entire screen (S3). Based on the result of this inference, the examination result acquisition unit 29 obtains and outputs a detection result of the lesion in the upper esophagus P2 (S4).
[0046] It is also assumed that the esophagus (mainly the lower part of the esophagus) P4 is imaged by the imaging element 11. In this case, the examination image may include not only the esophagus P4 to which the first AI is applied, but also the stomach P5 or its dark portion to which the second AI is applied.
[0047] The left side of Figure 5 shows an example of an image I1 obtained when imaging is performed from the esophagus P4 side in the insertion direction. Image I1 includes an image P4i of the esophagus P4 in the peripheral area and a dark image (dark area) P5d of the stomach P5 in the central area.
[0048] In this case (YES in S2), in the example on the left side of FIG. 5 , the AI application unit 27 or parameter setting unit 28 identifies at least the esophagus P4 as a region to which the first AI should be applied, and applies the first AI (AI for the esophagus) to the region to which the first AI should be applied. On the other hand, the AI application unit 27 or parameter setting unit 28 does not apply the first AI to the dark image P5d, which is a region to which the second AI should be applied or a region to which no AI should be applied. In other words, the first AI is not applied to regions other than the region to which the first AI should be applied (S5). In this case, an inference is performed by applying the first AI, which is the AI for the esophagus, to the region determined to be the esophagus P4 in the examination image (S5). Based on the result of this inference, the examination result acquisition unit 29 obtains and outputs a detection result of the lesion in the esophagus P4 (S6).
[0049] 5 shows an example of an image I2 obtained when imaging is performed from the stomach P5 side in the removal direction. Image I2 includes an image P5i of the stomach P5 in the peripheral area, and a dark image (dark area) P4d of the esophagus P4 and an image P10a of the insertion section 10a in the central area.
[0050] In this case (YES in S2), in the example on the right side of FIG. 5 , the AI application unit 27 or the parameter setting unit 28 identifies at least the stomach P5 as a region to which the first AI is to be applied, and applies the first AI (AI for the stomach) to the region to which the first AI is to be applied. On the other hand, the AI application unit 27 or the parameter setting unit 28 does not apply the first AI to regions other than the region to which the first AI is to be applied (S5). That is, in this case, an inference is performed by applying the first AI, which is the AI for the stomach, to the region in the examination image determined to be the stomach P5 (S5). Based on the result of this inference, the examination result acquisition unit 29 obtains and outputs a detection result of the lesion in the stomach P5 (S6).
[0051] Next, a process that enables a reduction in the load on the processor will be described with reference to FIG.
[0052] When the image acquisition unit 23 acquires an image from the image sensor 11 (S1), the region determination unit 25 performs region determination in step S11 and then performs boundary determination (S12). That is, the region determination unit 25 determines the region area for each organ, the detailed region area within the organ, and the non-region area.
[0053] (Boundary Determination 1) The region determination unit 25 determines a region to which AI is applied based on a region and a non-region region such as an insertion region or a dark region. For example, the region determination unit 25 may determine a region as a boundary region when the area of one region in one image is equal to or greater than a threshold and the area of a dark region in the non-region region is equal to or greater than a threshold. In the example on the left side of FIG. 5 (insertion direction observation from the lower esophagus), if the area of the esophagus image P4i is equal to or greater than a threshold and the area of the dark region image P5d is equal to or greater than a threshold, i.e., if the dark region (lumen) appears large even though the esophagus P4 is being observed, the region determination unit 25 determines that the image is the boundary region (cardia) between the esophagus P4 and the stomach P5.
[0054] Furthermore, region determination unit 25 may determine a region as a boundary region when the area of one region in one image is equal to or greater than a threshold, and the areas of the dark region and the inserted portion of the non-region region are equal to or greater than a threshold. In the example on the right side of Fig. 5 (observation in the direction of removal from the stomach to the esophagus), if the area of stomach image P5i is equal to or greater than the threshold, and the areas of dark region image P4d and inserted portion image P10a are equal to or greater than the threshold, the image is determined to be the image of the boundary region (cardia) between stomach P5 and esophagus P4.
[0055] (Boundary Determination 2) The method of boundary determination by the region determination unit 25 is not limited to the above method. For example, as another boundary determination method, the region determination unit 25 may determine a boundary based on the area ratio of a region region to a non-region region. For example, the region determination unit 25 calculates the effective area by subtracting the area of non-region regions, such as treatment tools, bubbles, residue, reflected light, and blur, from the total area of the entire image. The region determination unit 25 also calculates the effective area by subtracting the area of a region region where a pigment such as indigo carmine has been sprayed from the total area of the entire image. The region determination unit 25 calculates the ratio of the area of one region region and the ratio of the area of a dark region to the effective area, and determines a boundary if each ratio is equal to or greater than a threshold. The region determination unit 25 also calculates the ratio of the area of one region region and the ratio of the area of a dark region and an insertion region to the effective area, and determines a boundary if each ratio is equal to or greater than a threshold.
[0056] In the example on the left side of Fig. 5, the area ratio of the esophagus image P4i and the area ratio of the dark portion image P5d in the image are each larger than a predetermined threshold, and the region determination unit 25 determines that the image is a boundary portion. Also, in the example on the right side of Fig. 5, the area ratio of the stomach image P5i and the area ratios of the dark portion image P4d and the insertion portion image P10a in the image are each larger than a predetermined threshold, and the region determination unit 25 determines that the image is a boundary portion.
[0057] (Boundary Determination 3) The above two boundary determination methods by region determination unit 25 determine a boundary region based on a part region and a non-part region. Region determination unit 25 can determine a boundary region even when a single image contains two or more part regions.
[0058] Figure 7 is used to explain an example of this case. The left side of Figure 7 shows an example of image I3 obtained when imaging is performed from the esophagus P4 side in the insertion direction. Image I3 includes an image P4i of the esophagus P4 in the peripheral area and a relatively bright image P5i of the stomach P5 in the central area. Moreover, the right side of Figure 7 shows an example of image I4 obtained when imaging is performed from the stomach P5 side in the removal direction. Image I4 includes an image P5i of the stomach P5 in the peripheral area and a relatively bright image P4i of the esophagus P4 and an image P10a of the insertion section 10a in the central area.
[0059] The region determination unit 25 determines that an image is a boundary region when the areas of two region areas (stomach and esophagus) in the image are equal to or greater than a threshold. For example, as shown on the left side of Fig. 7 (observation in the insertion direction from the esophagus to the stomach), if the area ratio of the esophagus image P4i in the image and the area ratio of the stomach image P5di in the image are each greater than a predetermined threshold, the region determination unit 25 determines that an image is a boundary region. Also, as shown on the right side of Fig. 7 (observation in the withdrawal direction from the stomach to the esophagus), if the area ratio of the stomach image P5i in the image and the area ratios of the esophagus image P4i and the insertion portion image P10a in the image are each greater than a predetermined threshold, the region determination unit 25 determines that an image is a boundary region.
[0060] When an image P10a of the insertion part is present in the image, the region determination unit 25 may change the threshold value for boundary determination in consideration of the area of the image P10a of the insertion part.
[0061] 6, it is determined whether the inspection image is a boundary area. If the inspection image is not a boundary area image, the AI application unit 27 or the parameter setting unit 28 performs inference using the first AI specified by the AI application target determination unit 26 for the entire image (S13).
[0062] On the other hand, if the test image is an image of a boundary portion, the AI application unit 27 or the parameter setting unit 28 performs inference using the first AI specified by the AI application target determination unit 26 for one part region (S14). For example, in the example on the left side of Fig. 5, the AI application unit 27 performs inference using the AI for the esophagus as the first AI for the image P4i of the lower esophagus. Also, in the example on the right side of Fig. 5, the AI application unit 27 performs inference using the AI for the stomach as the first AI for the image P5i of the stomach.
[0063] In the example on the left side of Fig. 5, the parameter setting unit 28 performs inference on the image P4i of the lower esophagus using an AI with parameters optimized for the esophagus as the first AI. In the example on the right side of Fig. 5, the parameter setting unit 28 performs inference on the image P5i of the stomach using an AI with parameters optimized for the stomach as the first AI. Note that the parameters set by the parameter setting unit 28 include the reliability at the time of lesion detection determination, a weighting coefficient at the time of inference, etc.
[0064] (Display Example) Next, a display example will be described with reference to FIGS. 8 and 9. FIG.
[0065] The example on the left side of Fig. 8 shows a display example when the image on the left side of Fig. 5 is acquired by the image acquisition unit 23. In this case, the center of the image is a dark area, and the AI application unit 27 or the parameter setting unit 28 executes inference processing using AI suitable for the image P4i of the lower esophagus around the image. The examination result acquisition unit 29 detects a lesion in the esophagus P4 based on the inference result of the AI application unit 27 or the parameter setting unit 28.
[0066] The inspection result acquisition unit 29 performs masking processing to display an image based on the result of the inference processing. For example, the inspection result acquisition unit 29 sets a display area frame that identifies the area where inference using AI was performed and a non-display area frame that identifies the area other than the area where inference was performed. The inspection result acquisition unit 29 displays the endoscopic image within the display area frame and performs masking processing to display a black level within the non-display area frame.
[0067] The inspection result acquisition unit 29 sets the display area frame using coordinate information of the area based on the area information from the area determination unit 25. Alternatively, the inspection result acquisition unit 29 may set the display area frame based on an area of pixels for which the reliability of the inference process using AI is equal to or greater than a threshold value.
[0068] The test results from the test result acquisition unit 29 are supplied to the display control unit 31. As shown on the left side of FIG. 8 , the display control unit 31 displays a display P4h corresponding to the image of the esophagus P4. The display P4h also includes a display P4lh showing the detection results of the lesion. The display control unit 31 also displays a black level display P5m corresponding to the area in the center of the image that was dark and therefore not subjected to inference processing. In other words, the display control unit 31 displays a masked version of the area where inference processing was not performed.
[0069] 8 shows a display example when the image on the left side of Fig. 7 is acquired by the image acquisition unit 23. In this case, the center of the image is a relatively bright stomach image P5i, and the AI application target determination unit 26 selects an AI that matches this image P5i, and performs inference processing using the AI that matches the stomach image P5i in the center of the image. The examination result acquisition unit 29 detects lesions in the stomach P5 based on the inference results of the AI application unit 27 or the parameter setting unit 28.
[0070] The examination results from the examination result acquisition unit 29 are supplied to the display control unit 31. For example, as shown on the right side of Fig. 8, the display control unit 31 performs a display P5h corresponding to an image of the stomach P5, and also performs a black level display P4m on the region of the esophagus P4 surrounding the image for which inference processing has not been performed. That is, the display control unit 31 performs a display in which the region for which inference processing has not been performed is masked. This masking process allows the examination results of the region of interest to be displayed in an easy-to-see manner.
[0071] (Another Display Example) FIG. 9 shows an example in which the image in FIG. 7 is captured.
[0072] The left side of Figure 9 shows an example in which, when the image on the left side of Figure 7 is captured, the AI application unit 27 and the parameter setting unit 28 perform processing using esophageal AI adapted to the esophagus image P4i, or processing using stomach AI adapted to the image P5i. The examination result acquisition unit 29 performs mask processing to display an image based on the results of the inference processing. For example, the examination result acquisition unit 29 sets a display area frame that identifies the area where inference using AI was performed and a non-display area frame that identifies the area other than the area where inference was performed. The upper center of Figure 9 shows an example in which a display area frame P4ifd based on the image P4i of the lower esophagus and a non-display area frame P5ifu based on the image P5i of the stomach are set. The lower center of Figure 9 shows an example in which a non-display area frame P4ifu based on the image P4i of the lower esophagus and a display area frame P5ifd based on the image P5i of the stomach are set.
[0073] The examination result acquisition unit 29 performs mask processing to display an endoscopic image within a display area frame and display a black level within a non-display area frame. The examination results of the examination result acquisition unit 29 are supplied to the display control unit 31. The display control unit 31 displays the endoscopic image within the display area frame specified by the examination result acquisition unit 29 and displays a black level within the non-display area frame. As a result, in accordance with the setting in the upper center of Fig. 9, an image P4i of the lower esophagus is displayed on the periphery of the image, and a black level image P5ib is displayed in the center of the image. In addition, in accordance with the setting in the lower center of Fig. 9, an image P5i of the stomach is displayed in the center of the image, and a black level image P4ib is displayed on the periphery of the image.
[0074] In this embodiment, even if an image to which AI is applied contains multiple targets for AI application or targets to which AI is not applied, it is possible to perform inference processing using AI that is suitable for the required target for AI application, thereby improving inference accuracy and enabling reliable diagnosis, etc.
[0075] Second Embodiment Fig. 10 is a flowchart showing an operation flow employed in a second embodiment. In Fig. 10, the same steps as those in Fig. 2 are assigned the same reference numerals, and a description thereof will be omitted. The hardware configuration of this embodiment is the same as that of the first embodiment.
[0076] In the first embodiment, an example was described in which the first AI is applied to the first AI application target, and the AI is not applied to the second AI application target and the non-AI application target other than the first AI application target. For example, at the boundary of the cardia, control is performed such that inference using an AI suitable for the esophagus is performed when observing the insertion direction, and inference using an AI suitable for the stomach is performed when observing the removal direction. In contrast, the second embodiment applies AIs suitable for each of the first and second region regions within a single image. For example, as in the example of FIG. 7 , when multiple regions are captured relatively brightly at the boundary, inference is performed using AIs suitable for each region. In this case, this embodiment enables reliable recognition of multiple AI application targets.
[0077] In step S2 of FIG. 10 , the region determination unit 25 and the AI application target determination unit 26 determine AI application targets from the examination image acquired by the image acquisition unit 23. The region determination unit 25 and the AI application target determination unit 26 also determine whether a single examination image includes a first AI application target and also includes a second AI application target different from the first AI application target or an AI non-application target (S2). If the determination is YES in step S2, the region determination unit 25 determines whether the stomach and esophagus are included in the single image as the first and second AI application targets (S21). The region determination unit 25 can almost certainly determine whether an image includes images of the stomach and esophagus. Note that at this stage, it is only known that the image includes images of the stomach and esophagus, and the region determination unit 25 has not yet determined the regions of the stomach and esophagus.
[0078] If the region determination unit 25 determines in step S21 that the image does not include images of the stomach and esophagus (NO in step S21), it performs a process of selecting an AI that matches the region included in the image (not shown). If the image includes images of the stomach and esophagus, the region determination unit 25 determines in step S22 whether an endoscope (insertion portion) that is an object to which the first AI is not applied is reflected in the image.
[0079] As shown in Figure 7, an image P4i of the esophagus and an image P45 of the stomach are reflected in the cardia. Furthermore, when observing the removal direction from the stomach to the esophagus, an image P10a of the insertion part may also be reflected. The region determination unit 25 determines whether the image is an image for insertion direction observation or an image for removal direction observation based on whether or not the image P10a of the insertion part is included in the image in S22. Note that the image P10a of the insertion part has significantly different characteristics from images of organ parts, and the region determination unit 25 can relatively easily distinguish the image P10a of the insertion part.
[0080] If the region determination unit 25 determines in S22 that the image P10a of the insertion portion is not reflected in the image, it determines that the image is an image obtained by imaging the stomach P5 from the esophagus P4 side during insertion direction observation. In this case, as shown on the left side of FIG. 7 , an image P4i of the esophagus P4 is included in the peripheral portion, and a relatively bright image P5i of the stomach P5 is included in the central portion. Therefore, in this case, the region determination unit 25 recognizes the center side of the image as the stomach and the outer periphery side of the image as the esophagus (S23). The region determination unit 25 always recognizes the boundaries of each region, and the AI application target determination unit 26 selects an AI appropriate for each region. The AI application unit 27 and parameter setting unit 28 apply the respective AI, for example, a CAD for lesion detection, to each region (S24). The examination result acquisition unit 29 performs lesion detection for the stomach region and the esophagus region (S25).
[0081] Furthermore, if the region determination unit 25 determines in S22 that an image P10a of the insertion unit is reflected in the image, it determines that the image is an image obtained by imaging the esophagus P4 from the stomach P5 side during removal observation. In this case, as shown on the right side of FIG. 7 , an image P5i of the stomach P5 is included in the peripheral region, and a relatively bright image P4i of the esophagus P4 and an image P10a of the insertion unit 10a are included in the central region. Therefore, in this case, the region determination unit 25 recognizes the center of the image as the esophagus and the outer periphery of the image as the stomach (S26). The region determination unit 25 always recognizes the boundaries of each region, and the AI application target determination unit 26 selects an AI appropriate for each region. The AI application unit 27 and parameter setting unit 28 apply the respective AI, for example, a CAD for lesion detection, to each region (S27). The examination result acquisition unit 29 performs lesion detection for the stomach region and the esophagus region (S28).
[0082] In the second embodiment, even when an image contains multiple AI application targets, AI processing is performed using AI that is appropriate for each AI application target. Therefore, the display control unit 31 displays the endoscopic image as is without performing masking on the image. Note that, in the second embodiment, an image that has undergone masking processing may also be displayed, as in FIG. 9 .
[0083] In this embodiment, even if an image to which AI is applied contains multiple AI application targets, inference processing can be performed using AI that is appropriate for each AI application target, thereby improving inference accuracy and enabling reliable diagnosis, etc.
[0084] This embodiment can also be applied to the process shown in Fig. 6, which reduces the load on the processor. Fig. 11 is a flowchart showing the operation flow in this case. In Fig. 11, the same steps as those in Fig. 6 are denoted by the same reference numerals, and the description thereof will be omitted.
[0085] The flow of Fig. 11 differs from the flow of Fig. 6 in that step S15 is added. In step S14, when the examination image is an image of a boundary portion, the AI application unit 27 or the parameter setting unit 28 executes processing using the AI designated by the AI application target determination unit 26 for each region. For example, in the example on the left side of Fig. 7, an AI for the esophagus is selected as the first AI for the image P4i of the lower esophagus, and an AI for the stomach is selected as the second AI for the image P5i of the stomach. The AI application unit 27 and the parameter setting unit 28 detect a first lesion in the esophagus and a second lesion in the stomach by inference processing using an AI suitable for the esophagus and an AI suitable for the stomach, respectively.
[0086] Other effects are the same as those of the second embodiment.
[0087] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some of the components shown in the embodiments may be omitted. Furthermore, components from different embodiments may be appropriately combined.
[0088] Furthermore, among the technologies described herein, many of the controls and functions, mainly those described in the flowcharts, can be set by a program, and the above-mentioned controls and functions can be realized by having a computer read and execute the program. The program can be recorded or stored, in whole or in part, as a computer program product on portable media such as flexible disks, CD-ROMs, non-volatile memories, or storage media such as hard disks and volatile memories, and can be distributed or provided at the time of product shipment, via portable media, or via communication lines. A user can easily realize the image processing device of this embodiment by downloading the program via a communication network and installing it on a computer, or by installing it on a computer from a recording medium.
Claims
1. An image processing device comprising a processor consisting of hardware, which determines whether an image contains an object to which a first AI is to be applied, and further contains an object to which a second AI is to be applied or an object to which AI is not to be applied that is different from the first AI object, identifies an area in the image to which the first AI is to be applied, selects an AI to be applied to the image, and if the image does not contain an object to which the second AI is to be applied or an object to which AI is not to be applied, applies a first AI that is suitable for the first AI object to the image, and if the image contains an object to which the second AI is to be applied or an object to which AI is not to be applied, applies a first AI that is suitable for the first AI object to only the area of the image to which the first AI is to be applied, while having the area identification unit identify at least the area to which the first AI is to be applied.
2. An image processing device as described in claim 1, comprising: an AI application target determination unit that determines whether an image includes a first AI application target and further includes a second AI application target or an AI non-application target different from the first AI application target; an area identification unit that identifies an application area of the first AI application target in the image; and if the image does not include the second AI application target or the AI non-application target, applies a first AI that is suitable for the first AI application target to the image, and if the image includes the second AI application target or the AI non-application target, applies the first AI that is suitable for the first AI application target only to the first AI application target in the image while having the area identification unit identify at least the area of the first AI application target.
3. The image processing device described in claim 2, wherein when the image includes an object to which the second AI is to be applied, the AI application unit applies the first AI that is adapted to the first AI application object to the first AI application object while causing the area identification unit to identify the area to which the first AI is to be applied and the area to which the second AI is to be applied, and applies the second AI that is adapted to the second AI application object to the second AI application object.
4. The image processing device of claim 2, wherein when the image includes an object to which AI is not to be applied, the AI application unit causes the area identification unit to identify an area to which the first AI is to be applied and an area to which the AI is not to be applied, applies the first AI that is suitable for the first AI application object to the first AI application object, and does not apply the first AI to the object to which AI is not to be applied.
5. An image processing device according to claim 2, wherein the first AI and the second AI are mutually different models, or are the same model with mutually different parameters set.
6. The image processing device according to claim 2, wherein the object to which AI is not applicable is at least one of an endoscope, a treatment tool, bubbles, residue, a dark area, reflected light, a dye, and blur.
7. The image processing device described in claim 3, wherein, when one of the first AI application target and the second AI application target is the stomach and the other is the esophagus, the AI application target determination unit determines whether or not an endoscope is included in the image as an AI non-application target, and, when the endoscope is included in the image, the AI application unit applies an esophageal AI suitable for the esophagus to the first AI application target and the second AI application target that are shown toward the center of the image, and applies a stomach AI suitable for the stomach to the first AI application target and the second AI application target that are shown toward the outer periphery of the image, and, when the endoscope is not included in the image, applies an esophageal AI suitable for the esophagus to the first AI application target and the second AI application target that are shown toward the outer periphery of the image, and applies a stomach AI suitable for the stomach to the first AI application target and the second AI application target that are shown toward the center of the image.
8. The image processing device according to claim 1, wherein the first AI is an AI that performs lesion detection.
9. An image processing method in which a control unit determines whether an image includes an object to which a first AI is to be applied, and also includes an object to which a second AI is to be applied or an object to which AI is not to be applied that is different from the first AI object, identifies areas in the image to which the first AI is to be applied and the second AI object, applies a first AI that is suitable for the first AI object to the image if the image does not include an object to which the second AI is to be applied or an object to which AI is not to be applied, and applies a first AI that is suitable for the first AI object to only the area of the image to which the first AI is to be applied, while identifying at least the area to which the first AI is to be applied, if the image includes an object to which the second AI is to be applied or an object to which AI is not to be applied.
10. An image processing program for causing a computer to execute the steps of: determining whether an image includes an object to which a first AI is to be applied, and also whether it includes an object to which a second AI is to be applied or an object to which AI is not to be applied that is different from the first AI object; identifying an area in the image to which the first AI is to be applied; selecting an AI to apply to the image; if the image does not include an object to which the second AI is to be applied or an object to which AI is not to be applied, applying a first AI that is suitable for the first AI object to the image; and if the image includes an object to which the second AI is to be applied or an object to which AI is not to be applied, while having the area identification unit identify at least the area to which the first AI is to be applied, and applying the first AI that is suitable for the first AI object only to the area of the image to which the first AI is to be applied.
Citation Information
Patent Citations
Image identification method and device, object identification method and device, and program
JP2005063308A
A system and method for detection of suspicious tissue regions in an endoscopic procedure
US20180253839A1
Apparatus for Monitoring Treatment Side Effects
US20210345957A1
Medical image processing apparatus, endoscope system, medical image processing method, and program
WO2019138773A1
Machine learning method and device, program, learned model, and identification device
WO2019167884A1
