Image processing device, medical diagnostic device, endoscope device, and method of operating medical equipment

Abstract

To provide an image processing device, a medical diagnostic device, an endoscope device, and an image processing method capable of suppressing erroneous discrimination on a lesion.SOLUTION: A processor of an image processing device executes determination processing for acquiring a medical image including a lesion area, and determining a type of lesion by a determiner on the basis of the medical image. First determination processing determines whether the type belongs to a first group including a first type classified into a first category and a second type classified into a second category or whether the type belongs to a second group including a third type classified into the second category. Second determination processing determines whether the type is the first type or whether the type is the second type. The processor outputs a first signal capable of specifying whether the type is classified into the first category or whether the type is classified into the second category on the basis of the determination result of the first determination processing and the determination result of the second determination processing.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The technology of the present disclosure relates to an image processing device, a medical diagnostic device, an endoscope device, and an image processing method. [Background technology]

[0002] Patent Document 1 discloses a medical image diagnosis support device that identifies the type of lesion. The medical image diagnosis support device described in Patent Document 1 includes an image acquisition unit that acquires multiple cross-sectional images obtained by scanning the imaging target in a predetermined direction, a discrimination unit that uses a learning model trained on teacher data including lesion images and the types of lesions corresponding to the lesion images to discriminate the type of lesion for each of the multiple cross-sectional images acquired by the image acquisition unit, and an identification unit that identifies the type of lesion for the imaging target based on the discrimination results for the multiple cross-sectional images by the discrimination unit. The discrimination unit also calculates a probability that each of the multiple cross-sectional images is a specific lesion type, and the identification unit identifies the type of lesion for the imaging target based on the probability calculated for each lesion type by the discrimination unit and the size of the lesion area in each cross-sectional image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-010804 Summary of the Invention

[0004] One embodiment of the technique of the present disclosure provides an image processing device, a medical diagnostic device, an endoscope device, and an image processing method that can suppress erroneous diagnosis of lesions. [Means for solving the problem]

[0005] A first aspect of the technology of the present disclosure includes a processor, the processor acquires a medical image including a lesion area indicating a lesion, and a classifier performs a determination process to determine the type of lesion based on the medical image, the classifier includes a first classifier and a second classifier, the determination process includes a first determination process by the first classifier and a second determination process by the second classifier, and the first determination process is a process to determine whether the type belongs to a first group including a first type classified into a first category and a second type classified into a second category, or whether the type belongs to a second group including a third type classified into the second category. The image processing device is an image processing device that performs at least one of the following: a first output process that outputs a first signal that can identify whether the type is classified into the first category or the second category based on the judgment result of the first judgment process and the judgment result of the second judgment process; and a second output process that outputs a second signal that can identify whether the type is the first type or the second type based on the judgment result of the first judgment process and the judgment result of the second judgment process.

[0006] A second aspect of the technology of the present disclosure is an image processing device according to the first aspect, in which, when a first determination process determines that a type belongs to a first group and a second determination process determines that the type is a second type, the processor outputs, as a first signal, a second category signal that can identify that the type is classified into the second category.

[0007] A third aspect of the technology of the present disclosure is an image processing device according to the first or second aspect, in which, when a first determination process determines that a type belongs to a first group and a second determination process determines that the type is a first type, the processor outputs a first category signal as a first signal that can identify that the type is classified into the first category.

[0008] A fourth aspect of the technology of the present disclosure is an image processing device according to any one of the first to third aspects, in which a processor performs at least one of a display process in which a display device displays first category information indicating that the type is classified into the first category and second category information indicating that the type is classified into the second category in a distinguishable display manner based on the judgment result of the first judgment process and the judgment result of the second judgment process, and a storage process in which the first category information and the second category information are stored in a distinguishable manner on a medical image basis based on the judgment result of the first judgment process and the judgment result of the second judgment process.

[0009] A fifth aspect of the technology of the present disclosure is an image processing device according to the fourth aspect, in which the display mode includes a first mode in which the medical image and first category information are displayed on the display device so as to be contrasted with each other, and a second mode in which the medical image and second category information are displayed on the display device so as to be contrasted with each other.

[0010] A sixth aspect of the technology of the present disclosure is an image processing device according to the fourth or fifth aspect, in which at least one of the first category information and the second category information is information based on a feature map obtained from a classifier.

[0011] A seventh aspect of the technology of the present disclosure is an image processing device according to any one of the first to sixth aspects, in which the second judgment process is a process that uses intermediate features of the first classifier.

[0012] An eighth aspect of the technology of the present disclosure is an image processing device according to any one of the first to seventh aspects, in which the classifier is a neural network, and the first classifier and the second classifier share multiple layers from the input layer onwards.

[0013] A ninth aspect of the technology of the present disclosure is an image processing device according to any one of the first to eighth aspects, in which the first classifier is a model that learns a first lesion image showing a first lesion corresponding to a first type and a second lesion image showing a second lesion corresponding to a second type as images corresponding to a first group, and learns a third lesion image showing a third lesion corresponding to a third type as an image corresponding to a second group.

[0014] A tenth aspect of the technology of the present disclosure is an image processing device according to any one of the first to ninth aspects, in which a processor acquires medical images over a plurality of frames in chronological order, determines for each frame which of a plurality of lesion types including a first type and a second type is the type based on the determination results of a first determination process and a second determination process, outputs a signal corresponding to a second type determination signal indicating that the type has been determined to be the second type when the type for a first frame of the plurality of frames is determined to be the second type, and outputs a signal corresponding to the second type determination signal when the type for a second frame of the plurality of frames within a predetermined number of frames from the first frame is determined to be a lesion type different from the second type.

[0015] An eleventh aspect of the technology of the present disclosure is an image processing device according to any one of the first to tenth aspects, wherein the first category is a category indicating non-tumorousness and the second category is a category indicating tumorousness.

[0016] A twelfth aspect of the technique of the present disclosure is the image processing device according to any one of the first to eleventh aspects, in which the first type is a non-neoplastic lesion.

[0017] A thirteenth aspect of the technique of the present disclosure is the image processing device according to any one of the first to twelfth aspects, in which the second type is a serrated lesion.

[0018] A fourteenth aspect of the technology of the present disclosure is an image processing device according to any one of the first to thirteenth aspects, in which the third type is a tumorous lesion different from a serrated lesion.

[0019] A fifteenth aspect of the technique of the present disclosure includes acquiring a medical image including a lesion area indicating a lesion, and performing a determination process in which a determiner determines the type of lesion based on the medical image, wherein the determiner includes a first determiner and a second determiner, and the determination process includes a first determination process by the first determiner and a second determination process by the second determiner, and the first determination process is a process of determining whether the type belongs to a first group including a first type classified into a first category and a second type classified into a second category, or whether the type belongs to a second group including a third type classified into the second category. The image processing method includes performing at least one of a first output process that outputs a first signal that can identify whether the type is classified into the first category or the second category based on the determination result of the first determination process and the determination result of the second determination process, and a second output process that outputs a second signal that can identify whether the type is the first type or the second type based on the determination result of the first determination process and the determination result of the second determination process.

[0020] A sixteenth aspect of the technology of the present disclosure is an image processing device that includes a processor, which acquires medical images including a lesion area indicating a lesion in a time series across multiple frames, determines for each frame based on the medical images the type of the lesion as one of multiple lesion types including a first type and a second type, and if the type is determined to be the second type for a first frame of the multiple frames, outputs a signal corresponding to a second type determination signal indicating that the type has been determined to be the second type for a second frame of the multiple frames that is within a predetermined number of frames from the first frame, and if the type is determined to be the first type for the first frame, outputs a determination result signal corresponding to the determination result for the type for the second frame.

[0021] A 17th aspect of the technology disclosed herein is an image processing device according to the 16th aspect, in which the category to which the first type belongs is a category indicating non-neoplasticity, and the category to which the second type belongs is a category indicating neoplasticity.

[0022] An 18th aspect of the technology of the present disclosure is an image processing device according to the 16th or 17th aspect, wherein the first type is a type indicating non-tumorous lesions and the second type is a type indicating serrated lesions.

[0023] A 19th aspect of the technology of the present disclosure is an image processing device according to any one of the 16th to 18th aspects, in which a processor performs at least one of a display process that displays information based on a signal corresponding to a second type of determination signal and information based on a determination result signal on a display device on a frame-by-frame basis, and a storage process that stores information based on a signal corresponding to the second type of determination signal and information based on the determination result signal on a frame-by-frame basis.

[0024] A twentieth aspect of the technology of the present disclosure is a medical diagnostic device comprising an image processing device according to any one of the first to fourteenth aspects and the sixteenth to nineteenth aspects, and an imaging device that captures an image of an observation target area of ​​a subject to obtain an image showing the observation target area as a medical image.

[0025] A 21st aspect of the technology of the present disclosure is an endoscopic device comprising an image processing device according to any one of the first to fourteenth aspects and the sixteenth to nineteenth aspects, and an endoscope that is inserted into the body and captures an image of an area to be observed inside the body to obtain an image showing the area to be observed as a medical image. [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 1 is a conceptual diagram showing an example of an aspect in which an endoscope system is used. [Figure 2] 1 is a conceptual diagram showing an example of the overall configuration of an endoscope system. [Figure 3]1 is a conceptual diagram showing an example of an aspect in which an insertion portion of an endoscope is inserted into the large intestine of a subject. [Figure 4] FIG. 2 is a block diagram showing an example of a hardware configuration of the endoscope processing device. [Figure 5] 2 is a block diagram showing an example of main functions of a processor of the endoscope processing device. FIG. [Figure 6] FIG. 3 is a conceptual diagram showing an example of the configuration of a first determiner and a second determiner. [Figure 7] FIG. 4 is a conceptual diagram illustrating an example of processing content of a first learning execution unit. [Figure 8] FIG. 10 is a conceptual diagram illustrating an example of processing content of a second learning execution unit. [Figure 9] FIG. 10 is a conceptual diagram illustrating an example of processing contents of an acquisition unit and a determination unit. [Figure 10] FIG. 10 is a conceptual diagram showing an example of the processing performed by the control unit when the SSL score is equal to or greater than the HP score. [Figure 11] 10 is a conceptual diagram showing an example of processing performed by a control unit on a display device when an SSL score is equal to or greater than an HP score. FIG. [Figure 12] FIG. 10 is a conceptual diagram showing an example of processing performed by a control unit when an SSL score is less than an HP score. [Figure 13] 10 is a conceptual diagram showing an example of processing performed by a control unit on a display device when an SSL score is less than an HP score. FIG. [Figure 14] FIG. 10 is a conceptual diagram showing an example of the processing performed by the control unit when the HP score is less than the NP score. [Figure 15] 10 is a conceptual diagram showing an example of processing performed by a control unit on a display device when an HP score is less than an NP score. FIG. [Figure 16] 10 is a flowchart showing an example of the flow of a first learning process. [Figure 17] 10 is a flowchart showing an example of the flow of a second learning process. [Figure 18A] 10 is a flowchart showing an example of the flow of an inference process. [Figure 18B] This is a continuation of the flowchart shown in FIG. 18A. [Figure 19] FIG. 10 is a conceptual diagram showing an example of processing contents of a control unit according to a first modified example. [Figure 20] FIG. 10 is a conceptual diagram showing an example of the processing content of the control unit according to the second modified example when the SSL score is equal to or greater than the HP score. [Figure 21] FIG. 10 is a conceptual diagram showing an example of the processing content that a control unit according to a second modified example performs on a display device when the SSL score is equal to or greater than the HP score. [Figure 22] FIG. 10 is a conceptual diagram showing an example of the processing performed by the control unit according to the second modified example when the SSL score is less than the HP score. [Figure 23] FIG. 10 is a conceptual diagram showing an example of the processing content that a control unit according to a second modified example performs on a display device when the SSL score is less than the HP score. [Figure 24] FIG. 10 is a conceptual diagram showing an example of the processing performed by the control unit according to the second modified example when the HP score is less than the NP score. [Figure 25] FIG. 10 is a conceptual diagram showing an example of the processing content that a control unit according to a second modified example performs on a display device when an HP score is less than an NP score. [Figure 26] FIG. 11 is a conceptual diagram showing an example of processing contents of a control unit according to a third modified example. [Figure 27] FIG. 11 is a conceptual diagram showing an example of processing contents of a control unit according to a third modified example. [Figure 28] FIG. 11 is a conceptual diagram showing an example of processing contents of a control unit according to a fourth modified example. DETAILED DESCRIPTION OF THE INVENTION

[0027] Hereinafter, exemplary embodiments of an image processing device, a medical diagnostic device, an endoscope device, and an image processing method according to the techniques of the present disclosure will be described with reference to the accompanying drawings.

[0028] First, the terms used in the following description will be explained.

[0029] CPU is an abbreviation for "Central Processing Unit". GPU is an abbreviation for "Graphics Processing Unit". RAM is an abbreviation for "Random Access Memory". NVM is an abbreviation for "Non-volatile memory". EEPROM is an abbreviation for "Electrically Erasable Programmable Read-Only Memory". ASIC is an abbreviation for "Application Specific Integrated Circuit". PLD is an abbreviation for "Programmable Logic Device". FPGA is an abbreviation for "Field-Programmable Gate Array". SoC is an abbreviation for "System-on-a-chip". SSD is an abbreviation for "Solid State Drive". USB is an abbreviation for "Universal Serial Bus". HDD is an abbreviation for "Hard Disk Drive". EL is an abbreviation for "Electro-Luminescence". I / F is an abbreviation for "Interface". CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor". CCD is an abbreviation for "Charge Coupled Device". CT is an abbreviation for "Computed Tomography". MRI is an abbreviation for "Magnetic Resonance Imaging". AI is an abbreviation for "Artificial Intelligence". FIFO is an abbreviation for "First In First Out". SSL is an abbreviation for "Sessile Serrated Lesion". NP is an abbreviation for "Neoplastic Polyp". HP is an abbreviation for "Hyperplastic Polyp". GAP is an abbreviation for "Global Average Pooling". CAM is an abbreviation for "Class Activation Mapping".

[0030] As an example, as shown in FIG. 1 , an endoscopic system 10 includes an endoscopic device 12 and a display device 14. The endoscopic device 12 is used by medical professionals such as a doctor 16, a nurse 17, and / or a technician (hereinafter referred to as a "user"). The endoscopic device 12 includes an endoscope 18 and is an apparatus for performing medical treatment on the inside of a subject 20 (e.g., a patient) via the endoscope 18. The endoscopic device 12 is an example of a "medical diagnostic device" and an "endoscopic device" according to the technology of the present disclosure. The endoscope 18 is an example of an "imaging device" and an "endoscope" according to the technology of the present disclosure.

[0031] The endoscope 18 acquires and outputs images showing the state inside the body by imaging the subject 20 by the doctor 16. In the example shown in Fig. 1, the endoscope 18 is shown inserted into the body cavity from the anus of the subject 20. Note that in the example shown in Fig. 1, the endoscope 18 is inserted into the body cavity from the anus of the subject 20, but this is merely an example, and the endoscope 18 may also be inserted into the body cavity from the mouth, nostril, perforation, or the like of the subject 20, and the location where the endoscope 18 is inserted is determined by the type of endoscope 18, the surgical procedure, etc.

[0032] The display device 14 displays various information including images. Examples of the display device 14 include a liquid crystal display and an EL display. The display device 14 displays a plurality of screens arranged side by side. In the example shown in FIG. 1, a first screen 22 and a second screen 24 are shown as examples of the plurality of screens.

[0033] An endoscopic image 26 is displayed on the first screen 22. The endoscopic image 26 is an example of a "medical image" according to the technology of the present disclosure. The endoscopic image 26 is a circular image showing an observation target region. That is, the endoscopic image 26 is an image acquired by imaging the observation target region inside the body cavity of the subject 20 with the endoscope 18. In the example shown in FIG. 1, an image including a site region 26A and a lesion region 26B is shown as an example of the endoscopic image 26.

[0034] Site region 26A is an image region showing a site contained within the body of subject 20. In the example shown in FIG. 1, an image region showing the inner wall of the large intestine is shown as an example of site region 26A. The inner wall of the large intestine is merely one example, and may be the inner or outer wall of other sites such as the small intestine, duodenum, or stomach. Lesion region 26B is an image region showing a lesion. Examples of lesions include neoplastic polyps (hereinafter referred to as "NP") and non-neoplastic polyps.

[0035] The endoscopic image 26 displayed on the first screen 22 is one frame included in a moving image made up of multiple frames. In other words, the first screen 22 displays multiple frames of the endoscopic image 26 at a predetermined frame rate (e.g., 30 frames / second or 60 frames / second). An example of a moving image is a live-view moving image. The live-view method is merely an example, and a moving image that is temporarily stored in a memory or the like and then displayed, like a post-view moving image, may also be used. Furthermore, each frame included in a moving image for recording stored in a memory or the like may be reproduced and displayed on the first screen 22 as the endoscopic image 26.

[0036] The second screen 24 is smaller than the first screen 22. In the example shown in FIG. 1, it is superimposed on the lower right of the second screen 22 when viewed from the front. Although a superimposed display is illustrated here, this is merely an example and an embedded display is also possible. The display position of the second screen 22 may be anywhere within the screen of the first display device 14, but it is preferable that it be displayed in a position that can be contrasted with the endoscopic image 26. A position identification image 28 is displayed on the second screen 24. The position identification image 28 is an image that corresponds to the endoscopic image 26 and is an image that is referred to by a user or the like to identify the position of the lesion area 26B within the endoscopic image 26.

[0037] The position identification image 28 has an outer frame 28A, a target mark 28B, and a lesion mark 28C. The outer frame 28A is a frame in the shape of a circular frame obtained by reducing the circular outline of the endoscopic image 26, with upper and lower portions cut out by the upper and lower sides of the second screen 24.

[0038] The target mark 28B is a mark that intersects in a cross shape in the center of the display area of ​​the position identification image 28. The intersection of the target mark 28B corresponds to the center point of the endoscopic image 26.

[0039] Lesion mark 28C is a mark corresponding to lesion area 26B in endoscopic image 26, and is displayed in a display mode that corresponds to the size, shape, and type of the lesion indicated by lesion area 26B.

[0040] As shown in FIG. 2 as an example, the endoscope 18 includes a control section 30 and an insertion section 32. The insertion section 32 is tubular. The insertion section 32 has a distal end section 34, a bending section 36, and a flexible section 37. The distal end section 34, the bending section 36, and the flexible section 37 are arranged in this order from the distal end to the proximal end of the insertion section 32. The flexible section 37 is made of a long, flexible material and connects the control section 30 and the bending section 36. The bending section 36 partially bends or rotates around the axis of the insertion section 32 when the control section 30 is operated. As a result, the insertion section 32 is advanced deeper into the body cavity while bending or rotating around the axis of the insertion section 32 in accordance with the shape of the body cavity (e.g., the shape of the digestive tract, such as the esophagus, stomach, duodenum, small intestine, and large intestine, or the shape of the bronchial duct).

[0041] The distal end portion 34 is provided with an illumination device 38, an endoscope 40, and a treatment tool opening 42. The illumination device 38 has an illumination window 38A and an illumination window 38B. The illumination device 38 emits light (for example, white light or near-infrared light composed of three primary colors) through the illumination window 38A and the illumination window 38B. The endoscope 40 captures images of the inside of the body using an optical method. An example of the endoscope 40 is a CMOS camera. The CMOS camera is merely an example, and other types of cameras such as a CCD camera may also be used.

[0042] The treatment tool opening 42 is an opening for allowing the treatment tool 44 to protrude from the distal end portion 34. A treatment tool insertion port 46 is formed in the operation unit 30, and the treatment tool 44 is inserted into the insertion section 32 from the treatment tool insertion port 46. The treatment tool 44 passes through the insertion section 32 and protrudes to the outside from the treatment tool opening 42. In the example shown in FIG. 2, a puncture needle 48 protrudes from the treatment tool opening 42 as the treatment tool 44. Here, the puncture needle 48 is illustrated as the treatment tool 44, but this is merely one example, and the treatment tool 44 may also be a grasping forceps, a scalpel, a snare, and / or a puncture needle with a guide sheath. The treatment tool opening 42 also functions as a suction port for sucking blood, internal waste, and the like.

[0043] The endoscope device 12 includes a universal cord 50, an endoscope processing device 52, and a light source device 54. The endoscope processing device 52 is an example of an "image processing device" according to the technology of the present disclosure.

[0044] The universal cord 50 has a base end 50A, a first tip end 50B, and a second tip end 50C. The base end 50A is connected to the operation unit 30. The first tip end 50B is connected to an endoscope processing device 52. The second tip end 50C is connected to a light source device 54.

[0045] The endoscope system 10 includes a reception device 56. The reception device 56 receives instructions from a user. Examples of the reception device 56 include an operation panel having a plurality of hard keys and / or a touch panel, a keyboard, a mouse, a trackball, a foot switch, a smart device, and / or a microphone.

[0046] A reception device 56 is connected to the endoscope processing device 52. The endoscope processing device 52 exchanges various signals with the endoscope 40 and controls the light source device 54 in accordance with instructions received by the reception device 56. The endoscope processing device 52 causes the endoscope 40 to capture an image, and acquires and outputs an endoscopic image 26 (see FIG. 1 ) from the endoscope 40. The light source device 54 emits light under the control of the endoscope processing device 52 and supplies the light to the illumination device 38. A light guide is built into the illumination device 38, and the light supplied from the light source device 54 is irradiated from illumination windows 38A and 38B via the light guide.

[0047] The display device 14 is connected to the endoscope processing device 52. The endoscope processing device 52 controls the display device 14 in accordance with instructions received by the reception device 56. The endoscope processing device 52 causes the display device 14 to display an endoscopic image 26 obtained by capturing an image of the observation area with the endoscope 40 (see FIG. 1).

[0048] As an example, as shown in FIG. 3 , the insertion section 32 of the endoscope 18 is inserted into the large intestine 58 from the anus of the subject 20. The endoscope 40 generates an endoscopic image 26 by capturing images of the inside of the large intestine 58. The endoscopic image 26 is generated as an image showing the appearance of the inner wall 58A. For example, the endoscope 40 inserted into the large intestine 58 advances from the entrance side to the back side of the large intestine 58, and captures images of the inner wall 58A from the entrance side to the back side of the large intestine 58 using a live view method at a predetermined frame rate. As a result, a moving image including multiple frames of the endoscopic image 26 showing the appearance of the inner wall 58A from the entrance side to the back side of the large intestine 58 is generated.

[0049] 4, the endoscope processing device 52 includes a computer 60 and an input / output interface 62. The computer 60 includes a processor 64, a RAM 66, and an NVM 68. The input / output interface 62, the processor 64, the RAM 66, and the NVM 68 are connected to a bus 70.

[0050] The processor 64 is an example of a "processor" according to the technology of the present disclosure. For example, the processor 64 has a CPU and a GPU, and controls the entire endoscope processing device 52. The GPU operates under the control of the CPU and is responsible for executing various graphic-related processes. The processor 64 may be one or more CPUs that have integrated GPU functionality, or one or more CPUs that do not have integrated GPU functionality.

[0051] The RAM 66 is a memory that temporarily stores information and is used as a work memory by the processor 64. The NVM 68 is a nonvolatile storage device that stores various programs, various parameters, etc. An example of the NVM 68 is a flash memory (for example, an EEPROM and / or an SSD). Note that the flash memory is merely one example, and the NVM 68 may be another nonvolatile storage device such as an HDD, or may be a combination of two or more types of nonvolatile storage devices.

[0052] The input / output interface 62 is connected to the reception device 56, and the processor 64 acquires instructions received by the reception device 56 via the input / output interface 62 and executes processing according to the acquired instructions.

[0053] The endoscope 40 is also connected to the input / output interface 62. The processor 64 controls the endoscope 40 via the input / output interface 62, and acquires, via the input / output interface 62, an endoscopic image 26 (see FIGS. 1 and 3) obtained by imaging the inside of the body of the subject 20 with the endoscope 40.

[0054] The light source device 54 is also connected to the input / output interface 62. The processor 64 controls the light source device 54 via the input / output interface 62 to supply light to the illumination device 38 and adjust the amount of light supplied to the illumination device 38.

[0055] Furthermore, the display device 14 is connected to the input / output interface 62, and the processor 64 controls the display device 14 via the input / output interface 62 to display various information on the display device 14. For example, the processor 64 acquires an endoscopic image 26 (see FIGS. 1 and 3) from the endoscope 40, and displays the endoscopic image 26 on the display device 14 (see FIG. 1). Also, for example, the processor 64 generates a position identification image 28, and displays the position identification image 28 on the display device 14 (see FIG. 1).

[0056] Meanwhile, SSL, a serrated lesion, is known as a lesion occurring in the large intestine 58. SSL is a type of lesion belonging to NP and has a pathologically sawtooth structure. In endoscopic diagnosis, SSL is differentiated as a lesion requiring treatment, so it is necessary to prevent it from being mistakenly differentiated as HP, a hyperplastic polyp that has a similar appearance to SSL. However, if a doctor 16 relies solely on his or her own knowledge and experience to distinguish whether or not a lesion is SSL, there is a risk that he or she will mistakenly differentiate SSL for HP or vice versa. Therefore, the use of AI-based image recognition processing is currently being considered to assist doctors 16 in making highly accurate distinctions. However, if a classifier used in AI-based image recognition processing is created to determine whether a lesion is NP or HP, there is a risk that SSL will be mistakenly identified as HP. Because SSL is a lesion requiring treatment, it needs to be classified into the neoplastic category to avoid being mistakenly identified as HP.

[0057] In view of the above circumstances, in this embodiment, as shown in Fig. 5 as an example, a first learning process, a second learning process, and an inference process are performed by a processor 64 in an endoscope processing device 52. A first learning process program 72, a second learning process program 74, an inference process program 76, and a neural network 78 are stored in an NVM 68.

[0058] The processor 64 performs the first learning process by reading the first learning process program 72 from the NVM 68 and executing the read first learning process program 72 on the RAM 66. The first learning process is realized by the processor 64 operating as a first learning execution unit 64A in accordance with the first learning process program 72.

[0059] The processor 64 performs the second learning process by reading the second learning process program 74 from the NVM 68 and executing the read second learning process program 74 on the RAM 66. The second learning process is realized by the processor 64 operating as a second learning execution unit 64B in accordance with the second learning process program 74.

[0060] The processor 64 performs inference processing by reading the inference processing program 76 from the NVM 68 and executing the read inference processing program 76 on the RAM 66. The inference processing is realized by the processor 64 operating as an acquisition unit 64C, a determination unit 64D, and a control unit 64E in accordance with the inference processing program 76.

[0061] The neural network 78 is an example of a "determiner" and a "neural network" according to the technology of the present disclosure. In this embodiment, a convolutional neural network is used as the neural network 78. As shown in FIG. 6 as an example, the neural network 78 includes a first determinator 80 and a second determinator 82. The first determinator 80 has an input layer 84A, first to fifth convolutional layers 84B to 84F, a gap layer 84G, a fully connected layer 84H, and an output layer 84I. The second determinator 82 has an input layer 84A, first to fourth convolutional layers 84B to 84E, a sixth convolutional layer 84J, a gap layer 84K, a fully connected layer 84L, and an output layer 84M. In this way, the first determinator 80 and the second determinator 82 share multiple layers from the input layer 84 onwards. 6, the first determinator 80 and the second determinator 82 share the input layer 84A and the first to fourth convolutional layers 84B to 84E. In the following, unless it is necessary to distinguish between the first to fifth convolutional layers 84B to 84F and the sixth convolutional layer 84J, they will be referred to as "convolutional layers" without being assigned reference numerals.

[0062] In the first determiner 80, an endoscopic image 26 is input to the input layer 84A. For example, a lesion region 26B included in the endoscopic image 26 is broadly divided into an HP region 26B1, an NP region 26B2, and an SSL region 26B3. The HP region 26B1 is an image region indicating an HP appearing in the endoscopic image 26. The NP region 26B2 is an image region indicating an NP (here, as an example, a neoplastic polyp different from an SSL) appearing in the endoscopic image 26. The SSL region 26B3 is an image region indicating an SSL appearing in the endoscopic image 26. Note that an HP is an example of a "first type" and a "first lesion" according to the technology of the present disclosure, an SSL is an example of a "second type" and a "second lesion" according to the technology of the present disclosure, and an NP is an example of a "third type" and a "third lesion" according to the technology of the present disclosure.

[0063] The convolution layer performs convolution processing. In convolution processing, data related to the endoscopic image 26 (for example, a feature map, which is an image in which specified features are activated) is provided from the previous layer, and the feature data is condensed by performing filter processing on the data related to the endoscopic image 26, and the condensed feature data is output to the next layer. In the convolution layer, multiple filter processes are performed, and a unique filter is used for each filter process. Within the convolution layer, the data related to the endoscopic image 26 is multiplied by the filter while shifting the filter by a predetermined amount (for example, in units of one pixel) with respect to the data related to the endoscopic image 26, and the result of the multiplication (i.e., feature data) is expressed as a feature map. For example, in the feature map, portions of the data related to the endoscopic image 26 that react more strongly to the filter are expressed with larger values.

[0064] Between the convolution layers, there is a pooling layer (not shown), which performs pooling processing. The pooling processing is a process of reducing the feature map obtained by the convolution layer and outputting the reduced feature map to the next stage. Here, reduction refers to a process of reducing the amount of data while retaining important data (for example, the maximum value among 2 × 2 pixels). In other words, the pooling layer reduces the feature map so that the resolution gradually decreases from the input layer 84A side to the output layers 84I and 84M side.

[0065] The GAP layer 84G receives multiple feature maps from the previous layer, the fifth convolutional layer 84F. The GAP layer 84G calculates the average value for each feature map received from the fifth convolutional layer 84F and outputs the calculated average value to the fully connected layer 84H.

[0066] The fully connected layer 84H performs a fully connected process. The fully connected process of the fully connected layer 84H is a process of performing a convolution operation (e.g., weighted averaging) on ​​all nodes in the output layer 84I using weights specific to each corresponding feature map for multiple average values ​​input from the GAP layer 84G. All nodes in the output layer 84I refer to nodes corresponding to the HP class and nodes corresponding to the NP class. The HP class is the class to which the HP region 26B1 belongs, and the NP class is the class to which the NP region 26B2 belongs.

[0067] The output layer 84I calculates the HP score 86 and the NP score 88 by using a softmax function 84I1 as an activation function. Although the softmax function 84I1 is exemplified here, this is merely an example, and other activation functions that can realize input and output similar to that of the softmax function 84I1 may be used.

[0068] The HP score 86 is a class score for the HP class, and the NP score 88 is a class score for the NP class. Note that the HP score 86 and the NP score 88 may be class-activated scores. Class activation refers to a process of converting a class score expressed as a decimal into "0.0" or "1.0" based on a threshold value (e.g., 0.8).

[0069] In the second determiner 82, the sixth convolutional layer 84J receives as input a plurality of feature maps from the preceding fourth convolutional layer 84E. The sixth convolutional layer 84J performs convolution processing on the plurality of feature maps from the fourth convolutional layer 84E to generate new plurality of feature maps and output them to the GAP layer 84K.

[0070] The GAP layer 84K receives a plurality of feature maps from the preceding sixth convolutional layer 84J. The GAP layer 84K calculates the average value for each feature map received from the sixth convolutional layer 84J and outputs the calculated average value to the fully connected layer 84L.

[0071] The fully connected layer 84L performs a fully connected process. The fully connected process of the fully connected layer 84L is a process in which a convolution operation (e.g., weighted averaging) is performed on all nodes in the output layer 84M using weights specific to each corresponding feature map for multiple average values ​​input from the GAP layer 84K. All nodes in the output layer 84M refer to nodes corresponding to the HP class and nodes corresponding to the SSL class. The SSL class is the class to which the SSL area 26B3 belongs.

[0072] The output layer 84M calculates the HP score 90 and the SSL score 92 by using a softmax function 84M1 as an activation function. Although the softmax function 84M1 is exemplified here, this is merely an example, and other activation functions that can realize input and output similar to that of the softmax function 84M1 may be used.

[0073] The HP score 90 is a class score for the HP class, and the SSL score 92 is a class score for the SSL class. Note that the HP score 90 and the SSL score 92 may be class activated scores.

[0074] Next, an example of the contents of the first learning process and the second learning process will be described with reference to FIGS.

[0075] As an example, as shown in Figure 7, a data supply device 94 is used for the first learning execution unit 64A. An example of the data supply device 94 is a computer. The data supply device 94 holds training data 96 and supplies the training data 96 to the first learning execution unit 64A. The training data 96 includes multiple frames of training images 98 and multiple pieces of supervised answer data 100. One piece of supervised answer data 100 is associated with each of the multiple frames of training images 98.

[0076] The first classifier 80 is a model trained on a plurality of training images 98. The training images 98 are broadly divided into HP training images 98A and NP training images 98B. The HP training images 98A are an example of a "first lesion image" according to the technology of the present disclosure. The NP training images 98B are an example of a "third lesion image" according to the technology of the present disclosure. The HP is an example of a "first lesion corresponding to the first type" according to the technology of the present disclosure.

[0077] The HP training image 98A is an endoscopic image 26 that shows an HP or SSL. Here, an endoscopic image 26 that shows an HP or SSL is shown as the HP training image 98A, but this is merely an example, and an endoscopic image 26 that shows a non-neoplastic polyp such as a hamartomatous polyp and / or an inflammatory polyp may also be used as the HP training image 98A. The NP training image 98B is an endoscopic image 26 that shows a lesion that belongs to the NP other than an SSL.

[0078] The correct answer data 100 includes an HP score 100A and an NP score 100B. The HP score 100A included in the correct answer data 100 associated with the HP learning image 98A is "1.0," and the NP score 100B included in the correct answer data 100 associated with the HP learning image 98A is "0.0." Furthermore, the HP score 100A included in the correct answer data 100 associated with the NP learning image 98B is "0.0," and the NP score 100B included in the correct answer data 100 associated with the NP learning image 98B is "1.0."

[0079] Because the training data 96 is created in this manner, the endoscopic image 26 showing an HP and the endoscopic image 26 showing an SSL are used as images corresponding to one group (i.e., a group to which similar neoplastic and non-neoplastic lesions belong) for training the first classifier 80. Also, the endoscopic image 26 showing a lesion other than SSL that belongs to NP (i.e., NP training image 98B) is used as an image corresponding to one group (i.e., a group to which lesions other than SSL that are classified as NP belong) for training the first classifier 80.

[0080] The first learning execution unit 64A has a first determinator 80. A learning image 98 is supplied to the first learning execution unit 64A. The first learning execution unit 64A inputs the learning image 98 supplied from the data supply device 94 to an input layer 84A of the first determinator 80. In the first determinator 80, the learning image 98 input to the input layer 84A is subjected to various processes in the first to fifth convolutional layers 84B to 84F, the GAP layer 84G, the fully connected layer 84H, and the output layer 84I, and an HP score 86 and an NP score 88 corresponding to the learning image 98 are output from the output layer 84I.

[0081] The first learning execution unit 64A acquires, from the data supply device 94, supervised answer data 100 associated with the learning image 98 that was the subject of judgment by the first determinator 80 (i.e., the learning image 98 input to the input layer 84A). The first learning execution unit 64A then calculates an HP score error 102, which is the error between an HP score 100A included in the supervised answer data 100 acquired from the data supply device 94 and the HP score 86 output from the output layer 84I. The first learning execution unit 64A also calculates an NP score error 104, which is the error between an NP score 100B included in the supervised answer data 100 acquired from the data supply device 94 and the NP score 88 output from the output layer 84I.

[0082] The first learning execution unit 64A calculates a plurality of adjustment values ​​106 that minimize both the HP score error 102 and the NP score error 104. Then, the first learning execution unit 64A optimizes the first determinator 80 by adjusting a plurality of optimization variables in the first determinator 80 using the plurality of adjustment values ​​106 so as to minimize both the HP score error 102 and the NP score error 104. Examples of the plurality of optimization variables in the first determinator 80 include coefficients of filters used in the first to fifth convolutional layers 84B to 84F, and connection weights and offset values ​​used in the fully connected layer 84H.

[0083] 8, a data supply device 94 is used for the second learning execution unit 64B as well as for the first learning execution unit 64A. The data supply device 94 holds training data 108 and supplies the training data 108 to the second learning execution unit 64B. The training data 108 includes multiple frames of training images 110 and multiple pieces of supervised answer data 112. One piece of supervised answer data 112 is associated with each of the multiple frames of training images 110.

[0084] The training images 110 are broadly divided into HP training images 110A and SSL training images 110B. The HP training image 110A is an endoscopic image 26 showing an HP. While the HP training image 98A is an endoscopic image 26 showing either an HP or an SSL, the HP training image 110A differs from the HP training image 98A in that it is an endoscopic image 26 showing an HP but not an SSL. Here, an endoscopic image 26 showing an HP is illustrated as the HP training image 110A, but this is merely an example, and an endoscopic image 26 showing a non-neoplastic polyp such as a hamartomatous polyp and / or an inflammatory polyp may also be used as the HP training image 110A. The SSL training image 110B is an endoscopic image 26 showing an SSL.

[0085] The supervised answer data 112 includes an HP score 112A and an SSL score 112B. The HP score 112A included in the supervised answer data 112 associated with the HP training image 110A is "1.0," and the SSL score 112B included in the supervised answer data 112 associated with the HP training image 110A is "0.0." Furthermore, the HP score 112A included in the supervised answer data 112 associated with the SSL training image 110B is "0.0," and the SSL score 112B included in the supervised answer data 112 associated with the SSL training image 110B is "1.0."

[0086] The second learning execution unit 64B has a second determiner 82. The second learning execution unit 64B is supplied with learning images 110. The second learning execution unit 64B inputs the learning images 110 supplied from the data supply device 94 to the input layer 84A of the second determiner 82. In the second determiner 82, various processes are performed on the learning images 110 input to the input layer 84A in the first to fourth convolutional layers 84B to 84E, the sixth convolutional layer 84J, the GAP layer 84K, the fully connected layer 84L, and the output layer 84M, and an HP score 90 and an SSL score 92 corresponding to the learning images 110 are output from the output layer 84M.

[0087] The second learning execution unit 64B acquires, from the data supply device 94, supervised answer data 112 associated with the learning image 110 that was the subject of judgment by the second determinator 82 (i.e., the learning image 110 input to the input layer 84A). The second learning execution unit 64B then calculates an HP score error 114, which is the error between the HP score 112A included in the supervised answer data 112 acquired from the data supply device 94 and the HP score 90 output from the output layer 84M. The second learning execution unit 64B also calculates an SSL score error 116, which is the error between the SSL score 112B included in the supervised answer data 112 acquired from the data supply device 94 and the SSL score 92 output from the output layer 84M.

[0088] The second learning execution unit 64B calculates a plurality of adjustment values ​​118 that minimize both the HP score error 114 and the SSL score error 116. The second learning execution unit 64B then optimizes the second determinator 82 by adjusting a plurality of optimization variables in the second determinator 82 using the plurality of adjustment values ​​118 so as to minimize both the HP score error 114 and the SSL score error 116. Examples of the plurality of optimization variables in the second determinator 82 include filter coefficients used in the sixth convolutional layer 84J, and connection weights and offset values ​​used in the fully connected layer 84L. That is, in order to ensure the accuracy of the HP score 86 and the NP score 88 output from the first determinator 80, learning of the second determinator 82 may be performed while the optimization variables included in the first to fourth convolutional layers 84B to 84E that the second determinator 82 shares with the first determinator 80 are fixed.

[0089] Next, an example of the contents of the inference process will be described with reference to FIGS.

[0090] As an example, as shown in Fig. 9, the acquisition unit 64C acquires an endoscopic image 26 from the endoscope 40. Since the endoscopic image 26 is obtained by the endoscope 40 using a live view method, the acquisition unit 64C acquires the endoscopic image 26 in time series across multiple frames from the endoscope 40. The determination unit 64D processes the endoscopic image 26 acquired by the acquisition unit 64C for each frame. Note that in the description using Figs. 9 to 16, a case where inference processing is performed on one frame will be illustrated as an example to facilitate understanding of the technology of the present disclosure.

[0091] The determination unit 64D performs processing using a neural network 78 that has been optimized by performing the first learning processing and the second learning processing. The neural network 78 performs a determination processing based on the endoscopic image 26 acquired by the acquisition unit 64C. The determination processing is processing in which a first determiner 80 and a second determiner 82 determine the type of lesion (hereinafter also simply referred to as "lesion type") that appears in the endoscopic image 26. The determination processing includes a first determination processing by the first determiner 80 and a second determination processing by the second determiner 82.

[0092] The first determination process is a process for determining whether the type of lesion belongs to the first group 120 or the second group 122. The first group 120 is a group including an endoscopic image 26 showing an HP and an endoscopic image 26 showing an SSL. The second group 122 is a group including an endoscopic image 26 showing an NP. An HP is a lesion classified into the non-tumor category 124, and SSL and NP are lesions classified into the tumor category 126. The non-tumor category 124 is a category indicating non-neoplasticity, and the tumor category 126 is a category indicating neoplasticity. In the first determination process, if the HP score 86 is equal to or greater than the NP score 88, the type of lesion is determined to belong to the first group 120. In addition, in the first determination process, if the HP score 86 is less than the NP score 88, the type of lesion is determined to belong to the second group 122.

[0093] The second determination process is a process that uses the intermediate features of the first determiner 80. This is because part of the second determiner 82 (i.e., the input layer 84A and the first to fourth convolutional layers 84B to 84E) is shared with the first determiner 80. The second determination process is performed on the premise that the second determination process has determined that the type of lesion belongs to the first group 120. The second determination process is a process that determines whether the type of lesion is HP or SSL. In the second determination process, if the SSL score 92 is equal to or greater than the HP score 90, the type of lesion is determined to be SSL, and if the SSL score 92 is less than the HP score 90, the type of lesion is determined to be HP.

[0094] HP is an example of a "non-neoplastic lesion" and a "type indicating non-neoplastic lesion" according to the technology of the present disclosure. SSL is an example of a "serrated lesion" and a "type indicating serrated lesion" according to the technology of the present disclosure. NP is an example of a "neoplastic lesion other than a serrated lesion" according to the technology of the present disclosure. The first group 120 is an example of a "first group" according to the technology of the present disclosure. The second group 122 is an example of a "second group" according to the technology of the present disclosure. The non-neoplastic category 124 is an example of a "first category" and a "category indicating non-neoplastic lesion" according to the technology of the present disclosure. The neoplastic category 126 is an example of a "second category" and a "category indicating neoplastic lesion" according to the technology of the present disclosure.

[0095] As an example, as shown in FIG. 10 , an endoscopic image 26 acquired by an acquisition unit 64C is input to a first determiner 80 and a second determiner 82, whereby an HP score 86 and an NP score 88 are output from the first determiner 80, and an HP score 90 and an SSL score 92 are output from the second determiner 82. Here, if the HP score 86 is equal to or greater than the NP score 88 and the SSL score 92 is equal to or greater than the HP score 90, the control unit 64E generates a tumor location image signal 130 using the feature amount map 128 of the second determiner 82. The case where the HP score 86 is equal to or greater than the NP score 88 and the SSL score 92 is equal to or greater than the HP score 90 means that the type of lesion is determined to belong to a first group 120 (see FIG. 9 ) and is determined to be SSL. The tumor location image signal 130 is an example of a “first signal” and a “second category signal” according to the technology of the present disclosure.

[0096] The feature map 128 is obtained from the sixth convolutional layer 84J. The feature map 128 is an example of a "feature map" according to the technology of the present disclosure. The control unit 64E acquires, from the feature map 128, a high response region 128A, which is a region that has the highest degree of response to the filter used to obtain the feature map 128. The high response region 128A is obtained by binarizing the feature map 128 using a threshold value.

[0097] The control unit 64E extracts the contour 128A1 of the high response region 128A from the high response region 128A. The high response region 128A is a rectangular region or a region formed by combining multiple rectangular regions. Therefore, the control unit 64E generates a curved region 128A2 by shaping the contour 128A1 into a curved shape. For example, in this case, the curved region 128A2 is generated by circumscribing a curve around multiple vertices included in the contour 128A1. That is, the curved region 128A2 is a closed region whose contour is a curve that circumscribing the contour 128A1. The control unit 64E applies a color to the curved region 128A2. In the example shown in FIG. 10, the curved region 128A2 is colored yellow.

[0098] Control unit 64E generates position identification image 28 by processing feature amount map 128 having colored curved region 128A2. Because position identification image 28 is generated by processing feature amount map 128, position identification image 28 can be said to be information based on feature amount map 128. In the example shown in FIG. 10, tumor position identification image 134 is shown as position identification image 28. Tumor mark 132 is applied to tumor position identification image 134 as lesion mark 28C. Tumor mark 132 is colored curved region 128A2, i.e., curved region 128A2 colored yellow.

[0099] Control unit 64E generates tumor location image signal 130, which is a signal indicating tumor location identification image 134, and outputs generated tumor location image signal 130 to a specific output destination (e.g., display device 14, etc.). Here, the process of outputting tumor location image signal 130 to a specific output destination is an example of a "first output process" according to the technology of the present disclosure.

[0100] In this embodiment, the yellow color applied to curved region 128A2 indicates that the lesion is classified into tumor category 126. Therefore, tumor position image signal 130 is used as a signal that can identify that the type of lesion is classified into tumor category 126. Furthermore, in this embodiment, the position of curved region 128A2 in feature map 128 corresponds to the position of the SSL in endoscopic image 26. Therefore, tumor position image signal 130 is also used as a signal that can identify the position of the SSL in endoscopic image 26.

[0101] 11, as an example, control unit 64E displays endoscopic image 26 acquired by acquisition unit 64C, i.e., endoscopic image 26 input to first determiner 80 and second determiner 82 shown in FIG. 10, on first screen 22. Control unit 64E also outputs tumor location image signal 130 (see FIG. 10) to display device 14, thereby displaying tumor location identifying image 134 indicated by tumor location image signal 130 on second screen 24. That is, tumor location identifying image 134 is displayed on second screen 24 as information indicating that the lesion shown in endoscopic image 26 displayed on first screen 22 is classified into tumor category 126. Tumor location identifying image 134 is an example of "second category information" according to the technology of the present disclosure.

[0102] 12, when the HP score 86 is equal to or greater than the NP score 88 and the SSL score 92 is less than the HP score 90, the control unit 64E generates a non-tumor position image signal 136 using the feature amount map 128 of the second determiner 82. When the HP score 86 is equal to or greater than the NP score 88 and the SSL score 92 is less than the HP score 90, this means that the type of lesion is determined to belong to the first group 120 (see FIG. 9) and is determined to be HP. The non-tumor position image signal 136 is an example of the "first signal" and "first category signal" according to the technology of the present disclosure.

[0103] The control unit 64E acquires, from the feature amount map 128, a high response region 128B that is a region that has the highest degree of response to the filter used to obtain the feature amount map 128. The high response region 128B is acquired by binarizing the feature amount map 128 using a threshold value.

[0104] The control unit 64E extracts the contour 128B1 of the high response region 128B from the high response region 128B. Like the high response region 128A, the high response region 128B is a rectangular region or a region formed by combining multiple rectangular regions. Therefore, the control unit 64E generates the curved region 128B2 by shaping the contour 128B1 into a curved shape in the same manner as when generating the curved region 128A2. The control unit 64E applies a color to the curved region 128B2. In the example shown in FIG. 12, the curved region 128B2 is colored green.

[0105] Control unit 64E processes feature map 128 having colored curved region 128B2 to generate position identification image 28. In the example shown in FIG. 12, non-tumor position identification image 140 is shown as position identification image 28. Non-tumor mark 138 is applied to non-tumor position identification image 140 as lesion mark 28C. Non-tumor mark 138 is colored curved region 128B2, i.e., curved region 128B2 colored green.

[0106] The control unit 64E generates a non-tumor position image signal 136, which is a signal indicating the non-tumor position identifying image 140, and outputs the generated non-tumor position image signal 136 to a specific output destination (e.g., the display device 14, etc.). Here, the process of outputting the non-tumor position image signal 136 to the specific output destination is an example of a "first output process" according to the technology of the present disclosure.

[0107] In this embodiment, the green color applied to the curved region 128B2 indicates that the lesion is classified into the non-tumor category 124. Therefore, the non-tumor position image signal 136 is used as a signal that can identify that the type of lesion is classified into the non-tumor category 124. Furthermore, in this embodiment, the position of the curved region 128B2 in the feature map 128 corresponds to the position of the HP in the endoscopic image 26. Therefore, the non-tumor position image signal 136 is also used as a signal that can identify the position of the HP in the endoscopic image 26.

[0108] 13 as an example, the control unit 64E displays the endoscopic image 26 acquired by the acquisition unit 64C, i.e., the endoscopic image 26 input to the first determiner 80 and the second determiner 82 shown in FIG. 12, on the first screen 22. The control unit 64E also outputs the non-tumor position image signal 136 (see FIG. 12) to the display device 14, thereby displaying the non-tumor position identifying image 140 indicated by the non-tumor position image signal 136 on the second screen 24. That is, the non-tumor position identifying image 140 is displayed on the second screen 24 as information indicating that the lesion shown in the endoscopic image 26 displayed on the first screen 22 is classified into the non-tumor category 124. The non-tumor position identifying image 140 is an example of the "first category information" according to the technology of the present disclosure.

[0109] 14, when the HP score 86 is less than the NP score 88, the control unit 64E generates a tumor position image signal 144 using the feature amount map 142 of the first determiner 80. When the HP score 86 is less than the NP score 88, this means that the type of the lesion is determined to belong to the second group 122 (see FIG. 9) and is determined to be NP.

[0110] The feature map 142 is obtained from the fifth convolutional layer 84F. The control unit 64E acquires, from the feature map 142, a high response region 142A, which is a region that has the highest degree of response to the filter used to obtain the feature map 142. The high response region 142A is obtained by binarizing the feature map 142 using a threshold value.

[0111] Control unit 64E extracts contour 142A1 of high response region 142A from high response region 142A. Control unit 64E generates curved region 142A2 by shaping contour 142A1 into a curved shape, and applies color to generated curved region 142A2. In the example shown in Figure 14, similar to the example shown in Figure 10, curved region 142A2 is colored yellow.

[0112] Control unit 64E processes feature amount map 142 having colored curved region 142A2 to generate position identification image 28. In the example shown in FIG. 14, tumor position identification image 148 is shown as position identification image 28. Tumor mark 146 is applied to tumor position identification image 148 as lesion mark 28C. Tumor mark 146 is colored curved region 142A2, i.e., curved region 142A2 colored yellow.

[0113] Control unit 64E generates tumor location image signal 144, which is a signal indicating tumor location specifying image 148, and outputs generated tumor location image signal 144 to a specific output destination (for example, display device 14, etc.).

[0114] In this embodiment, the yellow color applied to curved region 144A2 indicates that the lesion is classified into tumor category 126, as in the example shown in Fig. 10. Therefore, tumor position image signal 144 is used as a signal that can identify that the type of lesion is classified into tumor category 126. Furthermore, in this embodiment, the position of curved region 142A2 in feature map 142 corresponds to the position of NP in endoscopic image 26. Therefore, tumor position image signal 144 is also used as a signal that can identify the position of NP in endoscopic image 26.

[0115] 15 as an example, control unit 64E displays endoscopic image 26 acquired by acquisition unit 64C, i.e., endoscopic image 26 input to first determiner 80 shown in FIG. 14, on first screen 22. Control unit 64E also outputs tumor position image signal 144 (see FIG. 14) to display device 14, thereby displaying tumor position identifying image 148 indicated by tumor position image signal 144 on second screen 24. That is, tumor position identifying image 148 is displayed on second screen 24 as information indicating that the lesion shown in endoscopic image 26 displayed on first screen 22 is classified into tumor category 126.

[0116] 11, 13, and 15, the control unit 64E performs display processing to cause the display device 14 to display the tumor location identifying images 134 and 148 and the non-tumor location identifying image 140 in a distinguishable display manner based on the determination results of the first determination processing and the second determination processing. The tumor location identifying images 134 and 148 are images indicating that the lesion shown in the endoscopic image 26 is classified into the tumor category 126, and the non-tumor location identifying image 140 is an image indicating that the lesion shown in the endoscopic image 26 is classified into the non-tumor category 124. The control unit 64E also displays the tumor location identifying images 134 and 148 in a first manner and the non-tumor location identifying image 140 in a second manner. The first manner refers to a manner in which the endoscopic image 26 and the tumor location identifying image 134 or 148 are displayed in a manner that allows them to be contrasted (see FIGS. 11 and 15). The second mode refers to a mode in which the endoscopic image 26 and the non-tumor position identifying image 140 are displayed so as to be comparable with each other (see FIG. 13).

[0117] Next, an example of the flow of the first learning process, the second learning process, and the inference process performed by the processor 64 of the endoscope processing device 52 will be described with reference to Figures 16 to 18B. The flow of the process shown in the flowcharts in Figures 16 to 18B is an example of an "image processing method" according to the technique of the present disclosure.

[0118] First, an example of the flow of the first learning process performed by the processor 64 of the endoscope processing device 52 when an instruction to start execution of the first learning process is accepted by the acceptance device 56 will be described with reference to FIG.

[0119] In the first learning process shown in FIG. 16, first, in step ST10, the first learning execution unit 64A acquires learning images 98 from the data supply device 94. Here, the learning images 98 acquired from the data supply device 94 are learning images 98 that have not been used in the processes from step ST12 onwards. The first learning execution unit 64A inputs the learning images 98 acquired from the data supply device 94 to the input layer 84A of the first determinator 80 (see FIG. 7). After the process of step ST10 has been executed, the first learning process proceeds to step ST12.

[0120] In step ST12, the first learning execution unit 64A acquires the HP score 86 and the NP score 88 from the first determinator 80 (see FIG. 7). After the processing of step ST12 is executed, the first learning processing proceeds to step ST14.

[0121] In step ST14, the first learning execution unit 64A acquires, from the data supply device 94, the supervised answer data 100 associated with the learning image 98 input to the input layer 84A in step ST10 (see FIG. 7). After the processing of step ST14 is executed, the first learning processing proceeds to step ST16.

[0122] In step ST16, the first learning execution unit 64A calculates an HP score error 102 from the HP score 86 acquired in step ST12 and the HP score 100A included in the supervised data 100 acquired in step ST14 (see FIG. 7). The first learning execution unit 64A also calculates an NP score error 104 from the NP score 88 acquired in step ST12 and the NP score 100B included in the supervised data 100 acquired in step ST14 (see FIG. 7). After the processing of step ST16 is executed, the first learning processing proceeds to step ST18.

[0123] In step ST18, the first learning execution unit 64A calculates a plurality of adjustment values ​​106 that minimize both the HP score error 102 and the NP score error 104 calculated in step ST16 (see FIG. 7). After the processing of step ST18 is executed, the first learning processing proceeds to step ST20.

[0124] In step ST20, the first learning execution unit 64A optimizes the first determinator 80 by adjusting multiple optimization variables in the first determinator 80 using multiple adjustment values ​​106 so as to minimize both the HP score error 102 and the NP score error 104 (see FIG. 7). After the processing of step ST20 is executed, the first learning processing proceeds to step ST22.

[0125] In step ST22, the first learning execution unit 64A determines whether a condition for terminating the first learning process (hereinafter referred to as the "first learning process termination condition") has been satisfied. A first example of the first learning process termination condition is a condition that learning using all learning images 98 included in the teacher data 96 has been completed. A second example of the first learning process termination condition is a condition that an instruction to terminate the first learning process has been received by the reception device 56. If the first learning process termination condition is not satisfied in step ST22, the determination is negative, and the first learning process proceeds to step ST10. If the first learning process termination condition is satisfied in step ST22, the determination is positive, and the first learning process ends.

[0126] Next, an example of the flow of the second learning process performed by the processor 64 of the endoscope processing device 52 when an instruction to start the execution of the second learning process is accepted by the acceptance device 56 will be described with reference to FIG.

[0127] In the second learning process shown in FIG. 17, first, in step ST30, the second learning execution unit 64B acquires learning images 110 from the data supply device 94. Here, the learning images 110 acquired from the data supply device 94 are learning images 110 that have not been used in the processes after step ST30. The second learning execution unit 64B inputs the learning images 110 acquired from the data supply device 94 to the input layer 84A of the second determinator 82 (see FIG. 8). After the process of step ST30 is executed, the second learning process proceeds to step ST32.

[0128] In step ST32, the second learning execution unit 64B acquires the HP score 90 and the SSL score 92 from the second determinator 82 (see FIG. 8). After the processing of step ST32 is executed, the second learning processing proceeds to step ST34.

[0129] In step ST34, the second learning execution unit 64B acquires, from the data supply device 94, the supervised answer data 112 associated with the learning image 110 input to the input layer 84A in step ST30 (see FIG. 8). After the processing of step ST34 is executed, the second learning processing proceeds to step ST36.

[0130] In step ST36, the second learning execution unit 64B calculates an HP score error 114 from the HP score 90 acquired in step ST32 and the HP score 112A included in the supervised data 112 acquired in step ST34 (see FIG. 8). The second learning execution unit 64B also calculates an SSL score error 116 from the SSL score 92 acquired in step ST32 and the SSL score 112B included in the supervised data 112 acquired in step ST34 (see FIG. 8). After the processing of step ST36 is executed, the second learning processing proceeds to step ST38.

[0131] In step ST38, the second learning execution unit 64B calculates a plurality of adjustment values ​​118 that minimize both the HP score error 114 and the SSL score error 116 calculated in step ST36 (see FIG. 8). After the processing of step ST38 is executed, the second learning processing proceeds to step ST40.

[0132] In step ST40, the second learning execution unit 64B optimizes the second determinator 82 by adjusting the multiple optimization variables in the second determinator 82 (i.e., the multiple optimization variables included in the sixth convolutional layer 84J and the fully connected layer 84L) using the multiple adjustment values ​​118 so as to minimize both the HP score error 114 and the SSL score error 116 (see FIG. 8). After the processing of step ST40 is executed, the second learning processing proceeds to step ST42.

[0133] In step ST42, the second learning execution unit 64B determines whether a condition for terminating the second learning process (hereinafter referred to as the "second learning process termination condition") has been satisfied. A first example of the second learning process termination condition is that learning using all of the learning images 110 included in the teacher data 108 has been completed. A second example of the second learning process termination condition is that an instruction to terminate the second learning process has been received by the reception device 56. If the second learning process termination condition is not satisfied in step ST42, the determination is negative, and the second learning process proceeds to step ST30. If the second learning process termination condition is satisfied in step ST42, the determination is positive, and the second learning process ends.

[0134] 18A and 18B, an example of the flow of the inference processing performed by the processor 64 of the endoscope processing device 52 when an instruction to start execution of the inference processing is accepted by the accepting device 56 will be described. Note that the inference processing shown in Fig. 18A and 18B uses a first determiner 80 optimized by a first learning processing and a second determiner 82 optimized by a second learning processing.

[0135] 18A, first, in step ST50, the acquisition unit 64C acquires the endoscopic image 26 from the endoscope 40 (see FIG. 10). After the processing of step ST50 is executed, the inference processing proceeds to step ST52.

[0136] In step ST52, the determination unit 64D inputs the endoscopic image 26 acquired in step ST to an input layer 84A shared by the first determiner 80 and the second determiner 82. As a result, an HP score 86 and an NP score 88 are output from the first determiner 80, and an HP score 90 and an SSL score 92 are output from the second determiner 82 (see FIG. 10). After the processing of step ST52 is executed, the inference processing proceeds to step ST54.

[0137] In step ST54, the determination unit 64D acquires the HP score 86 and the NP score 88 output from the first determiner 80. After the process of step ST54 is executed, the inference process proceeds to step ST56.

[0138] In step ST56, the judgment unit 64D judges whether or not the HP score 86 acquired in step ST54 is equal to or greater than the NP score 88 acquired in step ST54. If the HP score 86 is less than the NP score 88 in step ST56, the judgment is negative, and the inference process proceeds to step ST68 shown in Fig. 18B. If the HP score 86 is equal to or greater than the NP score 88 in step ST56, the judgment is positive, and the inference process proceeds to step ST58.

[0139] In step ST58, the determination unit 64D obtains the HP score 90 and the SSL score 92 output from the second determination unit 82. After the process of step ST58 is executed, the inference process proceeds to step ST60.

[0140] In step ST60, the judgment unit 64D judges whether the SSL score 92 acquired in step ST58 is equal to or greater than the HP score 90 acquired in step ST58. If the SSL score 92 is less than the HP score 90 in step ST60, the judgment is negative, and the inference process proceeds to step ST74 shown in Fig. 18B. If the SSL score 92 is equal to or greater than the HP score 90 in step ST60, the judgment is positive, and the inference process proceeds to step ST62.

[0141] In step ST62, the control unit 64E acquires the feature map 128 from the sixth convolutional layer 84J of the second classifier 82 (see FIG. 10). Then, the control unit 64E generates the tumor position image signal 130 using the feature map 128 (see FIG. 10). After the processing of step ST62 is executed, the inference processing proceeds to step ST64.

[0142] In step ST64, the control unit 64E displays the endoscopic image 26 input to the input layer 84A in step ST52 on the first screen 22 (see FIG. 11). After the processing of step ST64 is executed, the inference processing proceeds to step ST66.

[0143] In step ST66, control unit 64E outputs tumor location image signal 130 generated in step ST62 to display device 14, thereby displaying tumor location identification image 134 indicated by tumor location image signal 130 on second screen 24 (see FIG. 11). After the processing of step ST66 is executed, the inference processing proceeds to step ST80 shown in FIG. 18B.

[0144] In step ST68 shown in FIG. 18B, the control unit 64E acquires a feature map 142 from the fifth convolutional layer 84F of the first determinator 80 (see FIG. 14). Then, the control unit 64E uses the feature map 142 to generate a tumor position image signal 144 (see FIG. 14). After the processing of step ST68 is executed, the inference processing proceeds to step ST70.

[0145] In step ST70, the control unit 64E displays the endoscopic image 26 input to the input layer 84A in step ST52 on the first screen 22 (see FIG. 15). After the processing of step ST70 is executed, the inference processing proceeds to step ST72.

[0146] In step ST72, the control unit 64E outputs the tumor location image signal 144 generated in step ST68 to the display device 14, thereby displaying the tumor location identification image 148 indicated by the tumor location image signal 144 on the second screen 24 (see FIG. 15). After the processing of step ST72 is executed, the inference processing proceeds to step ST80.

[0147] In step ST74 shown in FIG. 18B, the control unit 64E acquires the feature amount map 128 from the sixth convolutional layer 84J of the second determiner 82 (see FIG. 12). Then, the control unit 64E generates a non-tumor position image signal 136 using the feature amount map 128 (see FIG. 12). After the processing of step ST74 is executed, the inference processing proceeds to step ST76.

[0148] In step ST76, the control unit 64E displays the endoscopic image 26 input to the input layer 84A in step ST52 on the first screen 22 (see FIG. 13). After the processing of step ST76 is executed, the inference processing proceeds to step ST78.

[0149] In step ST78, the control unit 64E outputs the non-tumor position image signal 136 generated in step ST74 to the display device 14, thereby displaying the non-tumor position identifying image 140 indicated by the non-tumor position image signal 136 on the second screen 24 (see FIG. 13). After the processing of step ST78 is executed, the inference processing proceeds to step ST80.

[0150] In step ST80, the control unit 64E determines whether or not a condition for terminating the inference processing (hereinafter referred to as the "inference processing termination condition") has been satisfied. A first example of the inference processing termination condition is a condition in which a specified number of frames of endoscopic images 26 have been input to the input layer 84A since the execution of the inference processing began. A second example of the inference processing termination condition is a condition in which an instruction to terminate the inference processing has been received by the reception device 56. In step ST80, if the inference processing termination condition is not satisfied, the determination is negative, and the inference processing proceeds to step ST50. In step ST80, if the inference processing termination condition is satisfied, the determination is positive, and the inference processing ends.

[0151] As described above, the endoscope system 10 determines whether the type of lesion shown in the endoscopic image 26 belongs to the first group 120 (see FIG. 9) or the second group 122 (see FIG. 9). The first group 120 is a group that includes HP and SSL, which are similar lesion types. The second group 122 is a group that includes lesions other than SSL that belong to NP.

[0152] In the endoscope system 10, in order to have the first determiner 80 determine whether the type of lesion belongs to the first group 120 or the second group 122, the first determiner 80 is made to learn an HP training image 98A and an NP training image 98B (see FIG. 7). The HP training image 98A is an endoscopic image 26 that shows an HP or an SSL, and the NP training image 98 is an endoscopic image 26 that shows an NP.

[0153] Therefore, an endoscopic image 26 that shows a lesion other than SSL that belongs to NP is classified by the first determiner 80 into the second group 122. That is, the first determiner 80 determines that the type of lesion shown in the endoscopic image 26 belongs to the second group 122. On the other hand, an endoscopic image 26 that shows HP or SSL is classified by the first determiner 80 into the first group 120. That is, the first determiner 80 determines that the type of lesion shown in the endoscopic image 26 belongs to the first group 120 (see FIG. 9 ).

[0154] Here, if it is determined that the type of lesion shown in the endoscopic image 26 belongs to the first group 120, it is determined whether the type of lesion shown in the endoscopic image 26 is HP or SSL. The determination of whether the type of lesion is HP or SSL is made by the second determiner 82.

[0155] In the endoscope system 10, in order to have the second determiner 82 determine whether the type of lesion is HP or SSL, the second determiner 82 is made to learn an HP training image 110A and an SSL training image 110B. The HP training image 110A is an endoscopic image 26 that shows an HP but not an SSL. The SSL training image 110B is an endoscopic image 26 that shows an SSL.

[0156] Therefore, the endoscopic image 26 showing an HP is classified as an image showing an HP by the second determiner 82. That is, the second determiner 82 determines that the type of lesion shown in the endoscopic image 26 is an HP. On the other hand, the endoscopic image 26 showing an SSL is classified as an image showing an SSL by the second determiner 82. That is, the second determiner 82 determines that the type of lesion shown in the endoscopic image 26 is an SSL.

[0157] HP is classified into non-tumor category 124, and NP and SSL are classified into tumor category 126. Therefore, in endoscope system 10, tumor location image signal 130, non-tumor location image signal 136, or tumor location image signal 144 is output based on the determination result of the first determination process by first determiner 80 and the determination result of the second determination process by second determiner 82. That is, if the type of lesion shown in endoscopic image 26 is HP, non-tumor location image signal 136 is output (see FIG. 13), if the type of lesion shown in endoscopic image 26 is SSL, tumor location image signal 130 is output (see FIG. 11), and if the type of lesion shown in endoscopic image 26 is a type of lesion other than SSL that belongs to NP, tumor location image signal 144 is output (see FIG. 15).

[0158] Therefore, the doctor 16 can distinguish whether the type of lesion is tumorous or non-tumorous by grasping which of the tumor position image signal 130, the non-tumor position image signal 136, and the tumor position image signal 144 has been output. This can prevent the doctor 16 from misidentifying the type of lesion (for example, misidentifying a tumorous lesion as non-tumorous, or misidentifying a non-tumorous lesion as tumorous). For example, compared to using only a classifier that determines whether the lesion is HP or NP, it can prevent the doctor 16 from misidentifying the type of lesion.

[0159] Furthermore, in the endoscope system 10, when the type of the lesion is determined to belong to the first group 120 and the type of the lesion is determined to be SSL, a tumor position image signal 130 is output (see FIGS. 10 and 11). Therefore, the doctor 16 can distinguish the type of the lesion as a tumor.

[0160] Furthermore, in the endoscope system 10, when the type of lesion is determined to belong to the first group and the type of lesion is determined to be HP, a non-tumor position image signal 136 is output (see FIGS. 12 and 13). Therefore, the doctor 16 can distinguish the type of lesion as non-tumor.

[0161] Furthermore, in the endoscope system 10, the tumor localization images 134 and 148 and the non-tumor localization image 140 are displayed on the display device 14 in a distinguishable manner. That is, the tumor localization images 134 and 148 are marked with yellow tumor marks 132 and 146, and the non-tumor localization image 140 is marked with a green non-tumor mark 138, so that the tumor localization images 134 and 148 and the non-tumor localization image 140 are visually distinguishable from each other. Therefore, the doctor 16 can visually recognize whether the type of lesion is tumor or non-tumor.

[0162] Furthermore, in the endoscope system 10, the endoscopic image 26 and the tumor location identifying image 134 or 148 are displayed on the display device 14 so as to be contrasted (see FIGS. 11 and 15). That is, the endoscopic image 26 is displayed on the first screen 22, and the tumor location identifying image 134 or 148 is displayed on the second screen 24, so the doctor 16 can visually compare the endoscopic image 26 with the tumor location identifying image 134 or 148. The tumor location identifying images 134 and 148 are marked with the tumor marks 132 and 146, so the doctor 16 can visually determine where in the endoscopic image 26 the tumor is located.

[0163] Furthermore, in the endoscope system 10, the endoscopic image 26 and the non-tumor location identifying image 140 are displayed on the display device 14 so as to be contrastable (see FIG. 13). That is, the endoscopic image 26 is displayed on the first screen 22, and the non-tumor location identifying image 140 is displayed on the second screen 24, so the doctor 16 can visually compare the endoscopic image 26 with the non-tumor location identifying image 140. Since the non-tumor mark 138 is affixed to the non-tumor location identifying image 140, the doctor 16 can visually grasp where in the endoscopic image 26 a non-tumor lesion is present.

[0164] Furthermore, in the endoscope system 10, a tumor location identifying image 134 and a non-tumor location identifying image 140 are generated based on the feature amount map 128 (see FIGS. 10 and 12), and a tumor location identifying image 148 is generated based on the feature amount map 142 (see FIG. 14). Therefore, images in which the location of a tumor and a non-tumor location can be identified can be easily obtained. For example, compared to artificially identifying the location of a tumor and a non-tumor location from the endoscope image 26 and then artificially creating an image in which the location of a tumor and a non-tumor location can be identified, an image in which the location of a tumor and a non-tumor location can be easily obtained.

[0165] Furthermore, in the endoscope system 10, the second determination process by the second determiner 82 is a process that uses the intermediate feature amount of the first determiner 80. Therefore, the second determination process by the second determiner 82 can be efficiently realized. For example, compared to when the second determination process is performed by the second determiner 82 that is constructed independently of the first determiner 80, the second determination process by the second determiner 82 can be efficiently realized.

[0166] Furthermore, in the endoscope system 10, the first determiner 80 and the second determiner 82 share multiple layers (e.g., the input layer 84A and the first to fourth convolutional layers 84B to 84E) subsequent to the input layer 84. This makes it possible to prevent the neural network 78 from becoming larger. For example, compared to when the first determiner 80 and the second determiner 82 are constructed in a completely separated state, it is possible to prevent the neural network 78 from becoming larger.

[0167] Furthermore, in the endoscope system 10, the endoscopic image 26 showing an HP and the endoscopic image 26 showing an SSL are used for training the first classifier 80 as images corresponding to one group (i.e., a group to which similar neoplastic lesions and non-neoplastic lesions belong). Furthermore, the endoscopic image 26 showing an NP (i.e., the NP training image 98B) is used for training the first classifier 80 as an image corresponding to one group (i.e., a group to which lesions other than SSL that are classified as NP belong). Therefore, the judgment accuracy of the first classifier 80 can be improved. For example, the judgment accuracy of the first classifier 80 can be improved compared to when only the endoscopic image 26 showing an HP and the endoscopic image 26 showing an NP are used for training the first classifier 80.

[0168] [First Modification] In the above embodiment, an example has been described in which tumor location image signal 130, non-tumor location image signal 136, and tumor location image signal 144 are selectively output to display device 14, thereby selectively displaying tumor location identifying image 134, non-tumor location identifying image 140, and tumor location identifying image 148 on display device 14, but the technology of the present disclosure is not limited to this. For example, as shown in Fig. 19, control unit 64E may perform a storage process (hereinafter referred to as a "first storage process") in which tumor location image signals 130 and 144 and non-tumor location image signal 136 are stored in NVM 68 in a distinguishable manner for each endoscopic image 26 (i.e., frame unit).

[0169] 19, an endoscopic image 26 showing an SSL and a tumor position image signal 130 are stored in association with each other in the NVM 68. Also, in the example shown in FIG. 19, an endoscopic image 26 showing an NP, which is a neoplastic lesion different from the SSL, and a tumor position image signal 144 are stored in association with each other in the NVM 68. Also, in the example shown in FIG. 19, an endoscopic image 26 showing an HP and a non-tumor position image signal 136 are stored in association with each other in the NVM 68. This allows the doctor 16 to determine whether the type of lesion shown in the endoscopic image 26 is tumorous or non-tumorous on a frame-by-frame basis.

[0170] The first saving process may be performed in parallel with the process of selectively displaying the tumor location specifying image 134, the non-tumor location specifying image 140, and the tumor location specifying image 148 on the display device 14.

[0171] Furthermore, the destination where tumor location image signal 130, non-tumor location image signal 136, and tumor location image signal 144 are saved as a result of the first saving process is not limited to NVM 68, but may be a storage device other than NVM 68. Examples of storage devices other than NVM 68 include storage devices that exist outside endoscope processing device 52 (for example, storage used in a server and / or personal computer, etc.).

[0172] [Second Modification] In the above embodiment, an example has been described in which it is possible to identify whether the type of lesion is classified into the non-tumor category 124 or the tumor category 126 by outputting the tumor location image signal 130, the non-tumor location image signal 136, and the tumor location image signal 144, but the technology of the present disclosure is not limited to this. For example, the type of lesion itself may be identified by outputting a signal that can identify whether the type of lesion is HP or SSL based on the determination result of the first determination process by the first determiner 80 and the determination result of the second determination process by the second determiner 82.

[0173] 20, when the HP score 86 is equal to or greater than the NP score 88 and the SSL score 92 is equal to or greater than the HP score 90, the control unit 64E generates an SSL position image signal 150 using the feature amount map 128 of the second determiner 82. When the HP score 86 is equal to or greater than the NP score 88 and the SSL score 92 is equal to or greater than the HP score 90, this means that the type of lesion is determined to be SSL. Note that the SSL score 92 in this case is an example of a "second type determination signal" according to the technology of the present disclosure.

[0174] Control unit 64E applies color to curved region 128A2. In the example shown in FIG. 20, curved region 128A2 is colored blue. Control unit 64E generates position identification image 28 by processing feature amount map 128 having colored curved region 128A2. In the example shown in FIG. 20, SSL position identification image 154 is shown as position identification image 28. SSL mark 152 is applied to SSL position identification image 154 as lesion mark 28C. SSL mark 152 is colored curved region 128A2, i.e., curved region 128A2 colored blue.

[0175] The control unit 64E generates an SSL position image signal 150, which is a signal indicating the SSL position identification image 154, and outputs the generated SSL position image signal 150 to a specific output destination (e.g., the display device 14, etc.). Here, the process of outputting the SSL position image signal 150 to a specific output destination is an example of a "second output process" according to the technology of the present disclosure. Furthermore, the SSL position image signal 150 is an example of a "signal corresponding to a second type determination signal" and a "determination result signal" according to the technology of the present disclosure.

[0176] In this embodiment, the blue color applied to the curved region 128A2 indicates that the type of lesion is SSL, and therefore the SSL position image signal 150 is used as a signal that can identify that the type of lesion is SSL.

[0177] 21 as an example, the control unit 64E displays the endoscopic image 26 acquired by the acquisition unit 64C, i.e., the endoscopic image 26 input to the first determiner 80 and the second determiner 82 shown in FIG. 20, on the first screen 22. The control unit 64E also outputs an SSL position image signal 150 (see FIG. 20) to the display device 14, thereby displaying an SSL position identification image 154 indicated by the SSL position image signal 150 on the second screen 24. That is, the SSL position identification image 154 is displayed on the second screen 24 as information indicating that the lesion shown in the endoscopic image 26 displayed on the first screen 22 is an SSL. The SSL position identification image 154 is an example of "information based on a signal corresponding to a second type determination signal" and "information based on a determination result signal" according to the technology of the present disclosure.

[0178] 22, when the HP score 86 is equal to or greater than the NP score 88 and the SSL score 92 is less than the HP score 90, the control unit 64E generates an HP position image signal 156 using the feature amount map 128 of the second determiner 82. When the HP score 86 is equal to or greater than the NP score 88 and the SSL score 92 is less than the HP score 90, this means that the type of lesion is determined to be HP. Note that the HP score 90 in this case is an example of a "determination result signal" according to the technology of the present disclosure.

[0179] Control unit 64E applies color to curved region 128B2. In the example shown in FIG. 22, curved region 128B2 is colored white. Control unit 64E generates position identification image 28 by processing feature amount map 128 having colored curved region 128B2. In the example shown in FIG. 22, HP position identification image 160 is shown as position identification image 28. HP mark 158 is applied to HP position identification image 160 as lesion mark 28C. HP mark 158 is colored curved region 128B2, i.e., curved region 128B2 colored white.

[0180] The control unit 64E generates an HP position image signal 156, which is a signal indicating the HP position identification image 160, and outputs the generated HP position image signal 156 to a specific output destination (for example, the display device 14, etc.). Here, the process of outputting the HP position image signal 156 to a specific output destination is an example of a "second output process" according to the technology of the present disclosure.

[0181] In this embodiment, the white color applied to the curve region 128B2 indicates that the type of lesion is HP. Therefore, the HP position image signal 156 is used as a signal that can identify that the type of lesion is HP.

[0182] 23, the control unit 64E displays the endoscopic image 26 acquired by the acquisition unit 64C, i.e., the endoscopic image 26 input to the first determiner 80 and the second determiner 82 shown in FIG. 22, on the first screen 22. The control unit 64E also outputs an HP position image signal 156 (see FIG. 22) to the display device 14, thereby displaying an HP position identification image 160 indicated by the HP position image signal 156 on the second screen 24. That is, the HP position identification image 160 is displayed on the second screen 24 as information indicating that the lesion shown in the endoscopic image 26 displayed on the first screen 22 is an HP. The HP position identification image 160 is an example of "information based on a determination result signal" according to the technology of the present disclosure.

[0183] 24, when the HP score 86 is less than the NP score 88, the control unit 64E generates the NP position image signal 162 using the feature amount map 142 of the first determiner 80. When the HP score 86 is less than the NP score 88, it means that the type of lesion is determined to be NP. Note that the NP score 88 in this case is an example of a "determination result signal" according to the technology of the present disclosure.

[0184] Controller 64E generates curved region 142A2 by shaping contour 142A1 into a curved shape, and applies color to generated curved region 142A2. In the example shown in FIG. 24, curved region 142A2 is colored blue-green. Controller 64E generates position identification image 28 by processing feature amount map 142 having colored curved region 142A2. In the example shown in FIG. 24, NP position identification image 166 is shown as position identification image 28. NP mark 164 is applied to NP position identification image 166 as lesion mark 28C. NP mark 164 is colored curved region 142A2, i.e., curved region 142A2 colored blue-green.

[0185] The control unit 64E generates an NP position image signal 162, which is a signal indicating the NP position identification image 166, and outputs the generated NP position image signal 162 to a specific output destination (for example, the display device 14, etc.).

[0186] In this embodiment, the blue-green color applied to the curve region 144A2 is a color that indicates that the type of lesion is NP. Therefore, the NP position image signal 162 is used as a signal that can identify that the type of lesion is NP.

[0187] 25 as an example, the control unit 64E displays the endoscopic image 26 acquired by the acquisition unit 64C, i.e., the endoscopic image 26 input to the first determiner 80 shown in FIG. 24, on the first screen 22. Furthermore, the control unit 64E outputs the NP position image signal 162 (see FIG. 24) to the display device 14, thereby displaying the NP position specifying image 166 indicated by the NP position image signal 162 on the second screen 24. That is, the NP position specifying image 166 is displayed on the second screen 24 as information indicating that the lesion shown in the endoscopic image 26 displayed on the first screen 22 is an NP.

[0188] 21, 23, and 25, the control unit 64E performs display processing to cause the display device 14 to display the SSL position identifying image 154, the HP position identifying image 160, and the NP position identifying image 166 in a distinguishable display manner, based on the determination results of the first determination processing and the second determination processing. The control unit 64E also causes the first display device 14 to display the endoscopic image 26 and the SSL position identifying image 154 so that they can be contrasted. The control unit 64E also causes the first display device 14 to display the endoscopic image 26 and the HP position identifying image 160 so that they can be contrasted. The control unit 64E also causes the first display device 14 to display the endoscopic image 26 and the NP position identifying image 166 so that they can be contrasted.

[0189] The SSL position identifying image 154 is an image indicating that the lesion shown in the endoscopic image 26 is an SSL, and the blue SSL mark 152 allows the doctor 16 to visually recognize that the lesion shown in the endoscopic image 26 is an SSL and the position of the SSL within the endoscopic image 26. The HP position identifying image 160 is an image indicating that the lesion shown in the endoscopic image 26 is an HP, and the white HP mark 158 allows the doctor 16 to visually recognize that the lesion shown in the endoscopic image 26 is an HP and the position of the HP within the endoscopic image 26. The NP position identifying image 166 is an image indicating that the lesion shown in the endoscopic image 26 is an NP, and the blue-green NP mark 164 allows the doctor 16 to visually recognize that the lesion shown in the endoscopic image 26 is an NP and the position of the NP within the endoscopic image 26.

[0190] Therefore, with the endoscope system 10 according to the second modification, the doctor 16 can distinguish whether the type of lesion is SSL, HP, or NP by understanding which of the SSL position image signal 150, the HP position image signal 156, and the NP position image signal 162 has been output through the image displayed on the second screen 24. This makes it possible to prevent the doctor 16 from misidentifying the type of lesion. For example, compared to when only a determiner for determining whether the lesion is HP or NP is used, it is possible to prevent the doctor 16 from misidentifying the type of lesion.

[0191] Furthermore, according to the endoscopic system 10 of this second modified example, the SSL position identification image 154, the HP position identification image 160, and the NP position identification image 166 are displayed on the display device 14 in a distinguishable display manner for each frame (i.e., for each endoscopic image 26), so that the doctor 16 can visually grasp the results of the determination of the type of lesion as an image.

[0192] The output of the SSL location image signal 150, the HP location image signal 156, and the NP location image signal 162 may be performed in parallel with the output of the tumor location image signal 130, the non-tumor location image signal 136, and the tumor location image signal 144 described in the above embodiment. In this case, for example, the display device 14 may be caused to display the tumor location identifying image 134 and the SSL location identifying image 154 in a contrastable manner, the non-tumor location identifying image 140 and the HP location identifying image 160 in a contrastable manner, or the tumor location identifying image 148 and the NP location identifying image 166 in a contrastable manner.

[0193] Furthermore, the size of the SSL mark 152 may be adjusted using a predetermined coefficient (e.g., a coefficient based on the ratio of the area of ​​the endoscopic image 26 to the area of ​​the feature map 128) and then the SSL mark 152 may be superimposed and displayed at a corresponding position within the endoscopic image 26. In this case, only a portion of the SSL mark 152 (e.g., the outline) may be superimposed and displayed, or alpha blending may be performed on the endoscopic image 26 and the SSL mark 152. The same applies to the HP mark 158, NP mark 164, tumor mark 132, non-tumor mark 138, and / or tumor mark 146.

[0194] Furthermore, the colors assigned to the SSL mark 152, HP mark 158, NP mark 164, tumor mark 132, non-tumor mark 138, and tumor mark 146 are not limited to those described above, and may be any colors that are distinguishable from one another. Furthermore, the SSL mark 152, HP mark 158, NP mark 164, tumor mark 132, non-tumor mark 138, and / or tumor mark 146 may be distinguished by adding a pattern to the SSL mark 152, HP mark 158, NP mark 164, tumor mark 132, non-tumor mark 138, and / or tumor mark 146 or by changing the outline display mode (thickness, line type, brightness, and / or color).

[0195] [Third Modification] In the second modified example described above, an example was given in which inference processing is performed on a frame-by-frame basis on a plurality of endoscopic images 26 obtained by the live view method, thereby determining the type of lesion for each frame. In this case, when inference processing is performed on a plurality of endoscopic images 26 in chronological order, an SSL position image signal 150 may be generated for only one frame of the plurality of endoscopic images 26, and an HP position image signal 156 may be generated for the remaining endoscopic images 26.

[0196] One possible cause of this phenomenon is a decrease in the accuracy of the determination by the first determiner 80 and / or the second determiner 82 due to insufficient image quality of the endoscopic image 26 and / or insufficient learning of the neural network 78. If the HP position image signal 156 is generated for a frame for which the SSL position image signal 150 should be generated instead of the HP position image signal 156, there is a risk that the doctor 16 will distinguish the lesion as HP even though it is SSL.

[0197] Therefore, in the endoscope system 10 according to the third modification, as shown in FIG. 26 as an example, the control unit 64E corrects the determination result for a frame that may have been erroneously determined by the determination unit 64D. In this case, the control unit 64E first selectively generates an SSL position image signal 150, an HP position image signal 156, and an NP position image signal 162 (hereinafter, when there is no need to distinguish between them, they will also be referred to as "position image signals" without reference numerals) for multiple frames in the manner described in the second modification. The control unit 64E then stores the position image signals in chronological order in the order in which they were generated, together with the endoscopic image 26, in a FIFO manner. That is, each time a new frame is added, the control unit 64E outputs the oldest frame together with the position image signal to the display device 14. In the example shown in FIG. 26, for convenience of illustration, the position image signals for the first to seventh frames are stored in association with the endoscopic image 26.

[0198] In the example shown in FIG. 26 , HP position image signals 156 are generated and stored for the first to third and fifth to seventh frames, and an SSL position image signal 150 is generated and stored for the fourth frame. When the SSL position image signal 150 is generated for the fourth frame, the control unit 64E outputs the SSL position image signal 150 for the fourth frame to the display device 14 together with the corresponding endoscopic image 26. In this case, the control unit 64E replaces the HP position image signals 156 for the first to third and fifth to seventh frames with the SSL position image signal 150, thereby aligning the position image signals for the first to third and fifth to seventh frames with the SSL position image signal 150. The control unit 64E then outputs the SSL position image signals 150 together with the corresponding endoscopic images 26 to the display device 14 in chronological order. As a result, the endoscopic image 26 and the SSL position specifying image 154 are displayed on the display device 14 for the first to seventh frames in the same manner as in the second modified example.

[0199] Furthermore, the control unit 64E outputs the SSL position image signal 150 to the display device 14, thereby displaying the characters "SSL" on the first screen 22. The location where the characters are displayed may not be within the first screen 22, but may be within the second screen 24, or may be the screen of another display device. Additionally, along with or instead of displaying the characters SSL, the word "SSL" may be output as sound. In the example shown in FIG. 26, an example is shown in which the characters "SSL" are displayed and sound is output, but similar characters may also be displayed and sound output for NP and / HP.

[0200] In the example shown in Figure 26, frames 1 to 7 are an example of "multiple frames" according to the technology of the present disclosure, frame 4 is an example of "first frame" according to the technology of the present disclosure, and frames 1 to 3 and frames 5 to 7 are examples of "second frames within a range of a predetermined number of frames from the first frame" according to the technology of the present disclosure.

[0201] 27, when an HP position image signal 156 is generated for the fourth frame, the control unit 64E outputs the position image signals currently generated for the first to seventh frames in chronological order to the display device 14 together with the endoscopic image 26. As a result, the display device 14 displays the endoscopic image 26 and the SSL position identification image 154 in the same manner as in the second modified example. That is, when the type of lesion is determined to be HP for the fourth frame, the control unit 64E outputs signals (i.e., position image signals) corresponding to the determination results by the determination unit 64D to the display device 14 together with the endoscopic image 26 for frames within a predetermined number of frames from the fourth frame (the first to third frames and the fifth to seventh frames in the example shown in FIG. 27).

[0202] As described above, in the endoscope system 10 according to the third modification, whether the type of lesion is SSL, NP, or HP is determined for each frame based on the determination results of the first determination process and the second determination process in the same manner as in the second modification. When the type of lesion for a first frame (e.g., the fourth frame) among the multiple frames is determined to be SSL, a position image signal indicating that the type of lesion has been determined to be SSL, i.e., an SSL position image signal 150, is output to the display device 14. Furthermore, when the type of lesion for a second frame (e.g., the first to third frames and the fifth to seventh frames) within a predetermined number of frames from the first frame among the multiple frames is determined to be a lesion type other than SSL, the SSL position image signal 150 is output to the display device 14. This prevents the doctor 16 from erroneously identifying an SSL lesion as a type other than SSL. For example, compared to a case where a position image signal is always output in accordance with the determination result when the type of lesion is determined to be a different lesion type from SSL for the second frame within a range of a predetermined number of frames from the first frame, it is possible to prevent doctor 16 from mistakenly identifying the lesion as a type other than SSL even though the type of lesion is SSL.

[0203] Furthermore, in the endoscope system 10 according to the third modification, when the type of lesion is determined to be HP for the first frame (e.g., the fourth frame), a position image signal corresponding to the determination result for the type of lesion is output for the second frame within a predetermined number of frames from the first frame (e.g., the first to third frames and the fifth to seventh frames). This allows the doctor 16 to know the type of lesion for each frame. Furthermore, since multiple lesion types including HP and SSL are subject to determination for each frame, the doctor 16 can know whether the type of lesion is HP or SSL for each frame. Furthermore, since HP is classified into the non-tumor category 124 and SSL and NP are classified into the tumor category 126, the doctor 16 can know whether the lesion is classified into the non-tumor category 124 or the tumor category 126.

[0204] Although the SSL position image signal 150, HP position image signal 156, and NP position image signal 162 are illustrated in the third modified example, this is merely an example. For example, the tumor position image signal 130, non-tumor position image signal 136, and tumor position image signal 144 may be used instead of or in addition to the SSL position image signal 150, HP position image signal 156, and NP position image signal 162. When the tumor position image signal 130, non-tumor position image signal 136, and tumor position image signal 144 are used instead of the SSL position image signal 150, HP position image signal 156, and NP position image signal 162, the tumor position image signal 130 should be applied instead of the SSL position image signal 150, the non-tumor position image signal 136 should be applied instead of the HP position image signal 156, and the tumor position image signal 144 should be applied instead of the NP position image signal 162. In this case, tumor location image signal 130 is an example of a "signal corresponding to a second type of judgment signal" according to the technology of the present disclosure, and non-tumor location image signal 136 and tumor location image signal 144 are each an example of a "judgment result signal" according to the technology of the present disclosure.

[0205] [Fourth Modification] In the second and third modified examples described above, an example has been given in which the SSL position specifying image 154, the HP position specifying image 160, and the NP position specifying image 164 are selectively displayed on the display device 14, but the technology of the present disclosure is not limited to this. For example, as shown in Fig. 28, the control unit 64E may perform a storage process (hereinafter referred to as a "second storage process") in which the SSL position image signal 150, the HP position image signal 156, and the NP position image signal 162 are stored in the NVM 68 in a distinguishable manner for each endoscopic image 26 (i.e., for each frame).

[0206] In the example shown in Fig. 28, an endoscopic image 26 showing an SSL and an SSL position image signal 150 are stored in the NVM 68 in a state of being associated with each other. Also, in the example shown in Fig. 28, an endoscopic image 26 showing an NP, which is a tumorous lesion different from an SSL, and an NP position identification image 164 are stored in the NVM 68 in a state of being associated with each other. Also, in the example shown in Fig. 28, an endoscopic image 26 showing an HP and an HP position image signal 156 are stored in the NVM 68 in a state of being associated with each other. This allows the doctor 16 to determine whether the type of lesion shown in the endoscopic image 26 is tumorous or non-tumorous on a frame-by-frame basis.

[0207] The second saving process may be performed in parallel with the process of selectively displaying the SSL position specifying image 154, the HP position specifying image 160, and the NP position specifying image 164 on the display device 14.

[0208] Furthermore, the destination where the SSL position image signal 150, the HP position image signal 156, and the NP position image signal 162 are saved by the second saving process is not limited to the NVM 68, but may be a storage device other than the NVM 68. Examples of storage devices other than the NVM 68 include storage devices that exist outside the endoscope processing device 52 (for example, storage used in a server and / or a personal computer, etc.).

[0209] [Other variations] In the above embodiment, the endoscopic image 26 showing an HP or SSL is used as the HP learning image 98A included in the training data 96 used in the first learning process, but the technology of the present disclosure is not limited to this. The HP learning image 98A may be an endoscopic image 26 showing an HP but not an SSL.

[0210] In the above embodiment, an example in which HP, SSL, and NP are classified has been described. However, in addition to HP, SSL, and NP, non-neoplastic polyps other than HP (e.g., hamartomatous polyps and / or inflammatory polyps) may also be classified. In this case, for example, endoscopic images 26 showing non-neoplastic polyps other than HP may also be used as learning images to train the neural network 78, and scores related to non-neoplastic polyps other than HP may be output from the neural network 78. The scores related to non-neoplastic polyps other than HP may be output from the first classifier 80 or the second classifier 82.

[0211] In the above embodiment, the first determinator 80 and the second determinator 82 are trained separately. However, the technology of the present disclosure is not limited to this. The first determinator 80 and the second determinator 82 may be trained together. In this case, for example, the correct answer data associated with each learning image included in the training data used for learning includes an HP score 100A, an NP score 100B, an HP score 112A, and an SSL score 112B. For example, in the correct answer data associated with the HP learning image 98A, the HP score 100A is set to "1.0" and the remaining scores are set to "0.0." In addition, in the correct answer data associated with the NP learning image 98B, the NP score 100B is set to "1.0" and the remaining scores are set to "0.0." In addition, in the correct answer data associated with the HP learning image 110A, the HP score 112A is set to "1.0" and the remaining scores are set to "0.0." Moreover, in the correct answer data associated with the SSL training image 110B, the SSL score 112B is set to "1.0" and the remaining scores are set to "0.0."

[0212] In the above embodiment, HP is given as an example of the "first type" and "first lesion" according to the technology of the present disclosure, and SSL is given as an example of the "second type" and "second lesion" according to the technology of the present disclosure, but the technology of the present disclosure is not limited to this. For example, two similar lesions other than SSL and HP may also be used. Examples of two similar lesions include lesions that are generally difficult for a doctor 16 to distinguish between.

[0213] In the above embodiment, the first learning process, the second learning process, and the inference process (hereinafter, when there is no need to distinguish between them, they will be referred to as "processing related to the technology of the present disclosure") using the endoscopic image 26 are illustrated, but this is merely an example, and the processing related to the technology of the present disclosure may be performed using medical images obtained by imaging the observation region of the subject 20 using various modalities such as an ultrasound diagnostic device, an X-ray diagnostic device, a CT diagnostic device, and / or an MRI diagnostic device, instead of the endoscopic image 26. Note that the medical images obtained by imaging the observation region of the subject 20 using various modalities are examples of "medical images" related to the technology of the present disclosure, and the various modalities are examples of "imaging devices" related to the technology of the present disclosure.

[0214] In the above embodiment, the inference processing is performed for each frame, but the technology of the present disclosure is not limited to this. For example, the inference processing may be performed every predetermined number of frames (e.g., every few frames or every few tens of frames). In this case, the number of times the inference processing is performed is reduced, so the load on the processor 64 can be reduced compared to when the inference processing is performed for each frame. Furthermore, when the inference processing is performed every predetermined number of frames in this way, the inference processing may be performed at a frame interval at which the display mode of the position identification image 28 (e.g., tumor mark 132, etc.) displayed on the second screen 24 is visually perceived due to the afterimage phenomenon.

[0215] In the above embodiment, an example was given in which the position-specific image 28 generated based on the feature map 128, etc. is displayed on the second screen 24, but this is merely one example, and a CAM image may also be generated based on multiple feature maps obtained from the neural network 68 and displayed on the second screen 24.

[0216] In the above embodiment, an example has been described in which the endoscopic image 26 is displayed on the first screen 22 and the position identification image 28 is displayed on the second screen 24, but this is merely an example, and the endoscopic image 26 and the position identification image 28 may be displayed on different display devices. Furthermore, the first screen 22 and the second screen 24 may be alternately displayed on the display device 14 in accordance with instructions and / or various conditions, etc., received by the reception device 56. Furthermore, the endoscopic image 26 and the position identification image 28 may be selectively displayed on the full screen in accordance with instructions and / or various conditions, etc., received by the reception device 56.

[0217] In the above embodiment, the processor 64 directly controls the display device 14 to display the endoscopic image 26 and the position identification image 28 on the display device 14. However, this is merely an example. For example, the processor 64 may indirectly control the display device 14 to display the endoscopic image 26 and the position identification image 28 on the display device 14. In this case, for example, screen information indicating the screens (e.g., the first screen 22 and the second screen 24) to be displayed on the display device 14 is temporarily stored in external storage (not shown). Then, the processor 64 or a processor other than the processor 64 acquires the screen information from the external storage and, based on the acquired screen information, displays the first screen 22 and the second screen 24 on the display device 14 or a display device other than the display device 14. A specific example of this case is an embodiment in which the processor 64 uses cloud computing to display the first screen 22 and the second screen 24 on the display device 14 or a display device other than the display device 14.

[0218] In the above embodiment, an example embodiment has been described in which the processing according to the technology of the present disclosure is performed by the processor 64 of the endoscope processing device 52. However, the technology of the present disclosure is not limited to this. For example, a device performing the processing according to the technology of the present disclosure may be provided external to the endoscope processing device 52. An example of a device provided external to the endoscope processing device 52 is a server. For example, the server may be implemented using cloud computing. While cloud computing is used here as an example, this is merely an example. For example, the server may be implemented using a mainframe or network computing such as fog computing, edge computing, or grid computing. The server is merely an example, and at least one personal computer may be used instead of a server. Furthermore, the processing according to the technology of the present disclosure may be performed in a distributed manner by multiple devices, including the endoscope processing device 52 and at least one device provided external to the endoscope processing device 52.

[0219] Furthermore, in the above embodiment, an example was described in which the first learning processing program 72, the second learning processing program 74, and the inference processing program 76 (hereinafter, these will be referred to as "programs related to the technology of the present disclosure") are stored in the NVM 68, but the technology of the present disclosure is not limited to this. For example, the programs related to the technology of the present disclosure may be stored in a portable storage medium such as an SSD or a USB memory. The storage medium is a non-transitory computer-readable storage medium. The programs related to the technology of the present disclosure stored in the storage medium are installed in the computer 60 of the endoscope processing device 52. The processor 64 executes processing related to the technology of the present disclosure in accordance with the programs related to the technology of the present disclosure.

[0220] In the above embodiment, the computer 60 is exemplified, but the technology of the present disclosure is not limited to this, and a device including an ASIC, an FPGA, and / or a PLD may be applied instead of the computer 60. Furthermore, instead of the computer 60, a combination of a hardware configuration and a software configuration may be used.

[0221] The hardware resources for executing the processes according to the technology of the present disclosure described in the above embodiments can be various processors, as listed below. Examples of processors include general-purpose processors that function as hardware resources for executing the processes according to the technology of the present disclosure by executing software, i.e., programs. Examples of processors also include dedicated electronic circuits, such as FPGAs, PLDs, or ASICs, which are processors with circuit configurations designed specifically for executing specific processes. Each processor has built-in or connected memory, and each processor uses the memory to execute the processes according to the technology of the present disclosure.

[0222] The hardware resource that executes the processing according to the technology of the present disclosure may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a processor and an FPGA). Also, the hardware resource that executes the processing according to the technology of the present disclosure may be a single processor.

[0223] As an example of a configuration using a single processor, first, one processor is configured by combining one or more processors with software, and this processor functions as a hardware resource that executes the processing related to the technology of the present disclosure. Second, there is a configuration in which a processor is used that realizes the functions of the entire system including multiple hardware resources that execute the processing related to the technology of the present disclosure on a single IC chip, as typified by SoC. In this way, the processing related to the technology of the present disclosure is realized using one or more of the various processors described above as hardware resources.

[0224] Furthermore, the hardware structure of these various processors can be, more specifically, an electronic circuit that combines circuit elements such as semiconductor devices. The above-described processing according to the technology of the present disclosure is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the present disclosure.

[0225] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[0226] In this specification, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0227] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference. [Explanation of symbols]

[0228] 10 Endoscopy System 12 Endoscopic equipment 14 Display device 16. Doctor 17. Nurse 18 Endoscopy 20 Subjects 22 1st screen 24 2nd screen 26 Endoscopic images 26A Site area 26B Lesion area 26B1 HP area 26B2 NP area 26B3 SSL area 28 Location-specific images 28A outer frame 28B Target Mark 28C Lesion Mark 30 Control section 32 Insertion section 34 Tip 36 Curved section 37 Soft part 38 Lighting equipment 38A, 38B Lighting window 40 Endoscope 42 Opening for treatment instruments 44 Treatment tools 46 Treatment tool insertion port 48 Puncture needle 50 Universal Code 50A proximal end 50B 1st tip 50C 2nd tip 52 Endoscope processing equipment 54 Light source device 56 Reception device 58 Large intestine 58A Inner wall 60 Computer 62 Input / Output Interface 64 processors 64A First Learning Execution Department 64B Second Learning Execution Unit 64C Acquisition Department 64D Judgment section 64E Control Unit 66 RAM 68 NVM 70 Bus 72 First Learning Processing Program 74 Second Learning Processing Program 76 Inference Processing Program 78 Neural Networks 80 1st determiner 82 Second determiner 84A Input layer 84B First convolutional layer 84C Second convolutional layer 84D 3rd convolution layer 84E 4th convolutional layer 84F 5th convolutional layer 84G,84K GAP layer 84H,84L fully connected layer 84I,84M Output layer 84I1,84M1 Softmax function 84J 6th convolutional layer 86,90,100A HP score 88,100B NP score 92 SSL Score 94 Data Supply Device 96,108 training data 98,110 training images 98A,110A HP learning images 98B NP learning image 100,112 correct data 102,114 HP score error 104 NP score error 106,118 adjusted value 110B SSL learning image 112A HP score 112B SSL score 116 SSL score errors 120 Group 1 122 Group 2 124 Non-oncology category 126 Tumor Categories 128,142 feature maps 128A, 128B, 142A Highly reactive region 128A1, 128B1, 142A1 Contour 128A2,128B2,142A2 curve area 130,144 Tumor location image signal 132,146 tumor marks 134,148 tumor localization images 136 Non-tumor location image signals 138 Non-tumor Marks 140 Non-tumor localization images 150 SSL position image signal 152 SSL Mark 154 SSL geolocation images 156 HP position image signal 158 HP Mark 160 HP location specific images 162 NP position image signal 164 NP Mark 166 NP localization images

Claims

1. a processor; The processor: obtaining a medical image including a lesion area indicative of a lesion; a classifier performs a determination process to determine the type of the lesion based on the medical image; the determiner includes a first determiner and a second determiner; the determination process includes a first determination process by the first determiner and a second determination process by the second determiner, the first determination process is a process for determining whether the type belongs to a first group including a first type classified into a first category and a second type classified into a second category, or whether the type belongs to a second group including a third type classified into the second category and different from the second type, the second determination process is a process of determining whether the type is the first type or the second type only when it is determined in the first determination process that the device belongs to the first group, The processor performs at least one of a first output process of outputting a first signal capable of specifying whether the type is classified into the first category or the second category based on a determination result of the first determination process and, if it is determined in the first determination process that the type belongs to the first group, also based on a determination result of the second determination process, and a second output process of outputting a second signal capable of specifying whether the type is the first type or the second type based on a determination result of the second determination process if it is determined in the first determination process that the type belongs to the first group. Image processing device.

2. When the first determination process determines that the type belongs to the first group and the second determination process determines that the type is the second type, the processor outputs, as the first signal, a second category signal that can identify that the type is classified into the second category. The image processing device according to claim 1 .

3. When the first determination process determines that the type belongs to the first group and the second determination process determines that the type is the first type, the processor outputs, as the first signal, a first category signal that can identify that the type is classified into the first category.

3. The image processing device according to claim 1.

4. The processor performs at least one of a display process of displaying, on a display device, first category information indicating that the type is classified into the first category and second category information indicating that the type is classified into the second category in a distinguishable display manner based on a determination result of the first determination process and a determination result of the second determination process, and a storage process of storing, on a medical image basis, the first category information and the second category information in a distinguishable manner. The image processing device according to any one of claims 1 to 3.

5. The display modes include a first mode in which the medical image and the first category information are displayed on the display device so as to be contrasted with each other, and a second mode in which the medical image and the second category information are displayed on the display device so as to be contrasted with each other. The image processing device according to claim 4 .

6. At least one of the first category information and the second category information is information based on a feature map obtained from the classifier.

6. The image processing device according to claim 4 or claim 5.

7. The second determination process is a process that uses intermediate features of the first classifier. The image processing device according to any one of claims 1 to 6.

8. the determiner is a neural network, The first and second decision units share a plurality of layers from the input layer onward. The image processing device according to any one of claims 1 to 7.

9. The first classifier is a model that has been trained to use a first lesion image showing a first lesion corresponding to the first type and a second lesion image showing a second lesion corresponding to the second type as images corresponding to the first group, and a third lesion image showing a third lesion corresponding to the third type as images corresponding to the second group. The image processing device according to any one of claims 1 to 8.

10. The processor: acquiring said medical images over a time series of multiple frames; determining, for each frame, which of a plurality of lesion types including the first type and the second type, the type is determined based on a determination result of the first determination process and a determination result of the second determination process; outputting a signal corresponding to a second type determination signal indicating that the type of a first frame of the plurality of frames has been determined to be the second type; When it is determined that the type of a second frame within a range of a predetermined number of frames from the first frame among the plurality of frames is a lesion type different from the second type, a signal corresponding to the second type determination signal is output. The image processing device according to any one of claims 1 to 9.

11. the first category is a category indicating non-tumor; The second category is a category indicating tumorous The image processing device according to any one of claims 1 to 10.

12. The first type is a non-neoplastic lesion. The image processing device according to any one of claims 1 to 11.

13. The second type is serrated lesions The image processing device according to any one of claims 1 to 12.

14. The third type is a neoplastic lesion that is different from serrated lesions. The image processing device according to any one of claims 1 to 13.

15. A method for operating a medical device, comprising: a processor acquiring a medical image including a lesion area indicative of the lesion; and the processor performs a determination process in which a determiner determines the type of the lesion based on the medical image; the determiner includes a first determiner and a second determiner; the determination process includes a first determination process by the first determiner and a second determination process by the second determiner, the first determination process is a process for determining whether the type belongs to a first group including a first type classified into a first category and a second type classified into a second category, or whether the type belongs to a second group including a third type classified into the second category and different from the second type, The second determination process determines whether the type is the first type or the second type only when the first determination process determines that the type belongs to the first group. This is a process of determining whether the object is a type or not. The processor performs at least one of a first output process of outputting a first signal capable of specifying whether the type is classified into the first category or the second category based on a determination result of the first determination process and, if it is determined in the first determination process that the type belongs to the first group, also based on a determination result of the second determination process, and a second output process of outputting a second signal capable of specifying whether the type is the first type or the second type based on a determination result of the second determination process, if it is determined in the first determination process that the type belongs to the first group. How medical devices work.

16. The image processing device according to claim 1, The processor: acquiring said medical images in a time series over a plurality of frames; determining, for each frame, which of a plurality of lesion types including the first type and the second type, based on the medical image; when it is determined that the type of a first frame of the plurality of frames is the second type, outputting a signal corresponding to a second type determination signal indicating that the type of a second frame of the plurality of frames that is within a range of a predetermined number of frames from the first frame is determined to be the second type; When the type of the first frame is determined to be the first type, a determination result signal corresponding to the determination result for the type of the second frame is output. Image processing device.

17. the category to which the first type belongs is a category indicating non-tumor; The category to which the second type belongs is a category indicating tumorous. The image processing device according to claim 16.

18. the first type is a non-tumorous type, The second type is a type that shows serrated lesions.

18. The image processing device according to claim 16 or 17.

19. The processor performs at least one of a display process for displaying information based on a signal corresponding to the second type determination signal and information based on the determination result signal on a display device in units of frames, and a storage process for storing information based on the signal corresponding to the second type determination signal and information based on the determination result signal in units of frames.

19. The image processing device according to any one of claims 16 to 18.

20. An image processing device according to any one of claims 1 to 14 and claims 16 to 19; an imaging device that captures an image of an observation target region of a subject to obtain an image representing the observation target region as the medical image; A medical diagnostic device comprising:

21. An image processing device according to any one of claims 1 to 14 and claims 16 to 19; an endoscope that is inserted into a body and captures an image of an observation target area within the body to obtain an image showing the observation target area as the medical image; An endoscope apparatus comprising:

Images